Thinking blocks in the tool loop

A tool loop sends a request, receives an assistant turn containing one or more tool calls, executes them, appends the results, and sends the conversation back for the next step. Loops written before reasoning existed build that second request by picking the tool-call blocks out of the response and discarding everything else, because everything else was either text the user had already seen or nothing at all.

Enabling reasoning changes what an assistant turn is, and therefore what replaying one requires. The deliberation arrives as a block inside that turn, carrying an integrity token, and it is part of the unit the provider validates and the model continues from.

What an assistant turn contains#

With reasoning enabled, an assistant turn that calls a tool arrives as an ordered list of blocks, and the deliberation is one of them:

{
  "role": "assistant",
  "content": [
    {
      "type": "thinking",
      "thinking": "The user gave three constraints: destination, price ceiling…",
      "signature": "ErUBCkYIAxgCIkAtq3F1Zk9vT2xQd0hyS2c9PS0tWm1sd..."
    },
    {
      "type": "tool_use",
      "id": "toolu_01A9pKzQ",
      "name": "search_flights",
      "input": {"to": "LIS", "max_price_eur": 400, "arrive_before": "12:00"}
    }
  ]
}

The signature is the piece that makes the rule strict. It is a provider-issued integrity token over the block's contents, and on the next request the provider verifies it. Modify a character of the thinking text, re-order the blocks, drop the signature, or synthesise a plausible-looking block of your own, and verification fails — which is deliberate, because a thinking block a client could author is a channel for putting words in the model's mouth and having it treat them as its own prior reasoning. Reading the block is fine. Editing it, in any degree, is not.

What that channel would be worth follows from how a model reads a transcript. Everything inside an assistant turn is conditioned on as the model's own prior output, with the same weight as text it actually generated, and nothing in the format separates a decision the model reached from text placed in the slot where its decisions go. A client able to author a block there could put arbitrary instructions into the model's own voice, upstream of whatever the system prompt carries. Verification therefore covers the block's contents rather than its presence, and an edit of any size fails because a token that still verified after the bytes beneath it changed would assert nothing about them.

So the second request in a tool loop carries the assistant turn back unchanged — thinking block first, then the tool-call block, in the order they arrived — followed by a user turn containing the matching tool results:

The reason the provider insists is not bureaucratic. Inside a multi-step task the deliberation is where the plan lives — which constraints have been satisfied, which tool is being called and why, what to do with the result when it comes back. The turn that follows a tool result is a continuation of the same piece of work, and the model resumes it by reading its own prior reasoning in the transcript. Remove that and you have asked a different question: not "here is the result of the search you decided to run", but "here is a search result; work out what you were doing".

That reframe is exact, and the property underneath it is that the model holds no state between calls. Each request is answered from the text of that request and nothing else, so whatever the model worked out on step one existed only as tokens it emitted, in the turn it emitted them, and the transcript you assemble is the only place those tokens can still be. Sending the tool result without that turn's deliberation therefore does not hide the plan from the model; it deletes it. What comes back is then the only thing available to a model with nothing to fall back on — a plan re-derived from the transcript in front of it, fluent, addressed to the same task, and worse — which is why the failure presents as a degradation rather than an error.

The belief that makes this easy to get wrong is that reasoning happens inside the model and can therefore be discarded in transit, the way a debug log is. It is tempting because on a single-turn request the block genuinely is disposable: the answer is read, the block has done its work, and nothing asks for it again. Within a loop it is text in a transcript, that transcript is the whole of the next step's input, and no copy of it exists anywhere else.

The loop that breaks is the one that selects blocks by type, which is the shape almost every pre-reasoning tool loop has:

tool_calls = [b for b in response.content if b.type == "tool_use"]

messages.append({"role": "assistant", "content": tool_calls})
messages.append({"role": "user", "content": [
    {"type": "tool_result", "tool_use_id": tool_calls[0].id, "content": results},
]})

The one-line fix is to stop filtering:

messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": [
    {"type": "tool_result", "tool_use_id": call.id, "content": run_tool(call)}
    for call in tool_calls
]})

Append the whole content array, whatever is in it, in the order it arrived. That form is also robust to block types that do not exist yet, which is worth something in an API surface that has added three of them in two years.

Two variations on the shape are common enough to plan for. A single assistant turn may contain several tool_use blocks when the model decides to run calls in parallel, and every result belongs in one user message: splitting them across consecutive user messages is either rejected or read as several separate turns, and in the second case the model learns from the transcript that its parallel calls were answered serially and makes fewer of them afterwards. The loop above handles this by building the result list as a comprehension over all calls in the turn, which is why the calls are collected rather than indexed.

The rule takes that shape because the turn is the unit the provider validates and the model continues from. One assistant turn made three calls, so a history answering them in three consecutive user messages describes an episode that did not occur: three turns, each with one call, each answered before the next was made. The model is not being punished for the split. It is reading a record of its own behaviour, and that record now shows calls taken one at a time.

The other variation is a block type you cannot read at all. Providers sometimes return deliberation as a redacted or encrypted block — a redacted_thinking block, or an encrypted reasoning item — when a safety system has flagged the content or when the model family never emits plaintext reasoning. Those blocks carry no text you can inspect, and they are subject to exactly the same rule: pass them back untouched, in order, alongside everything else. Code that filters on block.type == "thinking" and forwards only that will drop the redacted variant and produce a validation failure for a reason that is harder to see, which is another argument for forwarding the array rather than enumerating the types you know about.

In its general form that argument covers block types nobody has invented yet. Code that filters on the types it knows about treats every other type as absent, and absence is indistinguishable from not-applicable where the filter runs: the comprehension does not raise, it returns a shorter list, which is what a turn with no thinking in it would legitimately produce. So the block goes missing silently, in the transcript-assembling code, and the failure surfaces one layer away as a provider error about the first block of a turn.

The failure that does not raise#

Outright rejection — a 400 on the second request, before any tokens are generated — is the friendly version. Not every provider validates the sequence, and the ones that do not will accept a tool loop with the reasoning stripped out and keep going, because the request is well-formed — it just describes a model that called a tool for no stated reason.

Acceptance is the part that misleads, because a request the provider accepts reads as a request that is right. Validation answers a narrower question: whether the payload is well-formed enough to be processed. An assistant turn holding a bare tool call clears that bar, and no provider can tell a block that was stripped from the turn from one that was never emitted, since the two arrive as identical bytes.

The measured effect on one travel-booking agent, once the same bug was reproduced against a provider in the second column, was 5.1 tool calls per completed booking against 3.4 with the reasoning preserved, and a task success rate of 71% against 88%. The extra calls were mostly repeats: the model searched the same route twice with slightly different parameters, because on the second step it no longer knew that it had already decided the price ceiling was a hard constraint and the arrival time was the one to relax. None of that produces an error. It produces an agent that feels a bit dim and costs 50% more tool calls per task, which is a defect that survives a long time because there is nothing to grep for.

Those figures are what re-deriving a plan from a partial transcript produces, rather than a general loss of competence that happens to show up in tool calls. A constraint settled inside the deliberation is an open question again once the deliberation is gone, and a model with an open question and a search tool searches, which is why the surplus calls repeat calls already made: five point one against three point four is half again as many, and that extra half is one round of re-establishing what an earlier round had settled. The seventeen points of task success come from the same mechanism taking a worse turn, since a constraint re-derived from a partial transcript need not be resolved as it was the first time, and a run that relaxes the price ceiling where the first pass relaxed the arrival time books something the user ruled out.

Finding it requires a test that measures rather than asserts. Take thirty recorded multi-step tasks with known-good outcomes, run each twice — once with the assistant turns replayed intact and once with the reasoning stripped — and compare tool calls per task and completion rate. The stripped run is not a hypothetical: it is what your production loop does today if anything in the path drops blocks, so the comparison doubles as a regression test you can keep. Two numbers that differ by a third are hard to argue with in a review, where "the model seems worse in the agent" is not.

The same rule appears in different clothing on stateless-versus-stateful APIs. Where the provider retains conversation state server-side, you continue a loop by referencing the previous response identifier and the reasoning is carried for you. Where you have opted out of retention — store: false, as many organisations require — the reasoning must round-trip through your process instead, and the API hands it to you as an encrypted item you request explicitly and pass back untouched, since the plaintext is not something the provider will emit. The mechanism differs; the obligation is identical, and so is the failure when the item is dropped.

Where the obligation ends, and what carrying it costs#

Reasoning has to survive within the loop, from the turn that produced it until the loop emits its final answer. After that it can go, and it should: subsequent turns of the same conversation do not need step three's deliberation, providers that accept a stale block at all will usually ignore it rather than fail — behaviour worth confirming against the one you use rather than relying on — and the storage question it raises is not one you want to answer more times than necessary.

That gives a rule with two clauses rather than one. While a tool cycle is open — an assistant turn containing a tool_use whose tool_result has not yet been answered by a final assistant message — every block of that turn travels with the request, unmodified and in order. Once the loop has produced its final text, the reasoning from those intermediate turns has no further job, and the history you carry forward can be reduced to the visible messages and the tool trail.

The obligation ends there for the same reason it exists. Deliberation travels while a piece of work is in progress that the model will be asked to resume, because resuming it means reading how it was planned, and a final assistant message is the point at which nothing is outstanding — every tool cycle closed, the answer given, and whatever comes next starting from that answer rather than from the working behind it. The over-correction to expect, once a team has been burnt by a stripped loop, is a rule that carries every thinking block from every turn forever, inflating the prompt with material no step needs.

What re-sending costs is worth having a number for, because it is the argument people reach for when they want to drop it. Reasoning that goes back into the next request is billed as input, not output — at the rates in chapter 1, a fifth of the price per token — and it is eligible for prompt caching like any other stable prefix. A five-step loop carrying 1,200 tokens of deliberation per step re-sends roughly 12,000 accumulated input tokens over the loop, which is 3.6 cents at $3.00 per million, or under a cent with a cache hit on the earlier steps.

The word carrying that arithmetic is accumulated. The first request holds no deliberation at all, since none has been produced yet. The second carries one step's worth, which is 1,200 tokens; the third carries two steps' worth, or two thousand four hundred; the fourth three, or three thousand six hundred; the fifth four, or four thousand eight hundred. Those four add to twelve thousand, which is the 12,000 above — reached by summing what each request had to repeat, not by multiplying 1,200 by the five steps, because a step's deliberation is only re-sent on the requests that follow it.

Twelve thousand tokens is twelve thousandths of a million, and twelve thousandths of $3.00 is 3.6 cents for the loop. The input rate applies for a mechanical reason rather than as a billing courtesy: those tokens were billed as output when the model wrote them, and on the way back they are part of the prompt, which is what input means. A token is priced by the direction it travels on the request in hand.

The cache clause is derivable from the shape of a tool loop's history rather than quoted. A cache works on a prefix — a run of tokens at the start of a request that is byte for byte what an earlier request began with — and a tool loop grows only by appending, since the rule this chapter is about forbids rewriting anything already sent. Every step's request therefore begins with the previous step's request, so the deliberation being re-sent sits inside a prefix the provider has already seen — which is what makes it cache-eligible rather than freshly billed at every step.

Against a 50% increase in tool calls and a seventeen-point drop in task success, a few cents of re-sent input is not a trade anyone would take deliberately. It gets taken by accident, by a filter written before there was anything to filter out.

Three things this rule is not. It is not a licence to show the block to anyone: passing reasoning back to the model that wrote it and putting it in front of a user are different acts with different consequences, and the second is a separate decision made later in this topic. It is not a requirement across tasks — a fresh conversation, or a new task in the same session after the previous one finished, starts clean. And it is not something you can fabricate to steer the model — tempting, because a thinking block looks like the most direct place to install a plan and sits closer to the step than a system prompt does — because the signature check exists precisely to stop that; a synthesised thinking block is rejected, and an edited one is rejected too, however small the edit.

The whole rule reduces to a habit: in a tool loop, append what you received rather than what you understand. A content array is the provider's unit, it is validated as a unit, and the model continues from it as a unit, so any code that takes an interest in its individual blocks is code that will be wrong the next time a block type is added. The version that appends whole arrays and drops reasoning only at the end of a task is correct today and stays correct through API changes nobody has announced yet.

That loop will still fail in production the moment anything between it and the API takes an interest in trimming the conversation — and by the time an agent is worth running, something usually does.