Losses in a normalized response

A normalized response is the type your adapters return: usually something close to {text, tool_calls, usage}, small enough that every vendor can populate it and every caller can consume it without knowing who answered. It is the right shape for the thing it was designed for, which is letting domain code read an answer without importing an SDK. What it also does, silently, is define the set of things that can survive the crossing. A provider reply is a list of content blocks with metadata attached, and blocks with no destination in your type are skipped by the code that maps the ones with destinations.

Nothing about that is a bug in the ordinary sense. The adapter's job is to produce a valid response and it produces one on every call, because each field the type declares is correctly populated. A type system checks that the shape you declared is honoured; it has nothing to say about whether the shape is adequate. Tests check the behaviours you thought to assert, and an assertion about a field that does not exist is not one anybody writes.

Four things go missing this way — reasoning, conversation state, tool semantics and content ordering — and what makes them expensive is not that they are subtle individually. It is that no two of them surface in the same place. Each one manifests as a metric moving in a system other than the one you changed, watched by a different team on a different dashboard, so nobody is ever holding enough of the picture to see that four symptoms share one cause. That is why they run for weeks.

Where the reasoning went#

Extended thinking gives a model a budget of tokens to reason in before answering, and returns that reasoning as a content block of its own alongside the visible text. A single assistant turn comes back looking like this:

{
  "content": [
    {"type": "thinking",
     "thinking": "The user's invoice is dated before the contract renewal, so the...",
     "signature": "ErUBCkYIBRgCIkC7q1..."},
    {"type": "text", "text": "I'll check the contract start date first."},
    {"type": "tool_use", "id": "toolu_01A", "name": "get_contract",
     "input": {"account_id": "acct_8813"}}
  ],
  "stop_reason": "tool_use",
  "usage": {"input_tokens": 4102, "output_tokens": 511}
}

The adapter flattens it:

def to_response(raw) -> ChatResponse:
    text = "".join(b.text for b in raw.content if b.type == "text")
    calls = [ToolCall(b.id, b.name, b.input) for b in raw.content if b.type == "tool_use"]
    return ChatResponse(text=text, tool_calls=calls, usage=Usage(...))

The thinking block has no destination, so it is skipped by a comprehension that only looks for text and tool_use.

Why that skip is silent rather than loud is a property of how the mapping is written, and it is the one thing in this chapter that could be changed with a few lines. Both comprehensions are filters: they walk the blocks and keep the ones whose type they recognise. A filter has exactly two outcomes per block, keep and drop, and "dropped because I do not recognise it" is the same outcome as "dropped because it was not what I was looking for". Nothing distinguishes them, so nothing can report one and not the other. Written instead as an exhaustive match over block types — a case for text, a case for tool_use, and a default that raises or logs the unknown type with the response id — the same adapter would have announced the thinking block the first time one arrived. The information was never hidden by the provider. It was discarded by a control structure with no branch for surprise.

That is the shape to look for throughout the rest of this chapter. Every loss below is a value the provider sent, that your code received, and that a selection step declined to carry, and none of them can raise an error, because declining to carry something is what the selection step is for.

A dropped thinking block would be harmless if the conversation ended there. The damage happens on the next turn, because an agent loop rebuilds its message list from its own database, and the database only ever received what the adapter returned.

From turn two onwards the model sees a transcript in which its previous turn consisted of one sentence of narration, with the several hundred tokens of reasoning that produced that sentence removed. On a single-step task nothing detectable happens. On a planning task, where turn six depends on a constraint the model worked out in turn two and never wrote into its visible text, completion rates fall while output-token spend barely moves and no request is rejected. Some providers additionally reject or ignore thinking blocks whose cryptographic signature is missing or mismatched, so an adapter that tries to reattach reasoning from a text field it stored separately can turn a silent degradation into a 400 — which, awkwardly, is the better outcome, since somebody notices on day one.

The continuation handle nobody passed back#

Some vendors keep conversation state on their own side. The response carries a response_id, and the documentation says to send that id on the next call instead of replaying the transcript; the server attaches the prior turns, including its own internal representation of them, and bills you only for the new input. A {text, tool_calls, usage} shape has no field for it, so to_response drops it exactly the way it dropped the thinking blocks, and the agent loop does what it always does: rebuilds the entire message list from its store and sends the whole thing.

Nothing fails. The provider accepts a full transcript; that is a legal request. The consequences are arithmetic. A twelve-turn conversation with a 3.5k-token system prompt and roughly 2.1k tokens added per turn ends up at about 28k tokens of context by the last call. Replaying it every turn sends about 190k input tokens across the conversation, against roughly 32k if each turn had sent only the delta behind a continuation handle. At $3 per million input tokens that is $0.57 against $0.10. Latency moves too: prefill on 28k tokens adds around 1.1 seconds to time-to-first-token on the later turns of every conversation, enough to push a p95 from 3.2 to 5.6 seconds without any single request being slow enough to trip an alert.

Those figures are worth taking apart, because the shape of the growth matters more than any one of them. The context at the last call is the system prompt plus twelve turns of additions: 3.5k plus twelve times 2.1k, which is a bit over 28k. That is the size of one request. The total for the conversation is a different quantity, and it is the sum of every prefix — turn one sends the system prompt plus one turn's worth, turn two sends the system prompt plus two turns' worth, and so on to twelve. Each turn resends everything that came before, so the total is proportional to the square of the turn count rather than to the turn count. Sending only the delta behind a continuation handle makes the total proportional to the turn count, which is why the two totals differ by roughly six times at twelve turns and would differ by twice as much again at twenty-four. This is the "superlinear" in the table below, stated as the reason for itself.

The money then follows in one step, because input tokens are billed per token and nothing else in the request changed. At $3 per million, 190k tokens is a shade under a fifth of a million, and a fifth of three dollars is around $0.57; 32k is about a thirtieth of a million, and a thirtieth of three dollars is about $0.10. Neither number is alarming for one conversation, and that is exactly why this survives: the quantity that would alarm anybody is the difference multiplied by the conversation count, and the conversation count lives on a different dashboard from the per-conversation cost.

The latency figure behaves differently from the cost, and the difference is worth understanding rather than averaging away. Prefill time scales with the input, so the added second or so lands only on the turns with a large transcript behind them — the later ones. Those are the same requests that were already the slowest in the distribution, because a long conversation is slow for other reasons too. Adding time to the requests already at the top moves the tail without moving the middle, so a p95 can travel further than the per-request addition suggests while the median barely registers it, and no individual request is ever slow enough to look like a fault.

The subtler part of this loss is that server-held state is not only a transcript. Whatever the provider had cached, precomputed or retained about the conversation is reachable only through that handle, so discarding it does not merely cost you the resend — it starts every turn from a cold provider-side state you are then paying full price to reconstruct.

Tool calls that keep their names and change their meaning#

The third loss is the hardest to see because the field exists and is populated. ToolCall(id, name, input) looks vendor-neutral, and two providers do both return something that maps onto it. What does not map is everything around it.

One vendor supports a strict-decoding mode where the sampler is constrained to the tool's JSON schema, so arguments are guaranteed parseable and type-correct. Another validates after generation and returns whatever the model produced. A normalized ToolCall carries no notion of which regime produced it, so calling code written against the strict provider — and therefore calling json.loads without a fallback — meets malformed arguments on a few percent of calls after a move, which surfaces as a JSONDecodeError in a handler that treats all exceptions as retryable and retries the whole turn. Duplicate side effects follow from there: a retried turn re-executes a tool that had already succeeded, because the id that would have let the loop deduplicate was a provider-scoped id the adapter regenerated on each attempt.

What makes this loss different in kind from the first two is that the field is present and the type checks, so the artifact is a misrepresentation rather than an omission. ToolCall(id, name, input) says the same thing whichever adapter built it, and the thing it says is not the same thing in both cases. Under a strict-decoding vendor, input is a value the sampler was prevented from producing incorrectly. Under the other, input is a value the model was asked to produce correctly and usually did. Those are different guarantees wearing one type, and the caller has no way to interrogate which one it holds, because interrogating it would require a field the contract does not have. A dropped block leaves a caller with less than it wanted. A collapsed guarantee leaves a caller confident about something that is no longer true, which is the more expensive of the two.

Ordering is the other half. A provider that interleaves text and tool calls in a single content array lets an agent's user interface show the narration attached to each call. Flattening concatenates all text into one string and all calls into one list, which discards the interleaving, so a turn that said "checking the contract" before one call and "now looking at the invoice" before another comes out as a single paragraph followed by two calls. No error, no eval regression, and a support agent reading the trace can no longer tell which sentence belonged to which action.

The losses, their symptoms, and where they surface#

The pattern across the rows is that each loss produces a symptom in a system that is not the one you changed. The reasoning loss shows up in eval scores owned by a data science team, the continuation-handle loss in a finance dashboard, the tool-semantics loss in an error log that reads as flakiness. Nobody is looking at all six places, and each one individually has a plausible unrelated explanation.

A conformance suite goes green on exactly the losses#

That the shared suite passed throughout is structural rather than a matter of effort. A conformance suite asserts that every implementation satisfies the contract, so it is defined entirely in terms of fields the contract has. It cannot assert that reasoning survived, because there is no field to read it from; it cannot assert that a continuation handle was used, because the contract has no concept of one. Running it against both providers is a genuine and useful check that they agree, and they do agree — they agree on the flattened shape, which is precisely where the information already went missing.

It is worth asking, with the invoice in hand, whether the abstraction delivered what it was sold as. On the narrow reading it worked exactly as advertised: the code change was two days, and without the seam it would have been a month. What it did not do was make the vendors substitutable, because substitutability was never a property of the call sites — it is a property of the capabilities the application depends on, and those had drifted onto one vendor's extended thinking, one vendor's caching behaviour and one vendor's decoding guarantees while the interface stayed the same size. The seam bought a cheap code change and hid the fact that the expensive part of the move was never code.

The honest limitation is that some of this is irreducible. If two vendors genuinely disagree about whether conversation state lives on the client or the server, no shared response type represents both without becoming a union that callers have to branch on, at which point the contract has stopped hiding the difference and you are back to writing vendor-aware code with extra steps.

What a normalized response costs, then, is a list of capabilities you can no longer use, written nowhere and enforced by omission. The remedy is not a richer type — a richer type has the same property one level out — but knowing which capabilities your application actually depends on, checking whether they survive a specific crossing, and binding the features that need them to the vendor that provides them, deliberately.

A reasonable reaction to all of this is to stop maintaining adapters by hand and let something else own the normalization — a gateway that fronts every provider, holds the keys, and gives you one endpoint. That trade is real and sometimes correct, and it changes the question from what your contract loses to who is on call for the thing in the middle.