A second call that learns nothing the first already knew

Not every proposed decomposition is a trade. Some of them have no numerator at all, and the way to find those before they ship is to ask one question about each proposed stage: what does it receive that the stage before it did not have? A stage that reads a retrieved passage, a row from a policy table, the result of a computation somebody else verified, or an output from a different modality has an answer. A stage whose only new input is a summary of what it already holds does not, and the round trip it costs is pure overhead.

The shape shows up most often as reason-then-act: first classify, then produce the structured answer given the classification. That instinct is usually sound — a model that commits to a frame before it starts producing structure often produces better structure — and the mistake is in the delivery mechanism rather than the idea. Committing to a frame costs a handful of output tokens inside one call. Committing to a frame in a separate call costs a network round trip, a second queue wait, a second prefill, and a second prompt to version, and buys the same commitment.

What the second call knows#

Take a search endpoint that turns a one-line customer request into a structured catalogue query, and a proposal to split it into two calls: first detect the intent, then write the query given that intent. Call two receives the customer's request and a label that call one produced from that same request. It has the identical raw input plus roughly six tokens of derived information.

call 1 in : "cheap running shoes for flat feet under 60"
call 1 out: {"intent": "product_search"}

call 2 in : "cheap running shoes for flat feet under 60"
            + {"intent": "product_search"}
call 2 out: {"category":"footwear","subcategory":"running",
             "price_max":60,"attributes":["stability","arch-support"]}

Reading that trace, the question is what call two can now do that a single call could not. A single call reading the same string can perfectly well decide it is a product search and write the query in the same completion; deciding is not a step that requires its own network hop. Nothing was fetched, nothing was decided by a system with knowledge the model lacks, no document was pulled in, no policy was consulted. The label is a lossy function of something the second call is holding in its hands anyway, which makes the second call a relay.

The belief that makes this proposal attractive is worth naming precisely, because it is a good rule being applied slightly past where it reaches. "One prompt, one job" is sound advice and a reviewer who has spent a quarter untangling mega-prompts has earned the right to it. What it is advice about is responsibility — which prompt owns which decision, so that editing one does not put the others at risk. It is not advice about round trips, and the two come apart exactly here. Responsibilities can be separated inside a single call by naming the fields and constraining each one; a round trip cannot be separated from the network it runs over. The decompositions that earn their keep separate both at once. A relay separates the responsibility on paper and pays for the round trip regardless.

The test generalises past this example, and it is worth stating as a checklist because it can be run against a proposal document rather than against a running system. New information means one of four things: a document or record fetched from somewhere the previous stage did not read, a computation performed by something other than the model, an output produced in a different modality, or a decision made by a system holding knowledge the model does not have. A classification the model derived from text it is about to read again is none of those. Neither is a reformatting, a normalisation the model could have applied inline, or a restatement of the request in different words. The awkward middle case is a stage that produces something the model could have derived but reliably does not — and the honest way to settle that one is the ablation from the opening chapter of this topic, run on the single call with the intermediate field present and absent.

Per-call overhead dominates a fast task#

The cost of a relay is not proportional to how little it does, which is what makes the shape survive review. Timed over 200 warm requests against each design of the search endpoint, with the provider's own reported inference time separated from wall clock:

The generation row is the only one where the model is doing more work, and it is the smallest of the four by a wide margin. The other 175 ms is fixed per-call overhead being paid a second time, which is why shortening the prompts or trimming output tokens does not recover it.

The two totals in that table are additions worth doing, because they are what makes the argument arithmetic rather than distaste. Add the four deltas: 55 plus 70 plus 15 plus 50 is 190 ms, which is the whole cost of the change. Add the three that are not generation — 55, 70 and 50 — and you get the 175 that no amount of prompt work touches, because none of it is the model thinking. And notice why the generation row moves so little. The two designs produce almost the same number of output tokens in total; the split merely produces them in two sittings, so the extra 15 ms is the second call re-reading a string it has already been given rather than any new reasoning.

The general form of that is the sentence to carry away from the table. Per-call overhead is a constant, and a constant is a large fraction of a small thing. On a task whose generation takes a tenth of a second, one extra call is close to doubling the request; on a task whose generation takes ten seconds it would be lost in the noise. Nothing about the relay changes between those two cases — the same second call is just as empty in both — but only one of them makes it visible on a dashboard, which is why the shape survives review in slow systems and gets caught in fast ones. On any task whose generation time is around a tenth of a second, per-call overhead dominates the request, and adding a call is close to doubling its latency however small the added call is.

Quality does not move to compensate. On a 500-query eval set built from real search logs, the split scored 94.1% against the single prompt's 94.3%. Five hundred cases at a proportion near 0.94 give a standard error of about 1.1 percentage points, so a two-tenths difference is not a difference at all; the honest statement is that the two designs are indistinguishable on quality at this sample size, and you would need on the order of 10,000 queries to resolve a gap that small even if it were real.

That standard error comes from one expression and reproducing it takes a moment. Multiply the proportion by one minus the proportion, divide by the number of queries, take the square root. At ninety-four in a hundred over five hundred queries the first step gives a little under six in a hundred, dividing by five hundred gives about one in ten thousand, and the square root of that is the 1.1 points. The measured gap is two tenths of a point, which is under a fifth of one standard error — not a small effect that the sample is struggling with, but a difference the sample cannot see at all. Resolving a gap that size would need a set larger than this one by a very large multiple, and the useful conclusion is not "measure more" but "stop measuring this and go and look at the 6% that are wrong".

Two of everything, operationally#

Two calls per request is also two of everything on the operations side, and those items tend to bite some weeks after the change has shipped and been forgotten.

Rate limits arrive first, because a split doubles request volume against an account ceiling that was sized for the old design. Raising the limit is a conversation with an account manager rather than a code change, and the usual stopgap of queueing at the edge converts a rate-limit error into a latency spike on an endpoint that may have had very little headroom to begin with.

Prompt caching, which absorbs part of the duplicated input on a split over a large document, does nothing at all when the shared context is a customer's twelve-token query. There is no large block of context to amortise and most providers will not cache a prefix that short in the first place. Splits over small inputs get none of the mitigation available to splits over large ones, which means the cost analysis for a small task is closer to the uncached worst case than to the number a team may be used to quoting.

Retries acquire a semantics question the single call never had. A transient failure on call two can be retried on its own only if you have kept call one's output and made call two safe to repeat, and the three-line version most services ship replays the whole chain, so the effective cost of this endpoint's 0.7% transient failure rate roughly doubles. Two prompts also means two artefacts to version and two eval sets to keep in step, along with a coupling nobody writes down: changing an intent taxonomy from four labels to six silently invalidates the query prompt's few-shot examples — the handful of solved cases a prompt carries to demonstrate the format it wants — because every one of them carries an old label.

A relay stage removes a recovery path#

The split also introduces a failure mode the single call did not have, and it comes from the same property that made the round trip pointless. When call one assigns the wrong intent, call two receives that label as an input supplied by the system rather than as something it drafted itself, and it treats the label as settled.

On the same 500-query set, 14 queries received the wrong intent from call one, mostly the genuinely ambiguous ones — "where are my running shoes" read as a product search rather than an order lookup. In the split, all 14 produced a catalogue query consistent with the wrong label. In the single call, where the model writes the intent and then keeps reading the same customer string while it produces the query, 9 of the 14 produced a query a human rated usable, because the query was being generated from the customer's words rather than from a six-token label. A relay stage takes an internal draft and promotes it to an external premise, and a model has little licence to revise a premise it was handed.

Why the same model behaves differently on the same customer string is worth deriving, because the instinct is that it should not. In the single call the intent is a handful of tokens the model wrote itself, and every token of the query that follows is conditioned on the customer's sentence and on that draft together — the sentence is still the nearest and largest piece of evidence about what was actually wanted. In the split, the label arrives inside call two's input. Nothing in a flat sequence of tokens records where a span came from, so a label sitting in the input is indistinguishable from a fact the system looked up, and it is read the way the instructions in the same request are read: as given. That is the entire difference. A model will happily revise a wrong draft of its own; recognising that something handed to it in the input is wrong, and then contradicting it, is a move it has far less licence to make.

Five queries out of 500 is one percentage point, which is small in absolute terms and points the same direction as everything else. The split costs 190 ms against a 60 ms margin, doubles the request rate against a rate limit, adds a second prompt and a second eval set to maintain, and the only measurable quality effect it produced went the wrong way. There is no column in that accounting with a positive number in it, which is what distinguishes a relay from an ordinary expensive change.

Ordering fields inside one call#

The commitment effect the proposal was after does not require a second call. Asking for the intent as the first field of a single response gets it for a handful of output tokens and zero additional round trips:

{
  "intent": "product_search",
  "query": {
    "category": "footwear",
    "subcategory": "running",
    "price_max": 60,
    "attributes": ["stability", "arch-support"]
  }
}

Field order matters, because the model generates left to right and the query is conditioned on the intent it just wrote. That is an in-call ordering decision rather than a decomposition, and it costs about 8 output tokens — roughly 4 ms of generation on this endpoint. With a structured-output API the ordering stops being something you hope the model complies with, because constrained decoding — the provider restricting which tokens the model is allowed to emit next so that the output cannot leave the schema — follows the schema's declared property order:

SEARCH_SCHEMA = {
    "type": "object",
    "properties": {
        "intent": {
            "type": "string",
            "enum": ["product_search", "order_lookup", "store_locator", "returns"],
        },
        "query": {
            "type": ["object", "null"],
            "properties": {
                "category":    {"type": "string"},
                "subcategory": {"type": "string"},
                "price_max":   {"type": ["number", "null"]},
                "attributes":  {"type": "array", "items": {"type": "string"}},
            },
        },
    },
    "required": ["intent", "query"],
    "additionalProperties": False,
}

result = client.responses.create(
    model=FAST_MODEL,
    input=[{"role": "user", "content": user_query}],
    response_format={"type": "json_schema", "schema": SEARCH_SCHEMA},
)

The intent token is emitted before the query object is opened, so every token of the query is conditioned on a frame the model has already committed to, and the enum makes the label a closed set in exactly the way a separate classification call would have. One round trip, one prompt, one eval set.