Two calls sharing one document double the input and add the latency

A decomposition is normally proposed on quality grounds and reviewed on quality evidence, and then somebody asks what it costs. The answer that comes back is usually a call count, because that is the visible difference between the design before and the design after. It is the wrong unit. A model call is not a fixed quantity of anything: it carries however much input you put in front of it, it occupies however long the provider takes to read that input and generate an answer, and it does both of those things once per call rather than once per request.

Pricing a decomposition means converting the proposal into four numbers — input tokens, output tokens, round trips and tail latency — at the volume the endpoint actually runs, and then doing it a second time at your measured cache hit rate. The numbers below assume a two-stage chain over a document that both stages need, which is the shape that prices worst; a fan-out over independent halves prices differently on latency and identically on tokens, and the last section says how to tell them apart.

The input is paid for once per call, not once per request#

Take a chain whose two prompts each need a full 4,000-token document in context, because you cannot find deviating clauses without the text and you cannot write a risk note without it either. So the document is sent twice. Each call also carries its own instructions — call it 300 tokens for one prompt and 250 for the other — and each returns roughly 150 tokens of output. Laid against the single call it replaces:

The output column is the one people misread. Splitting a request does not generally produce more generated text, because the two halves produce the same two artefacts the single call produced; the generation work is being relocated rather than duplicated. The input column is where the duplication lives, and it is duplicated in full because context is not shared between calls. Whatever cache you have in front of the model may make the second copy cheaper to serve, and it does not make it free, and it does not exist at all across a deploy boundary or after the cache lifetime has lapsed on a low-traffic endpoint.

The input row repays being taken apart into its terms, because the percentage on the end of it is the least informative thing in the row. The single call carries the document once and one set of instructions: 4,000 plus 400. The split carries the document twice and two sets of instructions: 4,000 twice, plus 300 and 250. The term that doubled is the document, and it doubled because it is the only thing both stages need. Everything else moved by a couple of hundred tokens. That is also why the change is 94% rather than a clean doubling — twice 4,400 is more than 8,550, and the split lands short of it because the two prompts' instructions together are larger than the single prompt's and smaller than twice it. So the rule to carry is not "a split doubles the input" but "a split duplicates whatever both halves need", which predicts the answer for a split over a twelve-token query just as accurately as for a split over a contract.

Put that on a real volume. At 8 million calls a month and a working assumption of $3 per million input tokens and $15 per million output tokens — substitute your own contract rate, the shape of the answer does not change — the arithmetic is small enough to do in one block:

CALLS      = 8_000_000
IN_PRICE   = 3 / 1_000_000    # $ per input token
OUT_PRICE  = 15 / 1_000_000   # $ per output token

single = CALLS * (4_400 * IN_PRICE + 300 * OUT_PRICE)
split  = CALLS * (8_550 * IN_PRICE + 300 * OUT_PRICE)

print(round(single), round(split), round(split - single))
# 141600 241200 99600

Three multiplications produce those three numbers, and they are worth doing by hand once so that the next proposal can be priced on the back of an envelope. A single call costs its 4,400 input tokens at three dollars a million, which is a little over a cent, plus its 300 output tokens at fifteen dollars a million, which is under half a cent — a shade under two cents a call. The split call costs its 8,550 input tokens at the same input rate, which is about two and a half cents, plus the identical output charge, so a little over three cents. Multiply either by 8 million and the printed totals appear; take the difference between the per-call figures first and multiply that by 8 million and the $99,600 appears directly. Note where the money is: input is roughly three quarters of the single-call cost and about six sevenths of the split's, so a proposal that argues about output tokens is arguing about the small term.

Ninety-nine thousand six hundred dollars a month of additional input spend, for a change a proposal document would describe as "one more call". The point is not that the number is unaffordable — plenty of teams would pay it for six points of recall on a clause list — but that nobody had computed it, and a decision made without it is not a decision. Run the same arithmetic at 80,000 calls a month and the delta is $996, which is inside the noise of anyone's cloud bill and settles the question in the other direction. The volume is doing as much work in this decision as the design is.

The second figure is the first one scaled, and noticing that is what promotes volume from a footnote to a term. Eighty thousand calls is a hundredth of eight million and the per-call difference has not moved, so the monthly difference is a hundredth of $99,600. Nothing about the design changed between those two sentences: the same two prompts, the same duplicated document, the same round trips. One of them is a line item somebody has to defend in a budget review and the other is invisible, and the only thing separating them is a number that appears nowhere in the design document.

Sequential means the latencies add#

The second currency is time, and it behaves differently from cost because it is not a sum you can absorb by budgeting for it. Two calls where the second needs the first's output cannot overlap. The clock starts on call two only when call one's last token has arrived, which means you are paying not just for two inferences but for two full round trips, two queue waits, and two time-to-first-token intervals, that last being the stretch where the model has read the input and has not yet emitted anything.

A p95 of 2.5 s against 1.4 s is worth understanding rather than memorising, because 2.5 is not 1.4 doubled. Each call is shorter than the original — 150 output tokens instead of 300, and generation time is dominated by output length — so each half runs at roughly 0.9 to 1.1 s at the p95. What you get back on generation you hand straight over to the fixed per-call overhead, which on this stack was measured at 180 ms of network, TLS and queue wait before the first token, twice instead of once.

Tail latency compounds worse than the median, and this is the part that surprises people who have only looked at averages. If a single call's latency distribution has a p95 of 1.4 s, two independent draws from a similar distribution give you a request whose p95 sits above the sum of the two medians, because the request is slow whenever either call is slow. The rule of thumb that holds up in practice: the median roughly doubles, and the tail more than doubles, so a p99 that was 2.2 s lands somewhere near 4.5 s rather than 4.4 s. If you have a hard timeout at 3 s, a split that looks comfortable at the p50 will start shedding requests at the p99 the week after it ships.

The reason the tail behaves that way is one step of reasoning about probability and it is worth walking, because it is the step that turns a surprise into a prediction. A request made of two sequential calls is fast only when both calls are fast, so the chance of the pair coming in under some deadline is the chance for the first call multiplied by the chance for the second. To keep the pair inside its ninety-fifth percentile, each individual call therefore has to clear a stricter bar than ninety-five percent — the square root of it rather than it — because two probabilities only multiply back to ninety-five when each is a little above it. Percentile bars get further apart the further out you go, so demanding a stricter percentile of each call costs more time than the arithmetic of adding two medians suggests. That is the whole of "the median roughly doubles, and the tail more than doubles", and it holds for any dependent chain of any length.

Backend engineers have a name for the shape of this mistake. It is the N+1 query: a change that looks locally reasonable, adds one more round trip per unit of work, tests fine on a laptop where the round trip is 0.2 ms, and falls over in production where it is 40 ms and there are eight of them. Model calls are the same pattern with the round trip three orders of magnitude larger and the per-call cost visible on an invoice.

That comparison is an analogy and it stops holding at the fix, which is where it would otherwise mislead. The standard remedy for an N+1 query is to fetch everything in one round trip — a join, or a batched load — and the equivalent move here is to put both jobs back in one prompt, which is the mega-prompt the split was proposed to escape. So the shape of the problem transfers and the shape of the solution does not. What carries over is only the diagnostic instinct: count round trips per unit of work, and be suspicious of a change that adds one for a reason that sounds local.

Caching absorbs less of a split than the rebuttal implies#

The standard rebuttal at this point is that prompt caching absorbs the duplicated input, and it is a real effect that is smaller than the rebuttal implies. Caching works on a prefix: the leading tokens of a request, identical byte for byte, can be served from a stored state at a large discount. Whether it helps a split depends on how the two prompts are laid out, and the naive layout defeats it.

If each prompt begins with its own instructions and then includes the document, the two requests share no prefix at all — one starts with clause instructions and the other with risk instructions, so the cache sees two unrelated requests. Reordering so that the document comes first and the job-specific instructions come last gives the two calls a 4,000-token shared prefix, and the second call can hit it. That reordering is free and it is the single highest-value change you can make to a split's cost profile, so it belongs in the proposal rather than in a follow-up ticket.

The step that makes the naive layout fatal rather than merely suboptimal is that matching runs forward from the very first token and stops at the first byte that differs. A prefix cache is not a content-addressed store that could notice the same document turning up in two requests at different offsets; the only question it answers is how far two requests agree from the start. A 4,000-token block both requests genuinely share is therefore worth nothing to them while something different sits in front of it, because the match has already failed before the shared block begins. Moving the document to the front does not make it more cacheable. It makes it reachable, which is a different property and the one that was missing.

Even with the layout right, three things limit the discount. Cache entries expire, typically on the order of minutes, so an endpoint at 3 requests per minute across 50 distinct documents will miss most of the time while an endpoint at 300 requests per second on 20 documents will hit almost always. The discount is a discount rather than a waiver, so cached input still appears on the bill. And writing an entry usually costs more than a normal input token, which means the first call of a pair pays a premium to make the second one cheap, and on a document seen exactly once the pair is more expensive than the uncached version.

The practical consequence is that you price a split twice: once at zero cache hit rate, which is the honest worst case and the number a finance conversation should start from, and once at your measured hit rate, which is the number you expect to pay.