The output reserve

The output reserve is the number of tokens taken off the top of the context window and held for the completion, so that the assembler's ceiling is the window minus the reserve rather than the window itself. Topic 2.14 established the mechanism it defends against: the window is one pool that input and output draw from together, and a completion has whatever the input did not take.

What is specific to an assembler with several components is the shape of the failure. It appears exclusively in the tail of the input distribution, so it is invisible at the median; it is caused by whichever slice happened to grow on those particular requests; and no output parameter fixes it. Worse, on most providers it produces no error at all — a 200, a fluent answer, and a stop reason nobody is reading.

Why the two obvious fixes are both no-ops#

A request at 126,000 tokens of input on a 128,000-token model has about 2,000 tokens of room, and that figure is a subtraction rather than an estimate: the window is the total, the input has taken 126,000 of it, and 128,000 minus 126,000 is what is left. Setting max_tokens to 4,096 asks for more than that; setting it to 8,192 asks for even more.

Two limits are now in play, and it is worth being precise about which one binds. The completion may not exceed max_tokens, and it may not exceed the space left in the window. The generation stops at whichever of those two it reaches first, which is always the smaller one — so the effective bound is the minimum of the pair, and a parameter that is already the larger of the two can be raised without moving that minimum at all. Going from 4,096 to 8,192 raises the limit that was not binding. The model generates until it reaches the window and stops, which is exactly the same place it stopped before. The reason the change is so tempting is that max_tokens is the only output-shaped knob in the request, and the symptom is an output that was too short; the reason it does not work is that the shortage was created by the input.

Retrying is worse than useless, for the same reason and one more. The input is deterministic — the same document, the same query, the same retrieval — so the retry assembles the identical 126,000 tokens, leaves the identical 2,000 tokens of room, and truncates in roughly the same place. Nothing about a second attempt changes any term in the subtraction above. What it does change is the bill, since input is charged whether or not the answer is usable: at $3 per million input tokens, 126,000 tokens is 126,000 divided by a million, times three, which is $0.38 a retry. Multiply that by the 610 requests a month that truncate and retry, and it is $230 a month spent reproducing a bug rather than finding it.

What providers do when input_tokens + max_tokens exceeds the window is not uniform, and the difference determines whether you find this in an hour or in a quarter:

A validation error makes this an incident on the day it ships. A silent clamp turns it into a slow leak that support notices before engineering does.

The reason the difference is measured in months rather than in severity is which counter each behaviour moves. An error rate is built from responses that failed, and every monitoring convention in ordinary use — alert thresholds, error budgets, the red line on a dashboard — reads that counter. A clamped request does not fail. It returns a 200 with a well-formed body, so it is counted as a success by the same conventions, and the only field that distinguishes it is stop_reason, which is not a counter at all until somebody makes it one. The failure is therefore not hidden by being rare. It is hidden by being filed on the wrong side of a line that every tool in the stack draws in the same place.

Subtract before assembling, do not cap afterwards#

The correction is to make the reserve a property of the input ceiling rather than a property of the response. The assembler is never told the window is 128,000; it is told the ceiling is 124,000, and it has no mechanism for exceeding it:

WINDOW  = 128_000
RESERVE = 4_000          # sized from the output distribution, below

def input_ceiling() -> int:
    return WINDOW - RESERVE

def build_request(query: str, doc: Document, session: Session) -> dict:
    ceiling = input_ceiling()
    fixed = SYSTEM_TOKENS + TOOL_TOKENS          # measured per deploy, asserted in CI
    available = ceiling - fixed
    chunks  = retrieve(query, doc, token_cap=int(available * 0.62))
    history = select_recent(session, token_cap=available - tokens(chunks))
    req = {
        "model": MODEL,
        "system": SYSTEM,
        "tools": TOOLS,
        "messages": chunks + history,
        "max_tokens": RESERVE,                   # same number, both roles
    }
    total = count_tokens(req)
    if total > ceiling:
        raise BudgetViolation(f"{total} > {ceiling}")
    return req

Setting max_tokens from the same constant that was subtracted is what keeps the two halves honest. If the assembler ever overruns the ceiling, the request now asks for more than the window allows, and on a provider in the top row of that table it becomes a 400 during a deploy instead of a truncated answer in production three weeks later. The explicit BudgetViolation covers providers that clamp: it converts their silence into your exception, at the cost of one token count per request.

Sizing the reserve is an empirical question the output distribution answers directly, and the measurement has to be taken with the ceiling raised out of the way. Running 500 representative contract queries with max_tokens at 16,384 and recording usage.output_tokens gave a p50 of 890, a p95 of 2,740 and a p99 of 3,810, with the longest legitimate answer at 4,510 tokens for a query that asked for a clause-by-clause summary. The 16,384 is doing specific work there: measure the same 500 queries with max_tokens at 4,096 and every answer that wanted more than 4,096 tokens is recorded as 4,096, so the distribution you fit the reserve to is a distribution the current setting already truncated. Sizing a limit from data the limit produced returns the limit.

Reading the percentiles then goes in one direction only. A reserve of 4,000 tokens is above the p99 of 3,810, which by the definition of a percentile means fewer than one answer in a hundred wanted more room than the reserve holds — and the answers that did want more are somewhere between 3,810 and the 4,510 of the longest one, so the shortfall on a clipped answer is hundreds of tokens rather than thousands. Going to 6,000 covers every answer in the sample, including that longest one, and the price of the extra headroom is not zero: 6,000 against 4,000 is 2,000 tokens taken out of the input ceiling on every single request, including the overwhelming majority whose answers finish under 890. For that feature 2,000 tokens is roughly one and a half more contract pages that no longer fit. That trade is a decision with numbers on both sides, which is the form every slice of the budget should take.

Verify at the boundary, not at the median#

A reserve exercised only by median traffic is a reserve that has never been tested. Where the median request leaves 87,000 tokens spare it would survive any bug in the arithmetic, so the test that matters constructs a request deliberately at the edge:

def test_reserve_survives_truncation_at_the_boundary():
    doc = synthetic_document(tokens=200_000)     # far larger than the window
    session = synthetic_session(turns=40, tokens=30_000)

    req = build_request("list every termination right", doc, session)

    counted = count_tokens(req)
    assert counted <= WINDOW - RESERVE, f"input ceiling breached: {counted}"
    assert counted >= WINDOW - RESERVE - 2_000, f"under-filled: {counted}"
    assert req["max_tokens"] == RESERVE

    resp = client.messages.create(**req)
    assert resp.stop_reason != "max_tokens"
    assert resp.usage.input_tokens + resp.usage.output_tokens <= WINDOW

The second assertion is the one people leave out and the one that catches the more embarrassing regression, and the direction it guards is the opposite of the one the test was written for. A ceiling assertion fails when the assembler puts in too much. Nothing in it fails when the assembler puts in too little, because under-filling is not an error condition anywhere in the system — the request succeeds, the answer is fine, and the only thing wrong is that capacity was paid for and not used. An assembler that has become too conservative, through a reserve subtracted twice or a fixed-slice constant that grew, passes the first assertion easily by using 90,000 tokens of a 124,000-token ceiling, and the gap between those two figures is roughly a quarter of the window going unused on every request. Asserting a floor as well as a ceiling turns the boundary test into a check that the budget is being spent as well as respected.

Running this against real input rather than a mock is the point. synthetic_document(tokens=200_000) is deliberately larger than the window, so the assembler cannot satisfy the request without truncating and the truncation path is exercised rather than skipped. The live call at the end costs about $0.37 per run, which is the ceiling of 124,000 tokens priced at $3 per million, and that is why the test belongs in a nightly job or a pre-release suite rather than on every commit. The assertion on resp.usage is checking a different thing again: it reads back what the provider counted, not what you counted, and it catches the case where the two disagree at exactly the boundary — the one place a fraction of a percent has consequences.

When the input and a useful answer cannot both fit#

Some requests do not have a solution at the current window. A 340-page credit agreement is roughly 230,000 tokens, and no reserve arithmetic makes it and a useful answer coexist in 128,000 — the document alone is nearly twice the window, so there is no value of the reserve, including zero, that leaves room for both. That is worth separating from every other failure in this chapter, because the others are allocation problems and this one is not. Nothing is competing for space here. The input does not fit. What the system does then is a product decision that has to be made explicitly, because the default — truncate the document silently and answer from what is left — produces confident answers about a contract the model has only partly read, which is the worst outcome on the list.

The row engineers reach for least and users tolerate best is the fourth. A response saying the agreement is too long to analyse in one pass, followed by an offer to answer per article, costs one round trip and produces an answer grounded in a document the model actually read. Decomposition is covered as a technique in topic 2.14; the budgeting point is that it is a branch you take when the ceiling arithmetic fails, which means the ceiling has to be computed before you have committed to a single call.

A reserve is a bet on output length#

Output length is unknown until the output exists. Holding 4,000 tokens back does not mean the model will produce a complete answer in 4,000 tokens; it means that if the model needs more than 4,000 it will be cut off at a size you chose, reported through a stop reason you are logging, at a rate you can alert on. A length rate above roughly one percent is the signal that the reserve is undersized for the work rather than that a particular request was unlucky.

Reasoning models complicate the bet further, and the complication is a substitution rather than a new effect. The reserve holds a number of output tokens. On a reasoning model the output is thinking tokens followed by answer tokens, both generated, both counted, and only the second visible — so the reserve is being shared by two consumers when it was sized for one. A 4,000-token reserve spent on 3,100 tokens of thinking leaves 900 for the answer, and 900 is not what the p99 measurement said the answer needed. The measurement that produced 3,810 was taken on visible answers, which on that model was the whole of the output and on this one is a remainder. A reserve sized against a distribution of visible answers is therefore the wrong reserve on a reasoning model, and the measurement has to be redone against total output tokens rather than against what the user sees.

The direction the field is moving is towards making the reserve adaptive rather than constant — predicting the answer's length from the query type, so a yes-or-no question reserves 400 tokens and a clause-by-clause rewrite reserves 12,000, with the input ceiling recomputed per request. The prediction is itself unreliable, and getting it wrong in the optimistic direction produces exactly the truncation described here, so the usual shape is a per-intent constant chosen from measured distributions rather than a model-based estimate.

What the subtraction buys is narrow and worth stating plainly: it converts a failure that hides inside a success rate into either an exception at deploy time or a rate on a dashboard. It does not make answers shorter, it does not make documents fit, and it does not decide which of them to keep.

That last omission is the next problem. Subtracting the reserve guarantees the completion has room; it does not decide what gets removed from the input to make that room, and the default answer — take it off the end of the string — turns out to be the reason an assistant occasionally forgets its own formatting rules.