The context window and maximum output

A model exposes two limits that are easy to conflate and bound different things. The context window is the total number of tokens one request may occupy, counting everything you send and everything the model writes back. The maximum output is a separate ceiling on the reply alone, and it does not enlarge when the input is small: a model with a 128,000-token window and a 4,000-token output cap will emit at most 4,000 tokens whether you sent it 120,000 tokens or twelve. A packer that knows only the first number can produce requests that are accepted, billed and useless.

The two are also different kinds of object, and that difference explains why one of them feels arbitrary. The context window is structural: it falls out of how the model was built, and a request that exceeds it cannot be served at all. The maximum output is a serving policy — a ceiling the provider applies to a single reply — and a policy is under no obligation to bear any relation to how much room is left. That is why two deployments of the same weights can publish different values for it, and why the only reliable way to learn it is to read it off the model card rather than to infer it from the window.

This chapter derives the reserve that keeps a request from being accepted, billed and useless. The worked examples are a research summariser on a 32,000-token model and a contract-rewrite service that no single model can serve.

Budget backwards from the answer#

The two limits draw on one pool for a reason that is visible in how a model produces text. A model reads a sequence of positions, and how many positions it can read is fixed when the model is built. Generating an answer works by taking the piece just produced, appending it to that sequence, and running the model again on the now slightly longer sequence. A token the model wrote therefore occupies a position of exactly the same kind as a token you supplied, and by the time the model reads them there is no distinction left between the two.

What follows is that the limit is a limit on the sequence rather than on either half of it. Nothing is held back for the reply, so filling the window with input is not filling it up to a line; it is spending the completion's allowance on documents. The correction is to decide the answer's size first and treat it as spent before the packer sees the remaining space:

WINDOW   = 32_000
ANSWER   = 1_200        # what a good summary of this kind actually needs
FIXED    = 1_450        # system block, tools, framing — measured, not evictable
RESIDUAL = 0.042        # p99.9 estimator residual from chapter 3's calibration

def input_ceiling() -> int:
    return int((WINDOW - ANSWER - FIXED) * (1 - RESIDUAL))

def request_for(docs: list[Doc]) -> dict:
    req = build(docs, ceiling=input_ceiling())
    req["max_tokens"] = ANSWER          # the reserve and the cap are one number
    return req

Setting max_tokens equal to the reserve is the part that turns a convention into an enforced constraint. Most APIs validate input_tokens + max_tokens against the window at request time, so a packer that overruns its ceiling now gets a 400 during a deploy instead of shipping truncated answers for a quarter. The failure moves from silent and user-visible to loud and engineer-visible, which is the trade you want.

What makes it silent otherwise is where the truncation is reported. A model that runs out of room does not raise; it stops, and says so in a field beside the content:

{
  "id": "msg_01Hq2",
  "stop_reason": "max_tokens",
  "usage": { "input_tokens": 31900, "output_tokens": 100 },
  "content": [{ "type": "text", "text": "...particularly among users who had" }]
}

A stop_reason of max_tokens — finish_reason: "length" on OpenAI-shaped APIs — means the model was cut off rather than finished. It is not an error, it does not raise, and it will not appear on an error-rate graph. It is the only signal that distinguishes an answer the model chose to end from an answer that ran out of room.

It does not raise because, from the provider's side, nothing went wrong. The request was valid, the tokens were generated, the response was returned and the call was billed; the reply ended because it reached a ceiling the caller set, and the provider reports that in the same field it uses for every other reason a generation ended. An HTTP status describes the transaction and stop_reason describes the generation, and those are two different objects. Code that watches the first cannot see the second, which is how an error-rate graph reads as unbroken success while a large share of the answers stop mid-word.

Deciding the value of ANSWER is an empirical question with a cheap answer. Run 200 representative requests with max_tokens set generously — 4,000, say — and look at the distribution of output_tokens in the responses. If the p95 is 840 and the p99 is 1,150, then 1,200 is a defensible reserve and 400 was never going to work. Doing that measurement once is the difference between a number with a derivation and a number someone picked because it looked round.

The reserve is a bet rather than a guarantee, because output length is unknown until it exists. What makes the bet tolerable is that you now own the truncation: it happens at a size you chose, it is reported by stop_reason, and you can alert on it. A rate of max_tokens stops above about 1 percent of calls means the reserve is too small for the work; a rate near zero on a task where you expected long answers can mean the model is stopping early for other reasons and is worth looking into.

A generous cap is not a free option#

The reflex that avoids all of this arithmetic is to set max_tokens to whatever the model allows and let the answer be as long as it wants. On an API that validates input_tokens + max_tokens against the window, that reflex spends the whole allowance on every call: a max_tokens of 8,192 on a 32,000-token model means the packer may never send more than 23,808 tokens of input, even for the ninety percent of requests whose answers come in under 400 tokens. You have bought insurance against truncation by permanently surrendering a quarter of the window's reading capacity.

The subtraction is the whole of the argument: 8,192 taken from 32,000 leaves the 23,808, and it leaves it on every request rather than only on the ones that turn out to need a long answer. What makes the reservation binding rather than advisory is the validation itself. The check on input_tokens + max_tokens happens before generation starts, at which point the length of the answer does not yet exist, so the only quantity available to check is the one you declared. A declaration made under those conditions is spent at request time and refunded never.

Setting the cap generously is the stumble worth naming, because the reflex behind it is sound elsewhere. A cap that is never reached usually costs nothing, which is true of most limits an engineer sets and is why setting this one generously feels free. On an API that validates the sum it is not a limit on what the answer may cost but a claim on space, and the claim is honoured whether or not the answer ever uses it.

Different tasks in one service want different reserves, and treating the reserve as a per-task constant rather than a global one recovers most of that space. A classification call that returns one of six labels needs 20 tokens. An extraction call returning a twelve-field object needs 300. A drafted reply to a customer needs 900. A clause rewrite needs 5,000. Setting each from its own measured p99 gives the classifier 23,700 tokens of input room where a global reserve would have given it 15,500, which on a retrieval-backed classifier is eight more passages.

Truncation also has a form that is not visible as a half-finished sentence, and it is the one that produces confusing bugs. When the model is emitting structured output or tool arguments, hitting the cap cuts the JSON mid-object, so what the application receives is not a short answer but an unparseable one:

{
  "stop_reason": "max_tokens",
  "content": [{ "type": "tool_use", "name": "file_ticket",
    "input": "{\"customer_id\":\"cus_8812f\",\"summary\":\"Duplicate charge on the 3rd of\"" }]
}

The parse fails, an agent's error handler catches a JSONDecodeError, and the problem is filed as malformed tool arguments with the model blamed for producing invalid JSON. The model produced valid JSON and was interrupted. Checking stop_reason before attempting the parse turns a mystery into a one-line diagnosis, and it is the reason the field belongs in the error path and not only on a dashboard.

The misdiagnosis is consistent rather than unlucky, and the reason is where the exception comes from. The failure is raised by your own parser, on a string your own code handed it, so the traceback names the parser, the position it choked on and the tool arguments. The response's stop_reason appears nowhere in that traceback, because nothing on the parse path ever read it. Every piece of evidence in front of the engineer therefore points at the model's output, and the one fact that explains it is sitting untouched in a sibling field. That is why the check belongs immediately before the parse rather than anywhere else: it is the only position from which the truncation is still visible at the moment the confusion begins.

A dashboard is still worth having alongside that check. A counter incremented on every max_tokens stop, broken down by task type, is the cheapest possible instrument for a class of quality regressions that otherwise reaches you through customer complaints:

resp = client.messages.create(**req)
metrics.increment("llm.calls", tags={"task": task, "stop": resp.stop_reason})
if resp.stop_reason == "max_tokens":
    log.warning("output_truncated", task=task, reserved=req["max_tokens"],
                emitted=resp.usage.output_tokens, input=resp.usage.input_tokens)

The threshold worth alerting on is roughly one percent of calls for a given task. Below that, occasional truncation is the tail of a distribution and the answer is usually to raise that task's reserve by a few hundred tokens. Above it, the reserve is systematically wrong or the task is asking for more than the model's cap allows, and those two need different responses.

The second cap bounds something different#

The window bounds input plus output together. The maximum output bounds output alone, and published pairs of the two numbers vary enough that the same job is feasible on one model and impossible on another:

Two of those columns are derived rather than published, and both are worth checking once so the rest of the table can be read at a glance. The fourth column is the window minus the maximum output, because a request that intends to leave room for the longest answer the model could give has to reserve exactly that much and no less: 128,000 less 4,000 is the 124,000 on Model A's row, and the same subtraction produces the other three. The last column converts tokens into words and pages using the rates the contract scenario states further down — 1.35 tokens per word for legal English and 500 words per page — which puts a token at about three quarters of a word and a page at about six hundred and seventy-five tokens. Applying that, 4,000 tokens is about 3,000 words and about 6 pages, 16,000 is about 12,000 words and 24 pages, 64,000 is about 48,000 words and 96 pages, and 8,000 is about 6,000 words and 12 pages.

That conversion is a planning instrument rather than a boundary check, and the distinction matters because the rate it rests on is a property of the prose being measured. Punctuation-dense text, split identifiers and under-represented scripts all move it, sometimes by a large factor, which is the subject of this topic's first chapter. Use it to decide whether a job is the right shape for a model, and use a count of the actual payload to decide whether a particular request may be sent.

Model D is the row that makes the distinction concrete. A million-token window is an enormous reading capacity attached to a small writing capacity, and a job specified as "read everything and produce a full rewrite" is bounded by the second number no matter how impressive the first one is. Model B inverts it: a modest reader that can write a small book, provided the source fits in 16,000 tokens after you have reserved for the output.

Two details complicate the table in practice. Reasoning models, which are trained to work a problem through in writing before they commit to an answer, spend that working out as hidden reasoning tokens drawn from the same output allowance, so a 4,000-token cap on such a model may leave 1,200 tokens of visible answer after 2,800 tokens of thinking, and the reserve has to be sized for both. And some providers apply a lower default max_tokens than the model's documented ceiling, so the effective cap is whatever you did not set.

Sixty pages in, six pages out#

Budgeting backwards presumes the job fits somewhere, and there is a class of job for which no reserve computation exists. Reserving is a way of dividing a window rather than a way of enlarging one, and the same is true of eviction: dropping history frees input tokens, so when the binding constraint is on output, an eviction policy that frees 20,000 input tokens changes nothing.

The window and the maximum output are two constraints, and almost every silent truncation comes from budgeting against the first while the second was the one binding. Reserving the answer's tokens before the packer runs, sizing that reserve from a measured distribution per task, and setting max_tokens to the same number converts a class of user-visible defects into deploy-time rejections; reading stop_reason on every response converts the remainder into a graph. None of that arithmetic tells you whether an answer that fits is a good answer — a 1,200-token reserve on a summary of 30,000 tokens of documents is a quality decision made implicitly by choosing a number — but it removes one specific cause of an answer being unusable.

What is left over is the class of job that is output-bound. Sixty pages in, sixty pages out, when the largest writer available emits six. That job has to stop being one call, and turning it into several has its own arithmetic: what context each piece needs, what the duplication costs, and which properties of the finished document become your responsibility once no single generation holds all of it. That is the next chapter.