Large context windows

The reasoning that accompanies a migration to a 200,000-token model is usually the same: the retrieval cap existed because six sections was all that fitted, that constraint no longer applies, and more evidence cannot make an answer worse. On a corpus where a typical query matches around forty documents, coming to roughly 90,000 tokens, all of that comfortably fits.

Fitting was never the property that made a cap the right number. A cap derived from a quality curve that flattened at six sections was measured on a model whose window was irrelevant to the result — the seventh and eighth sections bought nothing when they fitted perfectly well. Removing the cap does not revisit that finding, and it adds three costs that a 32,000-token model was concealing by making them impossible: a bill charged per request, a prefill wait the user experiences, and an attention effect that runs in the wrong direction.

The bill arrives on every request#

The change that produces all three is four lines:

-    chunks = retrieve(query, token_cap=budget.retrieval)   # 14k, six sections
+    chunks = retrieve(query, match_all=True)               # everything that matches
-BUDGET = Budget(window=32_000, reserve=4_000, system=1_500,
-                tools=3_000, retrieval=14_000, history=9_000)
+WINDOW = 200_000   # we have room now

Input tokens are charged per request, so the difference between 14,000 and 90,000 tokens of retrieval is not a one-off allocation decision. At $3 per million input tokens, a 90,000-token prompt costs 90,000 divided by a million times three, which is $0.27, against $0.049 for the budgeted one, on a feature serving about 128,000 requests a month.

The monthly figures in the table are that per-request cost multiplied by the 128,000, and doing the multiplication once is what makes the change legible. $0.049 across 128,000 requests is $6,270; $0.27 across the same 128,000 is $34,600. The reason a token allocation and a monthly invoice look like different kinds of decision is that one of them has been multiplied by the traffic and the other has not — and it is the same decision. A budget is a per-request statement about a shared resource; on a large window it is also a per-request statement about money, and the resource it protects has moved rather than disappeared.

Time to first token is the number users feel and the one most easily forgotten in a capacity conversation, and the mechanism behind it is worth having rather than the rule of thumb. Before the model can emit a first token it has to read the whole input, because every position in the output attends to every position in the input, and that pass — prefill — is compute proportional to how much input there is. So the wait before anything appears is set by the size of the prompt and not by the length of the answer. The table's own two rows show it: the prompt goes from about 22,000 tokens to about 98,000, which is roughly four and a half times larger, and the p95 wait goes from 1.9 seconds to 9.2, which is roughly five times longer. Those two ratios being nearly equal is what "scales close to linearly" means, and it is why prompt size is a latency parameter and not only a cost one. A 9.2-second p95 is past the point where a streaming interface reads as responsive, and no amount of streaming helps, since streaming changes how the answer arrives after prefill rather than how long prefill takes.

Prompt caching does not rescue this one either. Caching lowers the price of a stable prefix, and forty retrieved documents are the least stable part of the request — they change with every query, so every request pays full prefill for 90,000 tokens of material that has never been seen before. The part of this prompt that caching can help is the 4,300 tokens of system and tools, which was already the cheap part.

More evidence can be worse evidence#

The quality rows in that table are the ones that surprise people, and they have a mechanical explanation rather than a mystical one. When the model reads a position in the prompt it distributes a fixed quantity of attention across every other position — the weights are normalised to sum to one, which is what makes them weights rather than scores. Adding thirty-five more documents does not create more attention to go round; it adds thousands more positions to divide the same total among. So a passage competing with 39 other documents receives less of it than the same passage competing with five. The cause is the arithmetic that produces the weights, which now has thousands more terms in its denominator, and nothing about the model having got tired.

Position matters as well as quantity: material in the middle of a very long prompt is attended to less reliably than material near the beginning or the end, which is the effect topic 4.5 was addressing when it argued for instruction placement and recency. At 22,000 tokens the instructions sat near the end of a prompt the model could hold comfortably; at 98,000 they sit after 90,000 tokens of contract clauses and runbook prose, and the formatting compliance rate on this deployment fell from 97 to 84 percent without a word of the instructions changing.

The wrong-document citation rate is the sharper signal. Passing forty documents does not merely dilute the one that answers the question; it supplies thirty-nine plausible alternatives, several of which describe similar procedures for different services. A model asked to ground an answer in the supplied evidence will ground it in something, and the probability that the something is the right document falls as the pool of confusable neighbours grows. Anthropic's writing on context engineering frames this as treating the window as an attention budget rather than a storage budget, and the practical consequence is that curation is doing work that capacity cannot do.

The counter-evidence usually offered is a long-context retrieval benchmark, where a model asked to find one planted fact in 200,000 tokens scores close to perfectly. Those results are real and they measure a different task. Locating a single distinctive string in a haystack of unrelated filler asks the model to retrieve; deciding which of forty genuinely similar runbooks describes this failure asks it to discriminate, and the filler in the benchmark is not competing for the answer while the thirty-nine other runbooks are. A vendor's benchmark showing clean recall at full window length is not evidence about your task, and the only thing that is remains your own eval set run at both prompt sizes.

The other ceiling is your rate limit#

Cost and latency are per-request effects, and there is a third that is shared. Input token throughput is rate limited per organisation, and the limit is denominated in tokens rather than requests, so the size of a prompt sets how many of them you are allowed to send:

organisation input limit     2,000,000 tok/min
budgeted request                22,000 tok  ->  ~90 req/min of headroom
uncapped request                98,000 tok  ->  ~20 req/min of headroom
observed weekday peak                           46 req/min

Each of those headroom figures is one division. The organisation may spend 2,000,000 input tokens a minute; a budgeted request spends 22,000 of them, and 2,000,000 divided by 22,000 is about 90 requests a minute. An uncapped request spends 98,000, and 2,000,000 divided by 98,000 is about 20. The limit did not change and the number of requests you may send fell by more than three quarters, because the thing being rationed is tokens and a request is only the container they arrive in.

The budgeted version sits at half its ceiling at peak; the uncapped version is over its ceiling by more than a factor of two, since the peak is 46 requests a minute against a ceiling of about 20, which arrives as 429s, the provider's too-many-requests refusal, rather than as a slow dashboard. The retry that follows a 429 re-sends the full 98,000 tokens, so the failure mode is a queue that deepens under exactly the load that caused it. The limit is also usually shared across every workload on the same key, which means the first symptom can be another team's batch job failing at 09:00 while your feature looks fine, and the connection between the two takes a while to find. Requesting a higher limit is a legitimate answer and it is a procurement conversation with a lead time, which is worth starting before the migration rather than during the incident.

When the bigger window is the right call#

None of that is an argument against large windows, which are genuinely the correct answer for a set of problems that were previously unserved. The distinction is whether the input is reducible.

A 180-page credit agreement, which comes to roughly 120,000 tokens, is not reducible by retrieval, because a question about the interaction between the negative covenants and the definitions section needs both, and chunking it produces exactly the seam problems that make decomposition expensive. Feeding it whole to a 200,000-token model costs $0.36 in input tokens — 120,000 tokens at $3 per million — and about 12 seconds of prefill, which is those same 120,000 tokens divided by the prefill rate the ceiling calculation at the end of this chapter passes in, a little under ten thousand tokens a second. Both numbers are large and neither is a reason not to do it, because that is the price of an answer no amount of budgeting produces on a 32,000-token model. Where the input is irreducible and the volume is low — hundreds of requests a day rather than hundreds of thousands — the arithmetic in the table above inverts, and the engineering time spent on decomposition costs more than the tokens.

Five questions separate the two situations well enough to be worth asking in order, and they disagree often enough that the value of the list is making the disagreement explicit rather than producing a verdict:

A contract assistant answers the first question with a firm no and the second with hundreds a day, so it migrates. A support assistant whose retrieval curve knees at six sections and which serves 128,000 requests a month does not. A workload that answers the last question with yes — the same 80,000-token policy manual on every request, never varying — is the case where a large window is close to free after the first call, because caching applies to precisely the material that is stable, which is what made the forty-document version of this feature the worst possible shape.

Long agentic sessions are the second genuine case. An agent working through a forty-step task accumulates tool results that are individually small and collectively large, and compaction — summarising or discarding the earlier steps to make room mid-session — throws away exactly the intermediate state the agent needs. A window that holds the whole session removes a category of subtle failure, and the cost is charged only on the sessions that get that long.

The defects a tight window made visible#

A tight window is an uncomfortable forcing function, and it forces a specific and useful question: what does this request actually need. A tool audit happens because 6,800 tokens hurt. The discovery that a retriever is ranking the answer at position eight happens because only three chunks survive and somebody looks at which three. Those are both retrieval and tool-design defects that exist identically on a 200,000-token model, where they are invisible — the mis-ranked document is now included at position 33 of 40, the answer is still worse than it should be, and the failure has no symptom that points at the ranker.

What the constraint was surfacing, then, was never the constraint's own symptoms. Scarcity does not create a ranking defect; it converts one into a visible loss, because when only three chunks fit, which three is a question somebody has to answer. Remove the scarcity and the defect is unchanged and the question stops being asked. A migration gives up the question, not the defect, and the question is not recoverable by adding a dashboard afterwards, because the measurement that would show it — which document the answer should have cited — is not something the request emits.

That is the real cost of removing the constraint without replacing it. The space problem was legible: requests failed, the eviction ladder fired, an alert went off. The cost and latency problem is a line in a monthly invoice and a percentile on a dashboard, and the quality problem is a slow erosion that no single request demonstrates.

The budget survives the migration with its structure intact and its ceiling derived differently. On a 32,000-token model the input ceiling came from the window; on a 200,000-token model it comes from whichever of cost or latency binds first:

def input_ceiling(window: int, slo_ttft_s: float, prefill_tok_per_s: float,
                  cost_ceiling_usd: float, input_price_per_mtok: float) -> int:
    by_window  = window - RESERVE
    by_latency = int(slo_ttft_s * prefill_tok_per_s)
    by_cost    = int(cost_ceiling_usd / input_price_per_mtok * 1_000_000)
    return min(by_window, by_latency, by_cost)

# 200k window, 2.5s TTFT budget, ~9.8k tok/s prefill, $0.08/request ceiling
input_ceiling(200_000, 2.5, 9_800, 0.08, 3.0)   # -> 24_500, set by latency

Each of those three branches inverts the same relationship, and reading them that way is what makes the function memorable rather than a piece of trivia to copy. The window branch is the old one: whatever the model accepts, less the reserve. The latency branch turns a time budget into a token budget, because prefill is tokens divided by a prefill rate, so tokens is a time multiplied by a rate — two and a half seconds at a little under ten thousand tokens a second is the roughly twenty-four and a half thousand the comment reports. The cost branch does the same with money, because a request's input cost is tokens times a price per token, so tokens is a cost divided by a price — eight cents at $3 per million works out somewhat higher than that. Taking the minimum is the whole of the policy: three ceilings apply at once and the binding one is the smallest, which here is the latency branch, well under the window branch's 200,000.

The five slices, the reserve subtracted before assembly, the eviction ladder and the record with its evidence all carry over unchanged; only the number at the top is derived from a service level objective instead of from a hard limit. That is the version of context budgeting that keeps working as windows grow, because a service level objective does not get larger when the model does.

What remains is to run it on something real. Instrument the five slices on one feature with the target model's tokenizer, on a worst-case corpus rather than replayed medians. Subtract a reserve sized from that feature's output distribution and add the boundary test. Write the eviction order down with floors, and alert on how often each rung fires. Sweep one slice against a frozen eval set and see where its curve flattens. The first of those takes an afternoon, and it will tell you more about why your assistant's answers are the way they are than another week on the retriever.