The window is a running total, not a per-request allowance

The ticket was ordinary: add VAT handling to the invoice builder in a 1.2-million-line Python monorepo, a single repository holding every service and shared package the company ships. The agent's first attempt compiled and failed two tests, its second attempt failed one, and its third attempt produced this.

attempt 3/4  model=coding-agent-v2  input_tokens=198,412  max_tokens=1,588
--- a/billing/invoice.py
+++ b/billing/invoice.py
@@ -212,6 +212,34 @@ class InvoiceBuilder:
     def _apply_vat(self, lines: list[Line], jurisdiction: str) -> list[Line
[stop_reason: max_tokens]

The patch ends inside a type annotation. Nothing about the model's grasp of VAT produced that; the request went out with 198,412 tokens of input against a 200,000-token hard limit, which left 1,588 tokens for the reply, and a patch of that shape needs about 6,000. The harness had done exactly what it was configured to do. Somewhere in the context builder was a line that read max_context_tokens: 180000, added by someone who reasoned that a 200k window minus a generous 20k for the answer leaves 180k for repository code, and that a bigger window ought to be used rather than left empty.

Two terms in that trace carry the whole of this topic, so they are worth stating plainly before the argument leans on them. A token is the unit a model reads and writes, roughly three quarters of an English word or a few characters of source code, and a model's context window is the maximum number of tokens it can have in front of it on a single call, which is 200,000 for the model above. The window is the entirety of what the model knows about the repository at the moment it answers, so a file that is not in it might as well not exist, because the model has no way to go and look on its own. A 1.2-million-line codebase is somewhere around 12 million tokens, which means that on any given call the model can see well under one percent of the code it is being asked to change.

Something has to choose what that one percent is, and the choosing is context selection over a codebase: deciding which files from a repository enter the window for a particular task, how much of each — the whole file, a single function, a list of signatures with the bodies left out — and in what order, all of it under a token budget. You meet it as the context builder, the part of the harness — the ordinary program wrapped around the model that calls it, executes the tools it asks for, and feeds the results back — that assembles the text of every request before it goes out. Every coding agent has a context builder, whether it was designed or accumulated, and the one in this story is a retrieval step, a sort, and a line that reads max_context_tokens: 180000.

That reasoning holds for exactly one request. A coding agent does not make one request; it makes a series of them, and every turn appends to the same transcript, the running record of instructions, model messages and tool results that is re-sent in full on each call. The 180k figure treats the window as an allowance renewed on each call, when the thing it actually governs is a running total that only goes up until something is deliberately removed.

Six occupants, one window#

It helps to write down everything that has to fit, because the retrieved code is the only occupant most builders account for and it is not the largest one by the end of a run.

The last row is the one that produces the failure above, because output room is not a separate pool. On most providers the limit is on the sum: whatever the input occupies, the reply has to fit in what is left. So the constraint is not "input ≤ 200k" but "input + the patch we are about to write ≤ 200k", evaluated at the last attempt, when input is at its largest.

The fourth and fifth rows are where the arithmetic goes wrong. A failing run of a targeted test suite in this repository emits roughly 18k tokens — pytest prints the assertion diff, the captured stdout of the fixture setup — the helper code that builds each test's data — and a traceback per failure, and three failures at 6k apiece is unremarkable. The loop allows four attempts. Each attempt appends its diff so the model can see what it already tried, and appends the fresh diagnostics. That is about 24k per attempt, and four attempts is 96k of material that did not exist when the builder chose its fill and cannot be predicted from anything in the repository.

The ledger for a 180k fill#

Walking the run turn by turn shows where it breaks and, more usefully, in what order. The harness here has a trim step that keeps the transcript under the limit by truncating the oldest evictable block, and the retrieved file blocks are pinned, so the only thing it can cut is tool output.

Two distinct failures are visible here, and the mid-function truncation is the second of them. The first is that from turn 2 onward the model is reading a truncated tail of its own test output. A pytest tail is the summary line — 2 failed, 41 passed — and the tracebacks that say why are at the head of the block, which is the part that got cut. So attempt 2 is written by a model that knows it failed and not what failed, which is a worse position than attempt 1 was in. The run degrades before it breaks, and the log line that finally gets reported is the truncated patch, three turns downstream of the cause.

This is also why the failure looks intermittent. A task whose first attempt passes never reaches turn 2, so the same configuration ships green for weeks on small tickets and falls over on exactly the tasks that need more than one iteration — which are the tasks the agent was bought for.

Deriving the ceiling from the loop#

The cap belongs on the first turn's fill, and it is derived by subtracting everything the loop will add from the window, not by subtracting a guess about the answer. Written as arithmetic with the numbers above:

WINDOW = 200_000

FIXED_PREFIX      = 9_000    # system prompt + tool definitions, measured
TASK_STATEMENT    = 1_000    # ticket text + acceptance criteria
PATCH_TOKENS      = 6_000    # p50 of merged patches over the last 200 runs
DIAGNOSTIC_TOKENS = 18_000   # p50 of a failing targeted suite
MAX_ATTEMPTS      = 4

def first_turn_ceiling(diagnostics: int) -> int:
    per_attempt = PATCH_TOKENS + diagnostics
    loop_growth = per_attempt * MAX_ATTEMPTS
    return WINDOW - FIXED_PREFIX - TASK_STATEMENT - loop_growth

first_turn_ceiling(DIAGNOSTIC_TOKENS)   # -> 94_000
first_turn_ceiling(24_000)              # -> 70_000  (p90 diagnostics)

The p50 line — p50 being the median, the size half the observed runs come in under — gives 94k, which is already about half of the 180k the builder was using. The line to ship is the second one. Test output is heavy-tailed in a way patch size is not: a single flaky integration test that dumps a request log, or a type checker that reports a cascade of 200 errors from one bad signature, turns an 18k block into a 40k one. Sizing the fill against the median means the budget holds on the runs that succeed anyway and fails on the runs that iterate. Taking the 90th percentile of observed diagnostics — 24k in this repository, measured over the same 200 runs the patch figure came from — puts the ceiling at 70k, or 35% of the window.

Thirty-five percent feels wasteful the first time you write it down, and that reaction is worth sitting with, because it is the reaction that produced the 180k. The window is not being wasted; it is being reserved for material that has not been generated yet. An engineer sizing a request queue does not fill it to capacity on the grounds that empty slots are inefficient, and the reasoning is the same here: the headroom is what absorbs the variance.

It is worth knowing where the 20k intuition comes from, because it is not stupid. It is the correct reserve for a chat completion, where one request produces one answer and the transcript is discarded, and every engineer who has sized a prompt has sized one of those. A coding agent looks like the same shape from the outside — a request goes out, text comes back — and the difference is entirely in what happens next, when the harness appends the result and calls again with everything. The habit transfers because the interface is identical and the failure it causes appears three turns later than the code that caused it.

Two of these constants should be measured rather than assumed, and both are cheap to measure. The fixed prefix is one call to the tokenizer against the rendered system prompt and the serialised tool schemas, and it drifts upward every time someone adds a tool — a team that added four MCP servers over a quarter — Model Context Protocol servers, each publishing a set of tools whose schemas land in that same prefix — found theirs at 21k, not the 9k the budget still assumed. The patch and diagnostic figures come from the run log, as percentiles, which means the budget has to be recomputed when the repository changes rather than set once in review.

Getting those percentiles requires the harness to record a token ledger per turn — a log line carrying the input size, the output size, and the delta attributed to each block that was appended — which is a dozen lines of instrumentation and the only way to know whether the budget you wrote down is the budget you are running. Teams that skip it debug the symptom in the opening trace, where the visible artifact is a malformed patch and the cause is three turns and one configuration file away.

One ceiling per task class#

A single ceiling for every task is a simplification worth being explicit about, because MAX_ATTEMPTS is the term with the most leverage in the arithmetic and it is not really a constant. A one-line configuration change that the agent verifies with a single unit test converges on attempt one and rarely runs the full suite, so its loop growth is closer to 24k in total than to 96k. A cross-cutting refactor across nine files runs everything, generates 40k of output when it fails, and uses every attempt it is given. Applying the refactor's budget to the one-line change throws away 70k of window that could have held the calling code, and applying the one-line change's budget to the refactor reproduces the failure this chapter opened with.

Routing on the task's declared scope handles most of it, since the ceiling is a function of the attempt limit and the expected diagnostic size for the suite the task will run, and the harness knows both before the first request. Where the classification is wrong — a task that looked small and turns out to need six attempts — the run degrades gracefully only if the harness can recover space that has already been spent. There is also a class of provider where the output limit is separate from the context limit rather than carved out of it, and on those the mid-function truncation in the opening trace does not happen; the transcript overflow still does, and it arrives as a hard error at the API boundary instead of as a half-written function, which is the better of the two ways to fail.

What the cap does not decide#

A ceiling on the first turn's fill answers how much repository code may enter and nothing else. It does not say which files, which is the subject of most of this topic, and it does not improve a bad selection — 70k of the wrong files is a worse run than 70k of the right ones, and both fit. Nothing in the arithmetic above prefers a relevant file to an irrelevant one, because the arithmetic never looks at content.

It is also not the prompt allocation you worked through in the earlier context engineering material. That allocation splits a single request between instructions, evidence and the answer, and it balances once, at assembly time. This budget has a time axis: the same window is re-served on every turn with more in it than the turn before, so the quantity being controlled is the trajectory rather than a snapshot. A builder can get the single-request split perfectly right and still fail on attempt three, which is what the 180k configuration was.

The obvious follow-on is that the loop growth in the arithmetic is an estimate, and estimates get exceeded. If the type checker returns 60k instead of the 24k budgeted, the ceiling on the first turn's fill has already been spent and cannot be renegotiated; the only lever left is deciding what to remove from a transcript that has overflowed. Retrieval scores cannot rank test output, superseded diffs or a stale copy of a file the agent has since edited, so the removal step needs a rule of its own — which is the other half of holding a budget, and the subject of the next chapter.