Long-context decay and the filler control

Long-context decay is the fall in quality that comes from the length of a prompt as such, with the task and the evidence unchanged. It is the hardest of this topic's effects to believe on description and the easiest to demonstrate, and the demonstration matters because it changes what a team should build. A quality problem that comes from confusable retrieved passages is fixed by reranking and a smaller cutoff. A quality problem that comes from length is fixed by deleting tokens, and the two look identical from the outside — a fluent answer that is worse than it used to be.

The control that tells them apart is a piece of deliberate vandalism. Take requests that score well, keep the question and the evidence document byte-for-byte identical, and pad each request with text that has nothing to do with the task until it is many times longer. Retrieval cannot explain whatever happens next, because there is no retrieval; the evidence is present, complete and unchanged in every condition.

The reason it has to be demonstrated rather than asserted is that most teams have already satisfied themselves it cannot be happening to them. The context window — the ceiling on how many tokens a single request may contain — is the only length figure anybody is handed, it is exact, and crossing it produces an immediate error with the request rejected. A boundary that is published, precise and enforced looks like the boundary that matters, so a prompt running at a fraction of it reads as comfortably inside the safe region. The step being skipped is that the window is a constraint the serving stack enforces on what will be accepted, and quality is enforced by nothing at all. No component anywhere in the request path has the job of noticing that an answer got worse, so there is no error available to be raised, and the two boundaries sit tens of thousands of tokens apart with only one of them reported. Being inside the window is a statement about whether the request will run, and it licenses nothing about how well it will run.

A control that changes only one quantity#

The harness is deliberately blunt. Everything the task depends on is fixed, one variable moves, and the padding is inserted where it cannot displace the evidence.

FILLER = load_paragraphs("wikipedia_geology.txt")   # unrelated to contracts

def pad_to(prompt_core, target_tokens, tok):
    """Append filler after the evidence, before the question restatement."""
    out, n = [], count_tokens(prompt_core, tok)
    i = 0
    while n < target_tokens:
        out.append(FILLER[i % len(FILLER)])
        n += count_tokens(FILLER[i % len(FILLER)], tok)
        i += 1
    return prompt_core.replace("<<FILLER>>", "\n\n".join(out))

CORE = (
    "You are reviewing a contract. Extract the termination terms.\n\n"
    "<CONTRACT>\n{contract}\n</CONTRACT>\n\n"
    "<REFERENCE>\n<<FILLER>>\n</REFERENCE>\n\n"
    "Question: {question}\n"
)

for target in (1_000, 4_000, 16_000, 32_000, 64_000, 128_000):
    scores = []
    for item in short_docs:                      # the 0.84 population
        prompt = pad_to(CORE.format(**item), target, tok)
        scores.append(judge(ask(prompt, temperature=0), item.gold))
    print(target, sum(scores) / len(scores))

Three properties of that harness are what make its output evidence rather than an anecdote. The evidence document is unchanged in every condition, so any claim that "long documents are harder" cannot apply — it is the same document each time. The filler goes after the evidence and before the question, so the evidence stays at a constant absolute position from the start of the prompt and position bias is not confounded with length. And the filler is drawn from a subject with no lexical overlap with contracts, which separates this measurement from the distractor sweep earlier in this topic: those near-misses were topically confusable by construction, and these paragraphs are chosen so that no reasonable reader could mistake a description of sedimentary basins for a termination clause.

Those three properties are not a checklist of good laboratory manners. Each one is eliminating a specific rival explanation, and the elimination works only because the quantity that explanation depends on was held fixed while the score moved. That is the whole logic of a control, and it is worth having explicitly rather than by habit, because a reader who can rebuild it can design the next control and a reader who cannot will build a harness that measures something it did not intend to.

Start from the position before the experiment is run. A score has fallen, and there are at least four things in the room that could account for it: the documents got harder, the evidence moved to a worse place in the prompt, the material around the evidence became confusable with it, or the request simply got longer. Any of the four produces the same symptom, so the symptom rules out none of them. An experiment can remove a candidate from that list in exactly one way — by holding it constant across conditions. If a candidate cause did not change and the score did, that cause is not what changed the score.

Each design choice then removes one candidate, and it is worth naming which does which. The byte-identical evidence document removes "long documents are harder", because it is the same document being scored at every length. The fixed insertion point after the evidence removes position bias, because the evidence's distance from the start of the prompt never moves. The unrelated subject matter removes confusability, because nothing in a paragraph about sedimentary basins is competing to be read as a termination clause. That leaves one quantity that did change, so a score that falls has one explanation available to it. Drop any one of the three and the surviving explanation is no longer unique: the number is still a real number, and it can no longer tell you whether a sprint on the retriever is worth running.

Length alone is enough#

One run over 400 short agreements, judge score on the termination extraction, at six total context lengths:

The left column is the result that matters for planning, because it falls 19 points with the task and the evidence held constant and with padding that is transparently irrelevant. No retriever change can recover that, since there was nothing to retrieve; the only lever it responds to is making the prompt shorter.

The shape of the left column is worth reading as well as its endpoints. The loss between 1,000 and 16,000 tokens is 4.8 points, and the loss between 16,000 and 128,000 is 14.3, so the slope steepens rather than being linear in length; there is no threshold where quality falls off, but there is a region where it starts costing enough to notice. Where that region begins is specific to the model and the task, which means the useful output of this experiment is not a rule of thumb about a number of tokens but a curve for your own workload, plotted once and re-plotted when you change models. It also gives you a currency for design conversations that otherwise run on preference: adding a 20,000-token document of "helpful background" to every request has a price on this curve, and it can be compared against whatever the background was supposed to buy.

A steepening slope is a different kind of finding from a threshold, and which one you have decides how the number gets used. The rows of that table roughly double as they descend, so the honest way to read the slope is per doubling of the prompt rather than per token, and read that way each doubling costs more than the one before it: the step from 32,000 to 64,000 tokens loses more than the step from 16,000 to 32,000, and the step from 64,000 to 128,000 loses more than either.

That shape predicts something a threshold cannot, and the prediction is the form of the finding that survives a design conversation. The price of a block of text is not a property of the block. It is a property of the block together with the request it is being added to, so the same material carries two different prices, and the expensive one is on the request that was already long. Adding a 20,000-token document of background to a short request costs little; adding the identical document to a request that has already grown by accretion is charged at the steep end of the curve.

The right column is the same measurement with filler drawn from other contracts instead of geology, and the gap between the columns is the interference effect, now visible as a second, larger contribution stacked on top of the first. At 128,000 tokens, roughly 19 points of the 33-point total drop are attributable to length as such and 14 to the material being confusable. Both are real, they respond to different fixes, and a system that measures only its retriever will see neither.

Acting on a result like this is mostly an audit of what is in the prompt that nobody chose to put there. In a request that has grown to 60,000 tokens, the evidence itself is often a minority of them, and the rest is accumulated sediment: a system prompt — the standing block of instructions sent ahead of every request, which nobody re-reads — that four teams have appended to, JSON schemas for twelve tools of which two are ever called for this task, an entire conversation history retained because retention was the default, few-shot examples added during a debugging session in March and never removed, and appendices and signature blocks that carry no clause language. Each of those was individually defensible and none of them was ever measured against the slope in the table. Counting tokens by section — a fifteen-line script over your own request bodies — usually finds a third of the prompt that can be deleted without anyone noticing, and the table says what deleting it is worth.

The slope is what makes that audit worth an afternoon rather than an act of tidiness. Sediment in a request that has reached 60,000 tokens is not being charged the average price per token across the table; it is being charged the marginal price at the length the request has already reached, which is the steepest price on offer. The same third of the prompt, deleted from a short request, would be worth close to nothing. Deleting tokens pays best exactly where prompts have grown worst.

One further variation is worth running once, because it is cheap and it constrains the explanation. Pad with the same paragraph repeated to length rather than with varied prose, and the degradation is consistently milder — in this run, 0.802 at 128,000 tokens against 0.698 for varied filler. The prompt is the same length and the token count is identical, so whatever is happening is not purely a function of how many tokens the model has to process.

The step from that observation to a constraint on the explanation is short, and it is worth taking slowly, because it is the one place in this chapter where a measurement rules part of a mechanism out. Suppose the loss were entirely a matter of how a fixed quantity of attention gets divided over the positions in the prompt — attention being the operation that weighs every token already in front of the model when it decides what to produce next. The weights in that division are never negative and they sum to one across the prompt, so how the total gets shared out depends on how many positions there are and not on what any of them says. Two prompts of identical token count would then divide the same total the same way and would score the same. These two prompts have identical token counts. They do not score the same, 0.802 against 0.698 at the same length, so token count alone does not determine the loss and the content of the filler is doing some of the work.

What that rules out is the strongest version of that supposition, in which the number of tokens is the only variable. What it does not rule out is a divided budget as a contributing cause, because a milder fall is still a fall, and repeated text may be cheaper to carry than varied text for reasons that have nothing to do with how attention is shared. It is one constraint on a space of explanations rather than a verdict, which is the most a two-hour experiment should be asked to deliver.

Where the result comes from, and how far it has been pushed#

The framing of "hold the task fixed and grow the haystack" comes out of the needle-in-a-haystack tradition, which began as an informal evaluation in 2023 and was then taken apart methodically. Chroma's Context Rot work is the most systematic public version: it ran controlled variations across eighteen models, changing one factor at a time — the length of the haystack, whether the needle and the question shared wording, whether the surrounding text was topically related, whether the haystack was coherent prose or shuffled — and found performance falling with length in every model tested, with the size of the fall depending strongly on those other factors. Hamel Husain's write-up and Matt Palmer's review of the same research are the accessible summaries, and both make the same operational point: degradation is not a cliff at the context limit but a slope that starts far below it.

The mechanism is not settled, and it is worth being honest that this chapter measures a phenomenon rather than explains one. Three candidate accounts are in circulation. Attention — the operation by which a model weighs every token already in the prompt when deciding what to produce next — is a normalised distribution over positions, so a fixed budget spread over 128,000 positions leaves less of it for any given one. Positional encodings, the scheme that tells the model where in the sequence each token sits, since attention on its own has no notion of order, are trained at some length and extrapolated beyond it, and extrapolation quality varies with the scheme. And training data contains far fewer 100,000-token examples than 2,000-token ones, so long-context behaviour is less well covered by whatever the model learned to do. These are not mutually exclusive, none of them is decisively established, and the repeated-paragraph result above is mildly awkward for the simplest version of the first account. The practical upshot does not depend on which is right, because all three predict the same fix.

That last claim is worth working rather than asserting, since it is the reason an unsettled mechanism does not leave you with an unsettled decision. Each of the three is a chain, and each chain ends somewhere different. The dilution account claims attention is a fixed total shared over positions, so it predicts that an added token costs something regardless of what it says, that the cost grows with the number of positions, and that removing positions is the intervention that helps. The extrapolation account claims the scheme encoding position was fitted at one length and is being used past it, so it predicts a loss that tracks how far into the sequence a token sits rather than how many tokens accompany it, and it predicts that a model refitted at longer lengths improves with the prompt unchanged. The coverage account claims the model saw few examples this long, so it predicts that long-context work is simply less practised, that the failures look like a model doing an unfamiliar task rather than one running out of a resource, and that the remedy is training data.

Set those three predictions side by side and only one of them names something a team calling a hosted model can act on. Refitting the positional scheme and adding long training examples are both done by whoever trains the model, on their schedule, and neither is a lever inside your request. What all three agree on is that a shorter prompt scores better — fewer positions to divide a budget over, fewer positions past the fitted range, fewer tokens of an under-practised regime — which is why the curve above is worth plotting on your own workload whichever account turns out to be right. The instrument does not change when the explanation does.

Two ways to invalidate the run#

The experiment has one design trap that will destroy it if you miss it. If your filler happens to contain anything the judge rewards — a paragraph that mentions dates, or numbers formatted like durations — you are no longer measuring length, you are measuring a weak distractor, and the two columns of the table collapse towards each other. Reading a sample of the padded prompts before trusting the run is worth the ten minutes.

The other trap is caching: providers keep the processed form of a repeated prompt prefix for a short window and serve later requests that share it more cheaply and faster, so if your filler is identical across items, the cheap runs and the expensive runs will differ in latency in ways that have nothing to do with quality, which matters if you are also using the sweep to argue about cost.

What the control establishes is narrow, and the narrowness is the point. It rules out retrieval as the sole explanation for a specific measured drop. It does not say retrieval is unimportant — the right column of the table is retrieval's contribution and it is the larger one — and it does not say your particular quality problem is a length problem. It says that if you run this control and the score falls, then a sprint spent on the retriever will not recover those points. It is also not a statement about the context window limit: every prompt in the table fit, comfortably, with no truncation and no error, and the model would have accepted a longer one. The limit is where requests start failing; this is where answers start getting worse, and the two are tens of thousands of tokens apart.

Telling the three effects apart#

The effects this topic has now measured separately are easy to conflate when you are looking at one bad answer, and the diagnostic is a matter of which thing you hold constant.

Running that tree end to end on one query set is a day of work and it replaces a fortnight of hypothesis-swapping in standups.

A filler control is the cheapest experiment in this topic and the one that most often changes a plan, because it is the only one that can rule a whole class of work out. Everything else here compares configurations that are all plausible; this compares a system against itself with nothing changed but length, so a score that falls has exactly one available explanation. The output is a curve that belongs to your model and your task, and its practical use is as a price list: every block of text somebody proposes adding to every request has a cost on it, and the cost is now a number rather than a matter of taste.

There is one more claim you are likely to meet before you finish this work, and the padding experiment is what equips you to answer it. Somebody will point out that the model vendor publishes a chart showing near-perfect retrieval at a million tokens, and ask why your scores fall at 32,000. That chart is not wrong, and the difference between what it measures and what your evaluation measures is visible in the harness above — in the fact that the question asked for an extraction across a whole document rather than for the location of a planted string.