Context degradation

Read these first

• Context window

What you will know

• Build a position-bias probe that places identical evidence at the beginning, middle, and end of prompts, then compare answer accuracy across positions.

  A U-shaped result is common, but the shape and severity vary by model, prompt format, and context length.

• Demonstrate quality dropping as irrelevant context is added

  A fixed task with increasing distractors exposes whether the model starts selecting plausible but unsupported evidence.

• Measure answer quality as retrieved chunk count increases and choose a retrieval cutoff from a held-out evaluation.

  The best cutoff balances missed evidence against interference, and it often changes with query type and chunk ordering.

• Decide where in a prompt to put the thing that must not be missed

  Placement is model- and template-dependent, so the choice should be verified with the same prompt structure used in production.

• Reproduce long-context decay by holding the task and relevant evidence fixed while increasing total context length.

  A falling score with stable relevant evidence indicates capacity or attention degradation rather than a retrieval failure.

Chapters

1. 1

   Position bias in long prompts

   Identical evidence is used more reliably at the start and the end of a long prompt than in the middle. This chapter defines position bias, builds the probe that measures it — one answer-bearing passage moved across positions with the token count held constant — reads the U-shaped curve it produces, and shows why the depth of that curve belongs to your model, your prompt template and your context length rather than to the literature.

   Read chapter
2. 2

   Retrievers, rerankers and recall

   The rest of this topic manipulates a retriever's output, so the object needs naming: a document store is split into chunks, a first-stage search scores them against the query and returns the top n in rank order, and the top k are pasted into the prompt. Two numbers then have to be held apart — recall, which asks whether the answer-bearing chunk appears in that list at all, and answer accuracy, which asks whether the final response is right. A reranker is a second, more expensive scoring pass over the first-stage candidates that raises precision at the top of the list, at a per-query cost proportional to how many candidates it is given.

   Read chapter
3. 3

   Distractors in the retrieval tail

   A retriever set to return twenty chunks does not become safer at fifty. Ranks 21 to 50 are the passages that resemble the answer most closely without being it, and a model reading them writes fluent answers from the wrong one. Holding one task fixed and adding distractors in steps shows answer accuracy falling while the correct chunk is still present and first — recall and accuracy moving in opposite directions on the same change.

   Read chapter
4. 4

   Choosing a chunk cutoff

   Once interference from near-miss chunks is established, the number of chunks in a prompt becomes a tuning parameter with a real optimum: too few and the evidence is missing, too many and it is drowned. Sweep that number on a held-out evaluation reporting recall and answer accuracy at every point, split the result by query type, and note that the best cutoff moves again when chunk ordering changes. The sequence that wins is reranking for precision and then trimming, rather than widening the prompt.

   Read chapter
5. 5

   Long-context decay and the filler control

   Something degrades beyond interference from near-miss chunks. Hold the question and the relevant evidence byte-for-byte constant and pad the prompt with transparently irrelevant filler until it is long, and the score falls anyway — 19 points between 1,000 and 128,000 tokens on one contract-extraction task. Because nothing about retrieval changed, the control isolates a length effect and points the fix at shortening the prompt rather than improving the retriever, which is a distinction that decides where a sprint goes.

   Read chapter
6. 6

   Needle-in-a-haystack and multi-fact integration

   A vendor chart showing 100% retrieval at one million tokens and an in-house summarisation eval decaying from 60,000 tokens can both be honest measurements of the same model. The needle test asks for one planted verbatim string, which is close to search; a production task asks the model to combine claims scattered across the context, and that capability falls an order of magnitude faster. This chapter gives you the two prompts side by side, the numbers that belong on the same axes, and a two-needle variant that isolates integration load in an afternoon.

   Read chapter
7. 7

   Instruction placement and compliance controls

   An instruction has no gold answer, so the probe that locates its best position scores a compliance rate over realistic requests instead of accuracy against a key: the same instruction moved through the slots of a real 30,000-token system prompt, at production temperature, checked by an exact predicate. Two models put their best slot in different places, so the answer expires when the model does. And whatever rate the probe reports, a categorical requirement needs a control outside the model — a validator in your own code that repairs or appends, with a counter on the branch.

   Read chapter