Distractors in the retrieval tail

A distractor, in this topic, is a passage that arrives in a retriever's results, looks like it answers the question, and does not. The word is doing specific work: these are not random or off-topic passages, they are near-misses, and they are near-misses precisely because a search engine ranked them highly against the same query the right passage was ranked highly against. The previous chapter established that recall — whether the answer-bearing chunk appears in the list at all — can only rise when the list gets longer. This chapter is about the quantity that moves the other way.

The intuition that says widening is safe is worth stating in its strongest form, because it is not stupid. A human researcher handed thirty extra documents skims them, notices that most are irrelevant, and discards them at some small cost in time; the marginal document is nearly free and occasionally decisive, so more is weakly better. Applied to a model, that reasoning assumes irrelevant context is inert. It is not, and the reason has to do with what is actually sitting at ranks 21 to 50.

The researcher is a comparison rather than a mechanism, and the place it breaks is the part worth keeping. A person handed thirty extra documents performs two separable acts: they discard the ones that do not apply, and then they answer from what survives. The discard is cheap, it is reversible, and it happens before the answering, so any document that gets as far as being read has already been vouched for by a step that cost almost nothing. A model has no separate discard step. Deciding which passage bears on the question is not a stage that runs before the task; on a question with one right chunk, it is very nearly the whole of the task.

That is why the marginal document is not free. It is not an item added to a pile that will be sorted later — it is another candidate in the one operation the accuracy number is measuring. Widening a retrieval list does not add optional material to a prompt. It adds entrants to a selection problem, and it adds them at the point in the ranking where they are least distinguishable from the right answer.

Nothing in the tail is random#

Take a query against a company's internal policy documents: what is the notice period for a contractor in Ireland? Rank 2 is the current Irish contractor policy, which answers it. Ranks 21 to 50 include the 2021 version of the same page, superseded but never deleted; the Irish policy for permanent employees, which has a different number; the contractor policy for the UK entity, which has another different number; a benefits FAQ that quotes the notice period in a worked example from three years ago; and a slide from an onboarding deck that paraphrases all of it loosely. Every one of those chunks contains the words "contractor", "Ireland" or "notice period", a plausible-looking duration, and no marker that says "this is the wrong one". That is precisely what earned them their rank.

It is worth naming those categories, because they suggest fixes that have nothing to do with the model. Superseded revisions of the same page are a lifecycle problem: the index contains documents that the source system has already replaced, and an ingestion pipeline that carried an effective_to field could filter them at query time. Sibling documents that differ by one dimension — entity, region, product tier, employment type — are a filtering problem, solvable when the user's context is known and the chunks carry the matching metadata. Loose paraphrases in decks and FAQs are a hygiene problem in the document collection itself, and often the right answer is to stop indexing the deck. These matter here because they put a ceiling on what any amount of chunk-count tuning can achieve: if half of your near-misses are stale copies, the durable fix is upstream of everything in this topic.

That example is one document collection and one query, and the reason it generalises lies in the ordering rather than in anything about policy documents. A first-stage retriever scores every chunk in the index against the query and returns them in descending order of that score. Descending order has a property worth stating slowly, because the rest of the chapter rests on it: the chunk at rank 21 scored higher than the one at rank 22, and it scored only slightly lower than the one at rank 20. Nothing falls off a cliff at the cutoff, because the cutoff is a number somebody chose and the scores know nothing about it.

The answer-bearing chunk earned its own place in that same ordering, under the same scoring function, against the same query. So whatever sits immediately below the cutoff is, by the ordering alone, nearly as similar to the query as the passage that answers it. That is the first causal step, and it is a claim about ranked lists rather than about this collection: a cutoff drawn anywhere in a ranked list has, directly beneath it, the material most similar to the material directly above it.

The second step is what the model is left holding. Similarity to the query is not only how the passages were ranked — it is also close to the only cue available for deciding which of them bears on the question. The facts that would settle it, such as which of two Irish contractor policies is the live one, are not written anywhere in the passages themselves. So the retriever sorted on similarity and then handed the model a selection problem to be made on something very close to similarity, with every tie-breaker withheld.

Together those two steps say something stronger than that widening is risky. They say that widening adds, first and preferentially, the material that is hardest to reject, because "ranked just below the cutoff" and "hardest to reject" are two descriptions of one property. The chain also makes a prediction the rest of this chapter can be read against: if the damage comes from confusability rather than from bulk, then replacing those near-misses with text on an unrelated subject should flatten the effect substantially while leaving the token count where it was.

A model reading the assembled prompt has to decide which of six passages that all look like answers is the answer. Nothing in the input makes the distinction easy, because the distinguishing information — which page is current, which entity the user belongs to — is metadata that lives outside the chunk text. The failure this produces is not "the model got confused and said it did not know". It is a fluent, confident answer citing the 2021 figure.

Holding everything fixed except the distractors#

Comparing two production configurations changes several things at once, since going from 20 to 50 chunks changes the length of the prompt, the position of the gold chunk within it, and the number of distractors together. To attribute a drop you need a sweep that moves one of those.

def build(question, gold, distractors, j, rng):
    """Gold is always present and always at slot 0.
       j distractors are drawn from ranks 21-50 of the same query."""
    docs = [gold] + rng.sample(distractors, j)
    body = "\n\n".join(f"[{i}] {d['text']}" for i, d in enumerate(docs))
    return f"{INSTRUCTIONS}\n\n{body}\n\nQuestion: {question}"

def sweep(items, js=(0, 4, 8, 16, 32)):
    rng = random.Random(17)
    for j in js:
        outcomes = [classify(ask(build(it["question"], it["gold"],
                                       it["tail"], j, rng)), it)
                    for it in items]
        yield j, tally(outcomes)

def classify(response, item):
    if abstained(response):            return "abstain"
    if item["answer"] in response:     return "correct"
    if any(d["answer"] in response for d in item["tail"]): return "distracted"
    return "other_wrong"

Pinning the gold chunk at slot 0 takes position out of the experiment, so whatever moves is attributable to the presence of the other chunks rather than to where the answer sits. The classify function is the part worth copying, because a binary correct/incorrect score hides the mechanism. Splitting wrong answers into distracted — the model gave an answer that appears verbatim in one of the distractor chunks — and other_wrong tells you whether adding context is causing the model to select bad evidence or merely to lose the thread.

One run over 640 policy questions, gold always present at slot 0:

Recall is 100% in every row of that table by construction. The answer-bearing chunk is present, first, and unmodified in all 640 prompts at every step, and accuracy still falls by 17.8 points. The other_wrong column barely moves, so the loss is almost entirely the distracted column: the model is not failing to find evidence, it is choosing different evidence. The abstention column moving in the wrong direction is the detail that makes this expensive in production, because as distractors accumulate the model becomes less likely to say it does not know. More plausible-looking material makes an unfounded answer feel better supported.

The direction that column moves in is the opposite of what a careful reader predicts, and the explanation is the one already running underneath the rest of the chapter. Abstaining is not a separate faculty the model consults when its evidence runs out. It is a continuation like any other: the model declines when a sentence declining to answer is the likely thing to write given everything in front of it. What makes that continuation likely is a prompt that looks like a prompt with no answer in it — few passages, off-topic passages, passages that do not resemble the question being asked.

Adding near-misses changes precisely that. A prompt carrying thirty-two on-topic, answer-shaped passages about contractor notice periods does not look like a prompt whose answer is missing; it looks like a prompt whose answer is somewhere inside it. The same material that makes the selection harder therefore makes the admission of failure less likely, and the table shows both halves of that at once: the error rate climbs while the abstention rate falls.

The production consequence is worth stating without softening, because the two failures are not equally expensive. An abstention is the cheap one — the user learns nothing, asks again, or falls through to a human. A distracted answer is the expensive one, because it is a specific, sourced-looking claim that somebody acts on before anyone checks it. Widening the retrieval list trades the cheap failure for the expensive one, and it does the trade silently, since the abstention rate is not a number most teams have on a dashboard and its fall looks from the outside like a system that has become more helpful.

The other columns of that table are the ones a finance conversation runs on. Going from 0 to 32 distractors takes the prompt from 1,050 to 12,300 tokens, which is an eleven-fold increase in input cost per query and, on a workload of 400,000 queries a month, the difference between roughly 420 million and 4.9 billion input tokens. At a representative $3 per million input tokens that is about $1,300 a month against $14,700, for a configuration that scores 17.8 points worse.

Each step of that calculation is permitted for a reason, and the reasons are what let you redo it on your own numbers rather than borrow the result. Input tokens are what a provider bills for on the input side, so the input cost of one query is its token count multiplied by a rate per token. The answers in this sweep are short and of much the same length in every row, so output cost is very nearly constant across the conditions and drops out of the comparison entirely. That is what licenses treating the ratio of the two monthly bills as the ratio of the two prompt lengths and nothing else.

The multiplication then runs in two steps. Four hundred thousand queries a month, each carrying 1,050 input tokens, is four hundred thousand lots of 1,050 tokens, which comes to roughly 420 million input tokens in the month. Those tokens are priced in millions, so the bill is the number of millions multiplied by the rate: 420 of them at $3 each is about $1,300. The second configuration is the same arithmetic with a longer prompt, and because the prompt grew by a factor of about eleven while every other term stayed put, the bill grows by that same factor to about $14,700.

Two conditions belong on that figure, since a finance conversation will find them anyway. It counts input tokens only, which makes it a lower bound on the difference rather than the whole of it. And it assumes nothing is cached — where a provider retains the processed form of a repeated prompt prefix and charges less for it, a fixed instruction block is cheaper than this arithmetic implies, while the retrieved chunks, which differ on every query, are not.

Prompt length also sets time-to-first-token, the delay between the request arriving and the first word of the answer appearing, since the model has to read the whole input before it writes anything; the same change that lowers accuracy therefore raises latency as well. It is unusual to find a parameter where the cheap setting and the accurate setting are the same setting, and this is one.

The effect is not an artefact of one collection of documents. The Chroma "Context Rot" work ran a related design across eighteen models — a fixed needle-and-question pair with topically related distractors added — and found degradation in every one, with the size varying by model and the direction never reversing. Swap the distractors for genuinely unrelated text, as a later chapter of this topic does deliberately, and the curve flattens considerably; the damage is a function of how confusable the added material is, not of how much of it there is.

Where a smaller list costs you#

None of this licenses "fewer chunks is always better". The sweep above holds the gold chunk present by fiat, which is exactly the assumption a real retriever cannot make. Cutting from 20 to 5 in production drops recall as well as distractors, and on a document collection where recall@5 is 74% and recall@20 is 91% the trade can easily go the other way. The claim is narrower and more useful: chunk count has an optimum rather than a monotone benefit, and the belief that widening is free is wrong.

It also does not apply uniformly across query types. Questions that need one fact from one place suffer most from distractors, because there is exactly one right chunk and every other one is competition. Questions that require assembling several facts — how many incidents were logged in Q3 across all regions, what changed between the two versions of this contract — need many chunks by their nature, and starving them is its own failure. A single global cutoff is being asked to serve both, which is a decision the next chapter makes explicit rather than accidental.

And this is a different effect from the position bias measured by moving a fixed passage across a prompt. There, the content was constant and only the location of one passage moved. Here the location is constant and the content grows. Both reduce accuracy, they have different fixes, and a team that has only measured one of them will apply the wrong one; reordering chunks does nothing about a distracted answer, and trimming the list does nothing about a passage that is being ignored at rank 12 of 12.

The tail of a retriever's ranking is the most confusable material in the whole document collection with respect to the query that produced it, which is what being ranked highly means. Pasting it into a prompt therefore buys a little recall on a few queries and pays for it with interference on all of them, and the sweep above prices that trade at 17.8 points for 32 extra chunks on one realistic workload. The measurement is cheap enough to redo on your own data, and it is worth redoing, because the optimum is a property of how much near-duplicate material your collection contains rather than of retrieval in general.

There is a version of the wider-retrieval proposal that survives contact with that measurement, and it is the one the next chapter develops. If ranks 21 to 50 sometimes contain the answer, the thing to do with them is score them properly rather than either discarding them unseen or pasting them in wholesale. Retrieving 50 candidates and putting 50 chunks in the prompt are separable decisions, and once they are separated the question becomes what number belongs in the prompt, and how you find it on data you did not tune on.