A corpus gap is not a retrieval miss, and no reranker closes it

Two quite different faults put the same thing on a user's screen: a wrong answer, with the paragraph that would have answered the question nowhere in sight. In one of them that paragraph is sitting in a searchable index and the search put it too far down the list to be used. In the other the paragraph is not in an index at all — nothing ever crawled it, or a parser handed back an empty string for it, or a reconciliation job deleted it — and no search setting returns a thing that is not there. The first is a retrieval miss. The second is a corpus gap. From the outside the two produce identical symptoms: the right passage was not in the prompt and the answer was wrong. This is the distinction that decides where a team spends the next two weeks, and it is the one most easily collapsed. Treating the second as the first sends people to tune search parameters against a question that will fail at every value they try, and after enough of those the conclusion drawn is that the retriever is fundamentally broken.

Telling them apart needs a labelled set: a fixed list of questions, each stored with the identifier of the chunk that contains its answer and a copy of that chunk's text. That identifier is the question's gold chunk, and it is what makes the first check possible at all, because without a recorded answer location there is nothing to look up. The other three checks read fields the previous chapters already emit — the ranks and chunk identifiers from the answer object, the archived rendered prompt, and the grade. Nothing below requires new instrumentation. It requires reading what is already emitted in a particular order.

Correctness is a product, and each factor has its own zero#

An answer is correct only when four separate things have all happened. The paragraph exists in the index. The retriever returned it inside the k results the request asked for, k being however many passages the search is configured to hand back. It survived into the prompt that was actually rendered. And the model used it rather than ignoring it. These are not four ways of being right. They are four conditions that all have to hold at once, which makes end-to-end correctness a product of four factors rather than a sum of four contributions.

The difference between a product and a sum is the whole of the diagnosis. In a sum, a small term can be compensated by a large one, so pushing hard on any term improves the total. In a product, one factor at zero sends the product to zero no matter how large the other three get. Doubling a factor that is already at its maximum multiplies the total by one. So the first question about any wrong answer is which factor is at zero, and the second question is whether the fix under discussion touches that factor or a different one.

That is why the instinct to add a reranker deserves to be derived rather than argued about. A reranker is a function from a candidate list to a candidate list: it takes the passages the first search stage handed it, scores each one against the query with a slower and more careful model, and returns the same passages in a different order. Its output is a permutation of its input. A permutation cannot contain an element that its input did not contain, because reordering a list never lengthens it. So any passage a reranker can lift to rank 1 was already somewhere in the list it received. A corpus gap means the passage is in no list the retriever can produce, at any k, including the list you would get by asking the index for every chunk it holds. Reranking a corpus gap therefore adjusts the ordering factor, which was never the problem, and leaves the existence factor at zero.

As an analogy, and it understates the problem: reranking a corpus gap is like re-sorting a page of search results that has no matching row on it. The analogy breaks in the direction that matters, because a results page at least reports that it found nothing, whereas a retrieval stage returns k passages whatever happens. All k will be on roughly the right topic, none of them will contain the answer, and the model will write a fluent paragraph out of them. The absence announces itself nowhere.

Each branch of that tree tests exactly one of the four factors, in the order the factors occur, and each branch corresponds to a stage from the first chapter's chain. The fields it reads are what the arrows in that chain were already carrying. Testing them in order matters: a passage cannot be badly ranked before it exists, and cannot be ignored by a model that never received it, so an earlier factor at zero makes every later question meaningless. The check is cheap enough to run over a whole labelled set on every evaluation, which converts a pass-rate drop into a distribution across four buckets before anybody forms a theory.

Four places, four fields#

A failure lives in exactly one of four places, and one field per stage separates each from its neighbour. Written out, the tree above is a function short enough to read in one sitting.

def triage(rec: LabelRecord, ans: Answer, graded: bool) -> str:
    if not index.exists(rec.gold_chunk_id):
        return "corpus_gap"

    rank = ans.ranks.get(rec.gold_chunk_id)
    if rank is None:
        deep = retrieve(embed(rec.question), rec.principal, k=50)
        deep_rank = next((i + 1 for i, h in enumerate(deep)
                          if h.payload["chunk_id"] == rec.gold_chunk_id), None)
        return f"retrieval_miss(deep_rank={deep_rank})"

    if normalise(rec.gold_text) not in normalise(ans.rendered_prompt):
        return "context_problem"

    return "passed" if graded else "generation_error"

The early returns are what make the four buckets mutually exclusive: a record that leaves on the first line never reaches the second test, so no question is counted twice and no question escapes uncounted. That is a property of the control flow rather than of the data, and it is what licenses reading the four counts as a partition of the failures later on.

The deep_rank probe inside the retrieval branch is small and pays for itself. A gold chunk that comes back at rank 12 on a top-10 search is a different problem from one that is not in the top 200: the first is a cut-off or ordering problem that a reranker over a wider candidate set genuinely fixes, and the second means the query and the passage are far apart in the embedding space, which reranking cannot help because the passage never reaches the reranker. Recording the deep rank turns "retrieval failed" into a number that says which of those two it is.

The useful reading of those two numbers is the gap between them rather than either level. Rank 12 against a cut-off of 10 is a miss by two places, and two places is inside the range a second scoring pass routinely moves things. Rank beyond 200 is not a near miss at all; it says the query vector and the passage vector are nowhere near each other, and a reranker over the top 50 will never be handed the passage to reorder. The same "retrieval miss" label covers both, and the deep rank is the only field in the record that separates a cheap fix from an embedding or query problem.

Read the last two columns across the rows rather than down them, and the table closes on itself. Every technique in the fourth column appears in the third column of some other row. Reranking, hybrid search and a larger k do nothing for a corpus gap and are exactly what works for a retrieval miss. Prompt edits and a larger model do nothing for a retrieval miss and are exactly what works for a generation error. Retriever tuning does nothing for a context problem and is, again, what works one row above. The fourth row states the pattern in full generality by naming a whole region of the pipeline rather than a technique. Nothing in the fourth column is a bad idea; each entry is a good fix aimed at the wrong factor, which is why the diagnosis is the work and the repair usually is not.

One row breaks the direction of that pattern, and the break is the informative part. For a corpus gap and a retrieval miss the useless fixes lie downstream of the fault; for a context problem the useless fix lies upstream of it. A context problem is the only bucket where the evidence was found and then lost afterwards, so the retrieval numbers all look healthy and tuning the stage that already succeeded is the tempting move. Teams do not expect this row to be populated and it frequently is. A chunk retrieved at rank 4 can be dropped before rendering by a prompt builder that truncates to a token budget, by a deduplication rule that collapses near-identical passages and discards the one that had the specific number, or by a re-ordering step that puts the highest-scoring chunk last where the truncation then cuts from the end. The archived prompt is the arbiter, and if you are not archiving the rendered prompt, this bucket is invisible and its failures get filed as generation errors, sending people to rewrite instructions for a model that never received the evidence.

Which contract a gap broke#

Corpus gaps decompose further, and the sub-type points at which contract from the first chapter failed. The document may never have been crawled, because it lives in a space, bucket or table the ingest job does not enumerate. It may have been crawled and parsed to empty or near-empty text, which is the sub-type the case worked below turns out to be and the one the mean-characters-per-document counter catches. It may have been parsed and then chunked in a way that dissolved the paragraph — a table flattened into a run of numbers with no header, a heading separated from the text beneath it. Or it may have been indexed correctly and then deleted, either by the source or by a reconciliation job acting on a truncated crawl. Four sub-types, four different owners, and the ingest run summary distinguishes them in about a minute: docs_listed for the first, mean_chars_per_doc for the second, the chunk record for the third, chunks_deleted for the fourth.

Those four fields are worth reading as a set rather than as four separate alarms, because each one goes wrong in a way the others cannot see. A crawl that enumerates half the source leaves docs_listed short while every document it did fetch parses perfectly. An extractor that returns empty strings leaves docs_listed untouched and drags mean_chars_per_doc down. A chunker that splits a paragraph across a boundary leaves both of those healthy and only shows up in the chunk record. A reconciliation job acting on a truncated crawl leaves all three healthy and shows up as a spike in chunks_deleted. A single summary line carrying all four separates the owner in one reading.

The four counts as a distribution#

Running the triage as a distribution rather than as a per-question tool is what makes it an early-warning system. Every evaluation run produces four bucket counts, and the counts have a stable baseline — a healthy system on this corpus sat at roughly 3% corpus gap, 12% retrieval miss, 1% context problem and 6% generation error.

Those four figures sum to twenty-two, and their complement is 78%, which is what the same system's pass rate reads on a good night. Two instruments landing on the same number is not a coincidence here, and it is worth being clear about why it has to happen. The early returns in the triage function send every question to exactly one bucket, and a question that reaches the end of the function without returning a failure label passed. So the four failure buckets and the pass rate partition the labelled set with no residual and no overlap. The agreement is exact at the rounding the figures are quoted to, and what it licenses is the thing the rest of this chapter rests on: there is no fifth cause. A drop in pass rate has to show up as a movement in one or more of the four counts, because there is nowhere else for it to go.

That is a far more specific signal than the pass rate itself, and each bucket is worth alerting on separately. Corpus gap exceeding 5% is an ingest incident regardless of what the pass rate did, because it means documents are disappearing and the labelled set is only seeing the fraction of them it happens to have questions about.

Reading the standard cases#

Take the quiz shape directly. A question whose gold paragraph returned at rank 2 of 5, with the archived prompt containing that paragraph in full, and a wrong answer. Ingest indexed it, retrieval ranked it well, augment rendered it, and the model still produced something wrong, so the failure belongs to generation and every earlier stage did its job on this question. Adding a reranker here would move a chunk that is already at rank 2 to rank 1 and change nothing. The productive moves are on the prompt or the model: requiring a citation per claim, so an unsupported sentence becomes visible; stating the precedence rule when two retrieved passages conflict; or noticing that the passage answers the question through a negation or a qualifier that the model flattened.

Now the two failures from an overnight drop, one where the gold paragraph is not in the index at all and one where it is in the index and came back at rank 40 on a top-10 search. The first is a corpus gap and the second is a retrieval miss, and they route to different teams with different work. The gap is an ingest question — was the document ever crawled, did the parser return text, did the chunker produce a chunk containing this paragraph, did reconciliation delete it — and it is answered by looking at the ingest run, not by touching search. The rank-40 case is a ranking question, and rank 40 on a corpus where the gold chunk is findable at all is the exact profile that a cross-encoder over the top 50 addresses, since Reranking established that a reranker can only reorder what the first stage hands it.

If you could only have one of the two, take the corpus gap. It is deterministic: the paragraph is absent, and once ingested it is present, verifiably, for that question and for every future question about the same document. The check is one index lookup per question, which makes it automatable over the entire labelled set at negligible cost, and that in turn means the whole class can be measured rather than sampled. The fix has no effect on any other question's ranking, so it carries no regression risk. A rank-40 retrieval miss is the opposite on each count: fixing it means changing the ranking function or the query, which changes the ordering for every other question in the corpus, so a change that rescues this question can quietly demote the gold chunk on nine others. That is exactly what the paired comparison from the previous chapter exists to catch — the same questions scored before and after the change, so each question is its own control — and it is why retrieval work always has to be measured on the whole set rather than on the question that motivated it.

The reason to prefer the gap is not that it is less serious. A corpus gap has a nastier property in aggregate: a labelled set only detects gaps for questions somebody wrote a label for, so a detected count implies a much larger number of undetectable ones, affecting questions nobody has asked yet. Preferring it is a statement about repair cost, and it comes with an obligation to ask what else the same mechanism broke.

The distinction has edges#

Two of the four buckets blur in specific ways worth naming. A paragraph that is indexed but chunked badly — split so that the sentence with the number lands in one chunk and the sentence naming which product it applies to lands in the next — is technically present, so the triage calls it a retrieval miss, while the actual fix is in chunking. The tell is that the gold chunk's text, retrieved on its own, does not actually answer the question without its neighbour, which is why the labelled set stores gold_text and why a human should read the retrieved chunk on any retrieval miss before assuming the retriever is at fault.

The second blur is at the generation boundary. An answer can be graded wrong while the model behaved correctly, because two retrieved passages contradicted each other — a current policy and a superseded one that reconciliation should have removed — and the model picked the wrong one. The trace says generation; the root cause is the version contract from the first chapter. When a generation-error bucket suddenly grows, checking whether the prompts in those cases contain two passages making incompatible claims separates a model problem from a corpus hygiene problem.

Both blurs share a shape, and naming it is more useful than memorising the two instances. The triage reports where a factor went to zero, and the stage that owns a factor is not always the stage that caused it to be zero. A badly chunked paragraph is a chunking fault reported at retrieval; a superseded policy left in the index is a reconciliation fault reported at generation. The bucket is where to start reading, not where to stop.

Finally, the triage assumes there is a single gold passage, and a real question set contains questions that need two documents. For those, the same four buckets apply per passage, and a question can be simultaneously a corpus gap for one and a retrieval hit for the other. The practical handling is to record a list of gold chunks and grade retrieval as all-present rather than any-present, since an answer built from half the evidence is usually wrong in a way that is harder to spot than one built from none.

The instinct this chapter is written against is the belief that ranking quality is the safe default explanation for a bad answer, and the reason it survives is that the evidence for it is a defence that never varies. A retrieval score reads much the same whether the gold passage was ranked second or was never in the index to be ranked, because a retriever asked for k passages returns k passages either way, all of them on topic and all of them scored. Four checks against four fields are what make the reading vary with the thing it has to discriminate: the index lookup separates existence from ordering, the deep rank separates a near miss from a distant one, the archived prompt separates evidence that arrived from evidence that was dropped, and the grade separates a model that ignored a passage from a pipeline that never delivered one.

With failures attributable to a stage, changes can be aimed at a stage, and the obvious next move is to aim several of them at retrieval and measure the result. That comparison has a counter-intuitive property: a configuration that gets the gold passage into the prompt far more often can produce fewer correct answers, which is where the next chapter starts.