Retrieval evaluation

Read these first

• Lexical search
• Building a golden dataset

What you will know

• Build a golden query set with known-correct documents

  If judgments are incomplete or inconsistent, recall and ranking scores become misleading, especially for queries with multiple valid sources.

• Compute recall@k and nDCG and explain what each one misses

  Recall can hide poor ordering, while nDCG depends on graded relevance and penalizes useful documents pushed below the cutoff.

• Measure the baseline before adding hybrid or reranking

  A lexical-only run provides the counterfactual; freeze the corpus, query set, and cutoff so later changes have an attributable comparison.

• Attribute an improvement to a specific change

  A gain may instead come from corpus changes, query leakage, or nondeterminism, so isolate variables and rerun the comparison.

• Refuse to ship a retrieval change with no before number

  This blocks anecdotal launches and forces a measurable rollback threshold when quality, latency, or index costs regress.

Chapters

1. 1

   Sampling and judging a golden set

   Questions you make up test the corpus you imagine; production logs test the one you have. Sample real queries stratified by type, with enough in each stratum that a change can move the number visibly, then judge them completely against a pool of candidates from every system you intend to compare. A query with three genuinely correct sources whose labeller recorded one understates recall and makes it unstable, because returning a different correct document scores zero — on one corpus that single defect was worth eighteen points of measured recall.

   Read chapter
2. 2

   nDCG and graded relevance

   Recall needs only a yes or no per document; nDCG needs a scale, so settle what a 3 means against a 1, write the rubric down, and check two labellers against each other before trusting either. The computation is then mechanical — each document contributes a gain set by its grade, discounted by a function of its position, summed to k, and divided by the same sum over the ideal ordering. Worked by hand on one query, with the discount table, the conventions implementations disagree about, and the structural limit a fixed cutoff imposes.

   Read chapter
3. 3

   Recall@k and nDCG@k

   Put the single correct document at rank 9 for every query and recall@10 reports a flawless 1.0 while nDCG@10 reports 0.301 on the identical ranking. Invert the case and a well-ordered run over a candidate pool that lost 37% of the relevant documents scores 0.88 on nDCG while recall@50 sits at 0.63. Recall exposes the candidate-stage ceiling and no position-aware metric at a fixed cutoff ever shows it, which is why the two are reported together, at different values of k, and why only one of them yields a list of things to go and fix.

   Read chapter
4. 4

   Choosing the headline metric

   Hit rate, precision@k, MRR, recall@k and nDCG@k computed on the same five queries give 0.60, 0.20, 0.517, 0.60 and 0.501, and each number is right about something different. MRR earns the headline when the consumer reads exactly one result and there is one correct answer; nDCG earns it when several documents contribute in graded amounts. Includes the per-query ceiling that makes precision@k misleading, the discrete ladder that makes MRR insensitive, and the two-tile dashboard that separates a ranking failure from an indexing one.

   Read chapter
5. 5

   Freezing a retrieval baseline

   A baseline is a number plus the exact conditions under which it can be regenerated: the corpus snapshot, the query set, the relevance judgments and the reported k cutoff, stored together and each versioned rather than edited. Includes the manifest and directory layout that make a March run replayable in September, why the raw bytes have to be content-addressed at ingest, and why the lexical run is the one to record first — a tuned BM25 baseline was worth nearly seven points of nDCG over the Lucene defaults on one corpus, and every point left on the table inflates the next change.

   Read chapter
6. 6

   Attributing a retrieval gain

   nDCG@10 rising from 0.61 to 0.68 in a fortnight that also contained an ingestion fix and a golden-set expansion credits nothing to the new embedding model, and subtracting the corpus delta does not repair it because the two changes interact by −0.036. Run the 2×2, measure the run-to-run spread before interpreting any delta, and rule out the two quieter rivals — leakage of golden queries into the corpus or a tuning loop, and nondeterminism from approximate indexes and hosted model calls. Includes the paired bootstrap and the per-stratum table that turns a scalar into a decision.

   Read chapter
7. 7

   Rollback thresholds and ship gates

   A rollback threshold needs a quantity, a value and a comparison, and shipping a chunking change unmeasured destroys the second one — so a later degradation in quality, latency or prompt cost can be neither proved nor attributed nor acted on. The objection that last month's baseline is stale is half right, which is exactly why the cheapest thing to accept is the old configuration replayed on today's corpus. Includes the six options with their real costs, the diff-set technique for a team with no current judgments, and a gate configuration that blocks per dimension rather than on a combined score.

   Read chapter