Sampling and judging a golden set

A golden set is a fixed collection of queries, each paired with the documents in a corpus that genuinely answer it. Scoring a retriever's ranked results against that list is retrieval evaluation: putting a number on how well the retrieval step finds the right documents, so that a change to it can be argued from evidence rather than from a demonstration. The plainest of those numbers is recall@k — for each query, the fraction of its known-correct documents that turn up anywhere in the retriever's top k results, averaged over the set. Read that way, a recall@10 of 0.94 says the top ten contained 94% of what it should have contained, on those queries, as judged by whoever wrote them. Every qualification in that sentence is load-bearing, and the two that decide whether the figure means anything are which queries were used and who judged what against them.

Measurement of this kind exists because retrieval is the stage of a search stack that resists inspection. A generated answer can be read and disliked; a ranked list of chunk ids returned for tens of thousands of distinct queries cannot be, and the handful of cases anyone checks by hand are the ones they thought to check. The practice turns up at three points in a working week. The first comes before putting hybrid search — a lexical and an embedding-based retriever run together with their result lists fused, which topic 8.13 covers — or a reranker, a slower second model that reorders what the first stage returned and the subject of topic 8.14, on top of a lexical index, since without a prior number those additions have nothing to be an improvement over. The second is the dashboard a search team watches daily, and the third is the check a change has to clear before it ships.

Topic 3.4, on building a golden dataset, covers how such a set gets assembled and kept alive as a general practice. Retrieval adds two constraints that will quietly invalidate the numbers if they are skipped: where the queries come from, and how completely each one is judged.

Sampling from the log rather than from the corpus#

A search box on one documentation product had logged 214,000 queries over 90 days. After lowercasing, trimming, and collapsing whitespace, 38,400 of them were distinct, and the distribution was the usual heavy head with a long tail — the top 200 normalized strings accounted for 31% of all traffic. Sampling uniformly from the distinct strings would have produced a set made almost entirely of tail queries, and sampling uniformly from raw traffic would have produced a set made almost entirely of pricing and api reference. Neither is what you want, because the two ends fail for different reasons and you need to see both.

The approach that survives contact with a real product is to bucket the log by query type first, then sample within each bucket. Classifying 38,400 strings by hand is not practical, so the classification runs as a cheap model pass with a handful of rules on top: anything containing an uppercase error token or a stack-frame pattern goes to the error bucket, anything under three tokens with no verb goes to navigational, and the rest gets classified by a small model against six labels with a few examples each. A sample of 200 was spot-checked by hand and the classifier agreed on 183 of them, which is good enough for a stratification that only needs to be roughly right.

Both numeric columns close, and checking that they do is the first thing to do with a table of this shape. The traffic shares are 31, 22, 18, 15, 6 and 8 percent, which add to exactly the whole — so the strata partition the traffic with no residual and no "other" bucket quietly absorbing whatever did not classify. The sampled column adds to 160, which is the set size every figure later in this chapter is computed over. A stratification whose shares fall short of the whole is one where some traffic is not represented at all, and the headline reweighting below would then be an average over a population smaller than the product's.

The classifier's error rate is small enough to leave alone, and the reason is worth working out rather than asserting, because the temptation is to spend a week improving it. Of 200 hand-checked strings the classifier agreed on 183, so it misfiles roughly one in twelve. On a stratum of 25 queries that is about two queries sitting in the wrong bucket, which shifts that stratum's measured rate by a few points at most. The sampling interval computed two paragraphs below is nearly twenty points wide on a stratum that size. The classifier's error is therefore well inside the noise the sample size already imposes, and improving the classifier without also enlarging the strata improves nothing that can be observed.

The sampled column is deliberately not proportional to traffic. Proportional sampling of 160 queries would put ten in the migration stratum, and with ten queries the smallest change any binary per-query metric can express is one query, which is ten points of recall. Nothing short of a catastrophic regression moves a number with that granularity, so the stratum would be present in the report and useless in it. Fifteen is not generous either, but it takes the resolution to 6.7 points and makes a two-query swing readable.

The arithmetic behind "enough to move visibly" is worth doing once explicitly. For a stratum of $n$ queries where per-query recall is either 0 or 1 and the true rate is around 0.7, the standard error of the stratum's mean is $\sqrt{0.7 \times 0.3 / n}$. At $n = 25$ that is 0.092, so the 95% interval spans roughly ±18 points and a genuine 5-point improvement is invisible. Across the whole 160-query set the same calculation gives a standard error of 0.036 and a band of about ±7 points. Those bands describe how much the number would wobble if you drew a fresh sample of queries, which is the right question when you are asking what the population rate is. They badly overstate the noise when you are comparing two systems on the same queries, because most of the variance is query difficulty and it cancels — which is why every run has to store its per-query scores and not just the mean, a point that becomes load-bearing later in this topic, where two runs are compared directly.

The relationship between those two bands is more useful than either of them alone. The set is six and a bit times the size of the stratum, and the band narrowed by a factor of about two and a half — which is the square root of six and a bit. That is not a coincidence about these particular numbers; the query count sits under a square root in the expression, so every band in this chapter shrinks with the square root of the sample and never faster. The practical consequence is the one that governs how a golden set is planned. Halving an interval costs four times the queries and four times the judging, so the decision about how deep to sample a stratum is made once, at the start, and cannot be repaired cheaply later by adding a few more.

Because the sample is not proportional, the headline figure has to be reweighted or it will report a fiction. A set that is 25% migration queries by count and 6% by traffic will show a headline that sags whenever migration queries are hard, even though almost nobody issues them.

import random
from collections import defaultdict

TARGETS = {
    "howto": 40, "navigational": 30, "error": 30,
    "entity_attr": 25, "migration": 15, "tail": 20,
}
TRAFFIC_SHARE = {
    "howto": 0.31, "navigational": 0.22, "error": 0.18,
    "entity_attr": 0.15, "migration": 0.06, "tail": 0.08,
}

def stratified_sample(labelled_queries, seed=20240301):
    rng = random.Random(seed)
    buckets = defaultdict(list)
    for q in labelled_queries:                 # {"text": ..., "stratum": ..., "count": ...}
        buckets[q["stratum"]].append(q)
    sample = []
    for stratum, target in TARGETS.items():
        pool = buckets[stratum]
        # weight by frequency so the head is represented without swamping the tail
        weights = [min(q["count"], 50) for q in pool]
        sample += rng.sample(pool, k=target) if len(pool) <= target else \
                  rng.choices(pool, weights=weights, k=target)
    return sample

def weighted_mean(per_stratum_score):
    return sum(TRAFFIC_SHARE[s] * v for s, v in per_stratum_score.items())

The seed is in the signature for a reason a later chapter of this topic makes formal: a query set you cannot regenerate byte for byte is not a query set you can compare against later.

One correct document is almost never the truth#

The second constraint does the most damage, because it looks like careful work. Take the query how do i rotate an api key. Three documents genuinely answer it: api-keys/rotation, the canonical procedure; security/credential-rotation-policy, which gives the required rotation interval and the approval step; and sdk/auth#refreshing-keys, which shows the eight lines of code that do it. A labeller working through the set marks the first one and moves on, because the first one is clearly right and the instruction said to record the correct document.

Now compare two retrievers on that query with the judgments as recorded.

The hybrid run scores zero on a query it answered. This is not a rounding problem. Unjudged documents are scored as irrelevant by every standard implementation, so an incomplete judgment can only ever move the measured number downward, which means measured recall is biased low rather than merely noisy. On top of the bias, the metric becomes unstable: a query whose three correct answers are collapsed to one flips between 0.0 and 1.0 on ranking changes that make no difference to a user, and a set of 160 queries with a dozen such flips will show movement of several points between two runs that are, in every way the user could perceive, identical.

The claim that the bias runs one way needs its condition stated, because it is not true of every possible edit to a judgment list. Recall is the count of relevant documents found, divided by the count of relevant documents known. Judging a document relevant when it was previously unjudged always adds one to the denominator, and it adds one to the numerator only if that document is inside the run's top k. Add a relevant document the run already returns and both counts rise, which lifts a recall below one. Add a relevant document the run does not return and only the denominator rises, which pushes recall down. So completing a judgment list is only guaranteed to raise a system's score when the newly judged documents are ones it was already returning — and that is precisely the situation pooling creates, since the pool is built from each system's own top results. Against a pool, every system gets credit for correct answers it was already producing and being scored zero for. Against judgments drawn from somewhere else entirely, the direction is no longer guaranteed.

The instruction given to the labeller is what decides this, and it is one sentence long either way. "Record the correct document" produces the set above. "Judge every document in the pool, independently of the others, against this query" produces a set where a query with three answers has three of them, and it costs the labeller nothing extra per query because they are already reading the whole pool to find the best one. Sets built under the first instruction cannot be repaired by re-reading them, only by re-judging against a pool, which is why the instruction is worth arguing about before the labelling starts rather than after.

Judging the pool instead of the corpus#

Judging completely does not mean judging all 48,000 chunks — the passage-sized pieces each document is split into before indexing — against all 160 queries, which is 7.7 million assessments. The technique that makes this tractable is pooling, and it is the reason the test collections built at TREC, the Text REtrieval Conference that has run shared search benchmarks since 1992, have stayed usable for decades. Run every system you intend to compare, take the union of the top 20 results from each, and judge that union exhaustively. With four systems and heavy overlap, the pool for a query typically lands between 30 and 50 documents, so 160 queries cost roughly 6,400 judgments — two labellers for about three days.

The size of that saving is worth naming, because it is the reason the technique exists rather than a convenience. Exhaustive judging is 7.7 million assessments; pooling is 6,400. That is a reduction of about twelve hundredfold, and it comes from one observation: a document no system in the comparison ever returns cannot change any of their scores, so judging it buys nothing today. The three days follow from the same arithmetic. Six thousand four hundred judgments split between two people is thirty-two hundred each, and thirty-two hundred spread over three working days is somewhere near two a minute — which is the right pace for reading a chunk against a query you have already read forty times that morning, and the wrong pace for reading a document you have never seen. Pool depth is what makes the rate achievable: the labeller reads one query and then forty candidates for it, rather than switching context on every row.

Four systems each contributing twenty results would produce eighty candidates per query if they agreed about nothing. The pool lands between thirty and fifty because they agree about a great deal, and the amount they agree about is the discount. A comparison between four systems that returned entirely different documents would cost nearly twice as much to judge — and would also be a comparison badly worth having, which is the trade in its honest form.

{
  "query_id": "q_0147",
  "query": "how do i rotate an api key",
  "stratum": "howto",
  "pool_depth": 20,
  "pooled_from": ["bm25", "bm25_synonyms", "dense_e5", "hybrid_rrf"],
  "judgments": {
    "api-keys/rotation": 3,
    "security/credential-rotation-policy": 2,
    "sdk/auth#refreshing-keys": 2,
    "api-keys/overview": 1,
    "changelog/2024-03": 0
  }
}

What pooling buys, and what it leaves uncovered#

Pooling buys completeness with respect to the systems in the pool and nothing beyond that. A retriever added two years later that surfaces a document no pooled system ever returned will be scored as if that document were irrelevant, and it will look worse than it is. The standard mitigations are to re-pool whenever a genuinely new retrieval family enters the comparison, and to watch the fraction of each run's top-k that is unjudged — once that fraction climbs past about 20%, the numbers are drifting away from what they were built to measure. Metrics such as bpref and infAP exist specifically to be robust under incomplete judgments and are worth knowing about, though they are harder to explain to a team than recall.

Why that watch threshold is a fraction of the run rather than a count of documents follows from what an unjudged result does to the score. Every unjudged document in a run's top k is scored as irrelevant, so it occupies a slot and contributes nothing, exactly as a genuinely irrelevant document would. Two systems can therefore only be compared on the part of their results the pool covers. When a fifth of a run's top ten is unjudged, a fifth of what that run is being credited for has been decided by the absence of a label rather than by a labeller, and the comparison is being made on eight slots rather than ten. The number to watch is a proportion because the harm scales with the share of each run that falls outside the pool, and it is a leading indicator rather than a lagging one — it rises the moment a new retrieval family starts returning documents nobody has judged, which is before any conclusion drawn from the run has had time to be wrong in public.

There are things a set built this way still cannot tell you. It says nothing about queries that were never typed because users learned the search box was bad, which is a survivorship problem in the log that no amount of sampling fixes; the only cure is to watch the set of zero-click queries over time and add new strata as the product changes. It is also not a test of the generator sitting downstream — retrieval can be perfect while the answer is wrong, and a golden query set has no opinion about that. And it measures nothing about latency or index cost, which are ship-blocking dimensions in their own right.

Two constraints, then, and both are about the inputs rather than the arithmetic. Queries come from the log, stratified by type and sampled deep enough per stratum that one query is not worth ten points; documents are judged against a pool drawn from every system in the comparison, so that a retriever answering a query differently is not scored as having failed it. Get either wrong and the metric is still computable, still reproducible, and measuring something other than what you think.

The judgments in the pooled block above are already graded rather than binary — api-keys/rotation carries a 3 and sdk/auth#refreshing-keys carries a 2 — because the canonical procedure and a code sample that happens to mention key refreshing are not equally useful to someone who typed that query. Recall does not care about the difference; it needs only a yes or a no per document. The metric that does care requires those numbers to mean something consistent across labellers, which is where the next chapter starts.