An afternoon on your own golden set settles what a leaderboard cannot

Two embedding models can sit within a point or two of each other on every published board and still behave differently on one company's documents. An embedding model here is the function that turns a piece of text into a vector, and the two candidates are being compared on one thing only: how well the vectors they produce put a document somebody wanted near the top of a real query's results. No further reading settles that, because every published number was produced on somebody else's material. What settles it is a golden set — a fixed collection of queries, each carrying a list of documents a human has judged relevant to it, held still while the encoder underneath is swapped. Running both candidates against that set gives one score per query per model, and the difference between those two columns is the entire result.

Two independent properties decide whether that result means anything. The first is control: the experiment must vary the model and nothing else, or the difference it reports is the sum of several causes with no way to separate them. The second is resolution: a query set is a sample, and a sample answers coarsely or finely depending on how many queries it holds and how often the two models disagree. Control is bought with discipline while the run is being set up. Resolution is arithmetic, and it can be worked out before a single query is embedded.

What has to stay still#

The experiment measures one difference, so everything that is not the model has to be identical between the two arms. That sounds obvious and it is where most homegrown comparisons leak, because the natural way to evaluate a new model is to build a new pipeline around it, and the new pipeline usually has a better chunker in it too.

The reason is arithmetic rather than etiquette. Each query produces one number under each arm, and the difference between those two numbers is a single quantity. If two things changed between the arms — the encoder and the chunk size, say — that quantity is the sum of two effects, with nothing inside it recording which part came from which cause. Adding queries does not recover the split, because every extra query has the same two causes fused into it the same way. The only route to attributing a difference to the model is to arrange, before the run, that the model is the last candidate explanation standing.

The last row is the one that has to be treated as part of "held constant" even though it is the one thing that legitimately differs between the arms. What is held constant is the policy — every candidate gets exactly the prefixes its own model card documents — rather than the strings themselves.

Every row names a way for a second cause to enter, and the rows are not equally easy to notice. A different cutoff or index type is visible in configuration and gets caught in review. A corpus snapshot taken a week apart is visible nowhere: both arms run, both produce plausible numbers, and the documents that arrived in between are retrievable by the arm that ran later and did not exist for the arm that ran first.

Only the diamond changes between runs. Everything upstream and downstream of it is the same object, ideally literally the same code path with the encoder injected, because two separate scripts diverge within a week.

Where the queries and judgments come from#

A hundred and twenty queries sampled from a month of production search logs is the starting point, and the sampling should be stratified rather than random. Random sampling from search logs gives you whatever the head of the distribution is, which in most internal search is a handful of queries repeated thousands of times, and a set of 120 rows that is really 14 distinct queries measures very little. Stratify by the dimensions you expect to matter — in the logistics case that was query shape, question-shaped against keyword-shaped, and language, English against Spanish — and sample within strata in rough proportion to traffic, with a floor of about 20 queries per stratum so that per-stratum numbers mean something. That floor is what let the logistics team in the earlier chapter on leaderboard scores, whose queries were split into question-shaped and keyword-shaped groups, see that an aggregate loss was a win on one subset and a large loss on another.

The floor follows from the same arithmetic that fixes the whole set's resolution. A stratum's score is a mean over the queries in it, so each query carries a weight of one divided by the stratum's size. At a floor of 20 queries, one query changing its answer completely moves that stratum's number by five points. Below the floor, the per-stratum figure jumps further on a single query than the model differences anybody is trying to read.

Judgments are the part that takes the ninety minutes, and the way they are collected decides whether the experiment is fair. Judging by running one model, reading its top ten, and marking the good ones produces a judgment set that is defined by what that model retrieved, and the other model is then scored on its ability to reproduce a competitor's output. The correction is pooling: run both candidates, take the union of each one's top 20 for each query, shuffle the pool so the judge cannot tell which model contributed what, and judge the pool. For 120 queries with two systems and heavy overlap, the pool comes to roughly 2,600 query-document pairs, which two people get through in an afternoon at four or five seconds each for anything obvious and longer for the genuinely ambiguous ones.

Both of those figures are worth deriving, because each says something the sentence hides. The ceiling on the pool is 120 queries times two systems times 20 documents, which is four thousand eight hundred pairs. The actual pool is roughly 2,600, so the union per query is a little under twenty-two documents against a ceiling of forty — meaning the two candidates' top-20 lists agree on about eighteen of their twenty entries. The judging bill follows: 2,600 pairs at four or five seconds each is something over three hours of attention, and split between two judges that is the ninety minutes above. Candidates agreeing on eighteen of twenty documents also put the same documents in the top five for most queries, which is the fact the confidence interval later rests on.

Existing signals cut that work substantially where they exist. Support tickets that were resolved with a link to a document give you a query and a known-relevant document for free. Click logs give a weak positive. Both are biased — a click log records what the current system surfaced — and both are good enough to seed a pool that you then extend by judging what the challenger surfaced and the incumbent did not.

Two people judging the same pool disagree more than anyone expects, and the disagreements are informative rather than annoying. Writing down what relevant means before judging starts — for this corpus, "a support engineer reading only this chunk could resolve the ticket" — and having both judges label the same 100 pairs before splitting the rest takes twenty minutes and surfaces the ambiguities that would otherwise be resolved differently by each person. Agreement in the low seventies as a percentage is normal for retrieval judgments and is good enough for a comparison, because the disagreements affect both models roughly equally and the comparison is paired. Agreement in the fifties means the definition is not usable and no amount of extra queries will fix it.

"Affect both models roughly equally" is the load-bearing clause, and pooling is what makes it true. A document both candidates retrieved enters both arms' scores when a judge marks it relevant, so it pushes both numbers the same way and moves the gap between them far less than it moves either one. A document only one candidate retrieved enters that arm's numerator and only the other arm's denominator, which is where a judge's decision genuinely changes the answer. Given how far the two lists overlap, most disputed documents are of the first kind, which is why a definition applied consistently seven times in ten survives here and would not survive on separate query sets.

Running the comparison on a 200,000-chunk sample rather than the full nine million is what makes the afternoon possible, and it changes the absolute numbers without invalidating the comparison. Recall is easier on a smaller corpus because there are fewer distractors, so a model scoring 0.81 on the sample will score somewhere lower on the full index, and quoting the sample number as a production expectation is the mistake to avoid. What survives the reduction is the ordering and roughly the size of the gap, provided the sample is drawn so that every judged document is in it and the rest is a random draw across the whole corpus rather than one product area. A sample built by taking the most recent 200,000 chunks fails that condition and will flatter whichever model handles recent vocabulary better.

The number to report, chosen before the run#

Decide the primary metric before you see any results, because there are half a dozen defensible ones and the temptation to report whichever favours your preferred candidate is real and mostly unconscious. Recall@k at the k your pipeline actually uses is the right primary number for a system that stuffs retrieved chunks into a prompt, since a chunk at rank 4 and a chunk at rank 1 are worth the same to a model that reads all five. If your pipeline reranks before truncating — passing the retrieved candidates through a slower second model that reads each query and document together and rescores them — the encoder's job is to get relevant documents into that reranker's candidate pool, so recall@50 or recall@100 is the number that matters and top-of-list ordering barely does. nDCG@10 belongs in the report as a secondary number because it is what the published scores use and it keeps your result comparable to them.

Both names need unpacking, since everything after this point is arithmetic on them. Recall@k is the fraction of a query's judged-relevant documents appearing anywhere in the top k results: a query with three relevant documents and two of them inside the top five scores two thirds, and where in those five they sit changes nothing. nDCG@10 — normalised discounted cumulative gain at a cutoff of ten — does care where they sit. It credits a relevant document more the higher it ranks, then divides by the score a perfect ordering of the same documents would have earned, which puts every query on the same scale however many relevant documents it has.

import numpy as np

def recall_at_k(ranked: list[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return float("nan")
    return len(set(ranked[:k]) & relevant) / len(relevant)


def evaluate(encoder, queries, judgments, index, k=5) -> np.ndarray:
    """One score per query, in a fixed order, so results stay paired."""
    scores = np.empty(len(queries))
    for i, q in enumerate(queries):
        vec = encoder.encode_query(q.text)          # model's own query prefix
        ranked = index.search(vec, k=k, filters=q.filters)
        scores[i] = recall_at_k(ranked, judgments[q.id], k)
    return scores


a = evaluate(EncoderA(), QUERIES, JUDGMENTS, index_a)   # shape (120,)
b = evaluate(EncoderB(), QUERIES, JUDGMENTS, index_b)
print(f"A {a.mean():.3f}   B {b.mean():.3f}   delta {b.mean() - a.mean():+.3f}")

A query with no judged-relevant documents returns a not-a-number value rather than a score, and those queries have to be dropped from both arms or from neither. Dropping them from whichever arm they happen to hurt changes one arm's denominator and turns the paired comparison back into an unpaired one — the same defect as adding queries mid-run, and harder to see because it hides in a filter expression rather than in a config file.

Keeping the per-query arrays rather than only the means is what makes the next step possible, and it costs nothing to store. The comparison is paired — both models answered the same 120 queries — and paired data supports much tighter inference than two independent means would.

Nothing is lost by holding the arrays instead of the averages. The mean of the per-query differences is exactly the difference of the two means, because the same values are being added in a different order: the same total ends up divided by the same count either way. What the arrays add is what the two averages threw away — which queries moved, in which direction, and by how much.

How many queries before a difference is readable#

Start from what one query is worth, because every figure below is built out of it. The reported score is a mean over the set, so each query contributes its own score divided by the count of queries. Moving one query from finding nothing to finding everything moves the aggregate by one part in 120, a little over eight tenths of a point. A 5-point gap is therefore worth about six queries flipping completely, out of 120. Real queries rarely flip completely: one with three judged-relevant documents moves by a third when a single document crosses the top-five boundary, so a gap of a few points is a dozen or so partial moves in one direction outnumbering the partial moves in the other.

The variance that matters is the variance of the per-query difference, not the variance of either model's scores. Recall@5 per query is a coarse, lumpy quantity that takes values like 0, 0.33, 0.5 and 1, so each model's scores have a standard deviation around 0.35. The differences are far tamer, because the two models agree completely on most queries: in a typical run, 60 to 70 of 120 queries produce an identical score under both models and the difference is exactly zero. That concentration is what buys the statistical power.

The concentration is not luck; it follows from the overlap already visible in the judging pool. Candidates sharing about eighteen of twenty retrieved documents usually retrieve the same handful in the top five, and recall@5 notices nothing except membership of that top five. Where both arms retrieve the same set, the difference is exactly zero and contributes exactly zero to the spread of the differences. A standard deviation is a root-mean-square of deviations from the mean, and more than half of this set contributes a squared deviation of nothing, which is why the paired spread lands at about half of either arm's own spread where two unrelated columns would have produced a spread larger than either.

Subtracting before averaging is what makes that small paired spread available, and the alternative shows what it is worth. Ignore the pairing and treat the two averages as though the models had answered different query sets. Each arm's scores scatter with a standard deviation around 0.35, so each average carries an uncertainty of that spread divided by the square root of 120, a little over three points. Two independent averages combine by adding their variances rather than their spreads, which multiplies the uncertainty of the gap by the square root of two, and a 95% interval reaches about two of those in each direction. The gap would arrive with a band of roughly nine points either way — wide enough to contain a win of a few points, its mirror image, and zero.

The paired version escapes that because most of the 0.35 is not about the models at all. It is a property of the queries: some have one relevant document and some have six, some are answered by any competent encoder and some by neither candidate. Every one of those properties is present in both arms identically, because both arms answered the same query against the same judgments over the same corpus. Subtracting the two scores for a query cancels that shared difficulty and leaves only what changing the model changed. The 0.17 that remains is the spread of the model effect alone.

def paired_bootstrap(a: np.ndarray, b: np.ndarray, iters: int = 10_000) -> tuple:
    d = b - a
    n = len(d)
    rng = np.random.default_rng(0)
    means = np.array([d[rng.integers(0, n, n)].mean() for _ in range(iters)])
    return d.mean(), np.percentile(means, 2.5), np.percentile(means, 97.5)

point, lo, hi = paired_bootstrap(a, b)
print(f"delta {point:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")
# delta +0.046  95% CI [+0.015, +0.077]

The bootstrap in that function is the standard way to get an interval without assuming the differences are normally distributed: it draws 120 of the per-query differences at random with replacement, takes the mean, repeats ten thousand times, and reports the range that the middle 95% of those means fall into. With a paired standard deviation around 0.17, the standard error on 120 queries is 0.17 divided by the square root of 120, or about 0.0155, so the 95% interval is roughly ±3.1 points. That is the resolution of the instrument: 120 queries can tell a 5-point difference from zero and cannot tell a 2-point difference from zero. Resolving 1.5 points at the same confidence needs the interval to shrink by a factor of two, which needs four times the queries — around 500 — and that is no longer an afternoon. Deciding in advance that differences under 3 points will be treated as a tie is more honest than collecting 120 queries and then reporting a 2-point win as a win.

Two steps in that paragraph are worth separating. The square root appears because independent quantities add their variances and not their spreads: a total of 120 per-query differences carries 120 times the variance of one, so the total's spread grows like the square root of the count while dividing by the count to get a mean shrinks it by the full count. What survives is one query's spread over the square root of the number of queries. The interval is then about two standard errors on each side, which is how 0.0155 becomes roughly ±3.1 points. Because the shrinking goes as a square root, resolution is bought expensively: halving the interval costs four times the queries and halving it again costs sixteen times the original.

The same currency says what would overturn a verdict. The distance between the 4.6-point win this run reported and the 3-point line below which the team had agreed to call a tie is a little under two points, which is about two queries' worth of score at a little over eight tenths of a point each. Two disputed judgments resolved the other way, or two queries swapped for two others from the same stratum, turns the reported win into a declared tie. That is a fair description of a thin result rather than an argument against it, and it is why the written definition of relevance, the pooled judging and the pre-declared threshold are load-bearing parts of the experiment.

A sign test is a useful second view because it makes no assumption about the shape of the scores. Of the 120 queries, 47 had a non-zero difference; 32 favoured B and 15 favoured A, which under a fair coin has a two-sided probability of about 0.02. When the bootstrap interval excludes zero and the sign test agrees, the result is as settled as this instrument can make it. When they disagree, usually because a handful of queries have enormous differences that drive the mean, read those queries by hand before believing either.

That probability is worth seeing rather than taking on trust. If the models were equally good, each of the 47 queries where they differed would fall to one side like a coin, so the expected count on B's side is half of 47, a little under twenty-four. The spread of a fair-coin count over 47 tosses is half the square root of 47, a bit under three and a half. The observed 32 sits about eight and a half above the centre, roughly two and a half of those spreads out, and about 0.02 is the chance of landing that far out or further on either side. The test reads only the direction of each difference and never its size, which is exactly what makes it a check on a mean one enormous query could drag.

Realism and resolution are different properties#

The belief that gets people into trouble here is that a golden set built from a company's own logs can be trusted because it is real. Realism is what makes the question the right one: these are queries your users typed, against documents you hold, judged by people who know what a good answer looks like. None of that bears on whether the answer is audible over the noise, which is settled by the count of queries and by how often the two models disagree.

What makes "it is our own data" a defence worth naming is that it returns the same verdict whatever is true. Run 120 real queries against two models separated by a large true difference and the harness prints a delta to three decimal places. Run the same 120 real queries against two models with no true difference at all and it prints a delta to three decimal places, usually a point or two from zero, because 120 lumpy per-query scores do not average to the same number twice. A check offering the same reassurance whether or not the effect is present is not discriminating between those cases. The one part of the output that does change with the truth is the interval: it covers zero when the evidence cannot rule zero out and excludes it when the evidence can. Report the interval beside the delta, fix the tie threshold before the run, and the set starts doing work its realism alone never did.

Answer quality, latency and cost are separate measurements#

The run measures retrieval quality on the queries you sampled, and that is a narrower claim than it feels like while you are reading the output. It says nothing about whether the end-to-end answers get better, because a system that retrieves the right chunk and then generates a poor answer scores identically to one that does not. It says nothing about latency or cost, both of which have to be measured separately and both of which can override a quality result. And it is not a production A/B test: it uses stored judgments rather than live user behaviour, so it cannot see the queries users would have typed if search worked better.

The queries your logs cannot contain#

The bias worth naming out loud is that a golden set sampled from your logs is a sample of queries your current system trained your users to type. People who searched three times, got nothing useful, and gave up are not in the log as a hard query — they are in it as three failed queries and then silence, or not at all if they stopped using search. A challenger that is dramatically better at a kind of query nobody currently types will look ordinary on this set. Adding twenty queries written by hand, describing things people say they wish they could find, is a cheap partial correction and should be kept as a separate stratum so it does not contaminate the traffic-weighted number.

The harness outlives the run, and the query set does not#

The four and a half hours are also a first-time figure only in the sense that the judgments dominate them. Once the harness exists as code — an encoder interface with two methods, a fixed corpus sample on disk, the query set and judgments in version control, and the bootstrap in a function — adding a third candidate is forty minutes of compute and no human time at all. That asymmetry is worth exploiting deliberately: the shortlist that came out of the leaderboard filtering had four models in it, and there is no reason to run two when the marginal cost of the other two is an hour of machine time against ninety minutes of judging that has already been paid for.

The asymmetry stops short of free, and the pooling rule says where. A third candidate retrieves documents the first two never surfaced, and those are unjudged, so a strictly fair addition means pooling its top 20 as well and judging what is new. That is a fraction of the original pool rather than another whole one, because the lists overlap heavily. It is not nothing either, and a comparison that skips it scores the newcomer on its ability to reproduce documents its predecessors happened to find.

The set also goes stale. Corpus composition drifts, a new product line arrives with vocabulary nobody has judged, and a query set from fourteen months ago is measuring a corpus that no longer exists. Re-sampling and re-pooling annually is roughly the right cadence, and it is the same afternoon each time once the harness exists.

A leaderboard cannot settle a choice between two models sitting a point apart on it, because its number was produced on somebody else's documents, with somebody else's queries, at somebody else's cutoff. An afternoon settles it, and what makes the afternoon work is not effort. The experiment varies one thing, scores every query under both arms, and subtracts before it averages, which turns most of the noise into exact zeros and leaves an interval narrow enough to act on. The realism of the set decides that the question is worth asking; its size decides whether the answer can be heard, and that part is arithmetic available before anything is embedded. At 120 queries the band is roughly three points either way, and a difference smaller than that is a tie whatever the mean says.

The afternoon produces a winner on quality. What it deliberately holds fixed is the shape of the vectors that winner produces, and that shape has consequences the retrieval numbers never show: a 4096-dimension winner and a 768-dimension runner-up impose very different bills at ten million chunks, which is arithmetic worth doing before the quality result gets ratified.