The first stage sets a ceiling the reranker cannot lift

Two numbers describe a two-stage search system, and only one of them usually gets quoted. The quoted one is how much better a reranking model orders a list than the retriever did. The other is how often the list contains anything worth ordering. Recall@50 is that second number: of the queries in a judged set — a set of queries where somebody has marked which documents count as correct answers — the fraction whose correct answer appears somewhere in the first fifty results the retriever returned. It is a statement about presence, and it is indifferent to position: a correct document at rank 50 counts exactly as much as one at rank 1.

A reranker's whole job is position. It takes the fifty candidates and puts them in a better order, which is measured by nDCG@10 — normalised discounted cumulative gain at ten, a score between 0 and 1 that compares the order a system returned against the best order the judgments allow, counting documents further down the list for less. Presence is decided upstream and order is decided downstream, and no amount of the second recovers a failure of the first. On a query whose correct answer was never retrieved, every possible ordering of fifty wrong documents scores identically, and that score is 0.

The oracle rerank tells you the whole budget in advance#

There is a cheap experiment that reports the ceiling directly, and it costs no GPU time at all. Take the candidate lists your first stage actually produced, and reorder each one by the judgments instead of by a model — relevant documents first, in descending grade, everything else after. That is the best ordering any reranker could ever produce over those candidates, so scoring it gives the exact upper bound on what the second stage can buy.

The reason it is an exact bound rather than an estimate is worth stating, because it is what separates this experiment from a benchmark. A reranker is a function from a candidate list to a permutation of that list. The judgments define one permutation that is at least as good as every other, since they are the same judgments the metric will score against. So the oracle ordering is not the best a good reranker manages; it is the maximum of the metric over the entire space of orderings the reranker could possibly output. Nothing a model does can exceed a maximum taken over everything it could have done.

def oracle_rerank(runs, qrels, k_candidate=50, k_final=10):
    """Best achievable metrics if the reranker were perfect."""
    oracle, actual = [], []
    for qid, ranked in runs.items():
        cands = ranked[:k_candidate]
        judged = qrels[qid]
        best = sorted(cands, key=lambda d: -judged.get(d, 0))
        oracle.append(ndcg_at_k([judged.get(d, 0) for d in best[:k_final]],
                                list(judged.values()), k=k_final))
        actual.append(ndcg_at_k([judged.get(d, 0) for d in cands[:k_final]],
                                list(judged.values()), k=k_final))
    return {"ndcg@10_now": mean(actual),
            "ndcg@10_oracle": mean(oracle),
            "headroom": mean(oracle) - mean(actual)}

Two details in that function carry the argument. best sorts the candidates by descending judged grade, which is the oracle permutation; actual leaves them in the order the retriever produced. Both are scored by the same ndcg_at_k call against list(judged.values()), the grades of every judged document for that query — including documents the retriever never returned. That second argument is what makes the ideal denominator honest, and it will matter again shortly.

Run against a dense-retrieval system serving support search, ndcg@10_now was 0.44 and ndcg@10_oracle was 0.61. The difference is the headroom: 0.61 less 0.44 is 0.17, so seventeen points of nDCG@10 is the entire prize, and it is the prize for a reranker that is never wrong. A real cross-encoder captures some fraction of an oracle's headroom — in this system, measured afterwards, it captured about 60%, which is a normal figure. Sixty per cent of 0.17 is about 0.10, so the honest projection for a reranker here was ten points of nDCG@10 rather than the eighteen a model comparison had shown.

That multiplication deserves a challenge, because writing gain = rate × headroom presumes that the rate is a property of the reranker rather than of the particular gap it happened to be measured on. Nothing derives it. It is an empirical regularity, and the way to earn the right to use it is to observe the same rate on a second, differently sized headroom, which the configuration table below does.

The 60% conversion rate deserves a second note, because it is the one figure in the projection that has to be estimated rather than measured up front. It is the fraction of the oracle's headroom a real reranker actually captures, and it varies with how well the reranker's training distribution matches your domain: a ms-marco cross-encoder on web-like question traffic lands around 0.55 to 0.7, the same model on dense legal or clinical text can drop under 0.4, and a reranker fine-tuned on your own judged pairs has been seen above 0.8. Until you run one, the honest thing to write in a proposal is a range, with the low end being the number the decision has to survive.

The oracle number is also the cleanest way to describe what recall@50 means to someone who does not want to hear about metrics. Recall@50 of 0.62 and an oracle nDCG@10 of 0.61 are nearly the same number, and that is not a coincidence. Follow it through on one query at a time, assuming for the moment that the query has exactly one relevant document. If that document is among the fifty candidates, the oracle ordering puts it at rank 1 and the query scores 1.0. If it is not among them, no ordering can reach it and the query scores 0. Every query therefore contributes either a one or a zero, so the mean of those contributions — which is what ndcg@10_oracle reports — is precisely the fraction of queries whose document was present, and that fraction is the definition of recall@50. Under the one-relevant-document assumption the two quantities are not merely close; they are the same quantity computed twice.

The assumption is what the missing point measures. Where a query has several relevant documents and the retriever found some of them, the oracle scores between zero and one rather than at either end, because the ideal denominator in ndcg_at_k was built from all the judged documents, including the ones the retriever missed. A perfect ordering of an incomplete set still loses ground to an ideal that includes what was never retrieved. That pushes the oracle figure below the recall figure and cannot push it above, which is the direction observed: 0.61 sits one point under 0.62. The same check on the hybrid configuration further down gives 0.84 under 0.86, two points, in the same direction. An identity that reproduces itself on a second configuration, with the residual on the side the mechanism predicts, is one you can use as a diagnostic. Whenever those two numbers sit close together, the ceiling is being set by presence rather than by ordering.

Thirty-eight percent of queries, and why they missed#

Recall@50 of 0.62 is a fraction of queries whose answer was present, so its complement is the fraction whose answer was not, and that complement is 38%. Against a judged set of 200 queries, thirty-eight per cent of two hundred is 76 queries for which the cross-encoder receives fifty wrong answers and returns ten of them in an excellent order. Its score for each of those queries is 0 however good the model is.

Averages hide the shape of a miss, and the shape determines where the money goes. Pulling the 76 failing queries out and reading them took an afternoon, and they did not form one population.

Add the Queries column downwards — 22, then 31, then 9, then 14 — and the total is 76, which is the whole miss population with nothing left over. The absence of a residual is the useful part. There is no fifth bucket, no "other", and therefore no unexamined cause that could turn out to be the real one. The table can be read as a budget: fixing every vocabulary mismatch removes 31 of the 76 misses and leaves the rest untouched, and both halves of that sentence are known in advance rather than discovered afterwards.

Read the last column downwards and a second thing falls out. Ingestion, retrieval fusion, chunking, retrieval again — the reranker is named in none of the four rows. That is not an argument that rerankers are useless; it is the observation that across all 76 of these queries, zero are addressed by ordering the candidates better, because in all 76 the candidates do not contain the answer.

The 22 ingestion failures are the ones worth sitting with. No retrieval change of any kind touches them, and neither does a reranker, and 22 out of 200 is 11% of the judged set. They were found by asking, for each miss, whether the gold document existed in the index at all — a single lookup by document ID, which is the first check to run whenever recall is disappointing and the second stage is being blamed.

The 31 vocabulary-mismatch misses are the reason the cheaper counterproposal existed. A dense retriever alone was doing the candidate generation; adding the lexical arm and fusing with RRF — reciprocal rank fusion, which merges two ranked lists by scoring each document on its position in each, and which the hybrid search work had already built and tested — lifted recall@50 from 0.62 to 0.86 on the same 200 queries, a gain of twenty-four points. That is a first-stage change costing no additional inference at query time beyond a BM25 lookup measured in single-digit milliseconds, and it moved the oracle nDCG@10 from 0.61 to 0.84.

Reading the two numbers as a decision#

The sequencing follows from the arithmetic rather than from taste. A reranker converts existing headroom into realised quality at a fixed conversion rate of somewhere between a half and two thirds; a first-stage improvement creates headroom that did not exist. When headroom is small, the reranker has little to convert regardless of how good it is, and when headroom is large the first-stage fix and the reranker are not competing, because the reranker will convert more of a bigger number afterwards.

The Headroom column is not a fifth measurement. It is the Oracle column minus the now column, and it reproduces itself on every row: 0.61 less 0.44 is 0.17, 0.61 less 0.54 is 0.07, 0.84 less 0.51 is 0.33, 0.84 less 0.71 is 0.13. Checking a derived column against its own inputs is a thirty-second habit that catches a transcription error before anybody builds a decision on one, and here it also makes the table legible: every row says what is left, not just what was reached.

Now the conversion rate can be tested rather than assumed. In row two the reranker moved 0.44 to 0.54, a gain of 0.10, out of the 0.17 of headroom row one reported — a shade under 60%. In row four it moved 0.51 to 0.71, a gain of 0.20, out of the 0.33 that row three reported — a shade over 60%. Two different starting points, two different headrooms, one rate. That is what licenses writing the projection as a multiplication in the first place, and it is why the 0.20 in row four was predictable before the money was spent: sixty per cent of 0.33 is about 0.20.

The second and third rows are the decision. A dedicated GPU endpoint and a sprint of integration bought 0.10. A configuration change and a re-index bought 0.07 at effectively no marginal cost, and left 0.33 of headroom standing behind it — roughly three times what the reranker had just delivered on its own, and still there to be converted later. The fourth row is where the system ends up, and the ordering matters because the reranker's contribution in row four is 0.20 rather than the 0.10 it managed in row two. The same model, the same weights, the same fifty candidates per query, and twice the effect, because there was twice as much correct material in the candidate lists for it to promote.

The stumble here is not an arithmetic slip. It is the belief that a benchmark result transfers, and it is tempting because this particular benchmark was unusually well controlled: same corpus, same queries, same judgments, one variable changed. What was not controlled is whether the candidate list handed to the model contained the answer, and in the measurement it always did. Ask what that measurement would look like in a world where the first stage was much worse, and the answer is that on every query the first stage missed, a perfect reranker and a terrible one both score 0. The measurement is identical in both worlds, so it separates them not at all — which is the same reason a year with no reported incidents is not evidence that reporting works.

Pooled judgments and the depth the ceiling is quoted at#

Recall@50 measured on a judged set inherits every limitation that set has. If the judgments were pooled from the systems you already run — pooling being the standard practice of sending only the documents those systems surfaced to a human for labelling — documents that only a new retriever would surface are unjudged and therefore counted as wrong. That biases recall downward for exactly the change you are evaluating, and it biases it in the direction that makes the first-stage improvement look smaller than it is. A recall figure that has never had a manual look at its misses is a figure with unknown sign on its error.

The ceiling is also stated at one depth, and depth is a choice rather than a fact. Recall@50 of 0.62 in this system was recall@200 of 0.79 and recall@1000 of 0.91. Read those three as gaps rather than as levels. Quadrupling the depth from fifty to two hundred adds seventeen points of recall; multiplying by five again, to a thousand, adds twelve more. The gain per candidate is falling steeply while the cost per candidate is flat, since every additional candidate is one more forward pass at the same price as the last. A reranker over 200 candidates therefore has a materially higher ceiling than one over 50, a reranker over 1,000 has a higher one still, and the marginal candidate is worth less each time you add one.

That reframes the decision. Rather than reranker-versus-retriever, the question becomes how deep the candidate list should go, which trades a ceiling that keeps rising against a cost that rises with it and against a precision effect that runs the other way. Nothing in the oracle experiment tells you where on that trade to sit. It reports what a perfect reranker could do at a given depth, and says nothing about what a real one does when the candidates get worse, which is what happens as depth increases. Both of those curves have to be measured on the same data before a depth can be defended.

There is a version of the ceiling argument that goes too far. Recall@50 of 0.62 does not mean a reranker is worthless — seven to ten points of nDCG@10 is a real improvement that users notice, and there are systems where the first stage is already hybrid, already well-tuned, and the ingestion is clean, so 0.62 is simply what a hard collection permits. The claim is narrower: the number tells you which of two purchases has more room in it. Buying the second-stage one without reading it means finding out afterwards that a third of the traffic was never reachable.

Two numbers, then, and an order to read them in. Recall@50 says how often anything worth ranking is present; the oracle nDCG@10 says how much better the present material could possibly be ordered. The first bounds the second, the gap between the second and today's score is the whole prize, and a real model collects a stable fraction of that prize rather than all of it.

The experiment that produces both is a sort and a metric over run files you already have, which is why it belongs before the purchase rather than after it. A reranker bought against a measured headroom is a priced decision that can be defended when it lands. A reranker bought against a benchmark gap is a bet that the candidate lists in production look like the candidate lists in the benchmark, and the whole point of measuring recall is that they usually do not.