Short passages and a high precision@5 leave a reranker nothing to win

Reranking pays when a first stage returns the right documents in the wrong order. Both halves of that sentence are conditions rather than description, and a retrieval system can satisfy the first while failing the second completely. In that situation the second model is still accurate, still faster than the alternatives, and still worth nothing, because there is nothing left in the candidate list for accuracy to act on. An internal search over API reference documentation is such a system: 24,000 chunks, each one an endpoint description averaging 94 tokens, served into a typeahead box with a 150 ms p95 budget because results update as the user types. It was the second system in a company whose support-corpus search had just bought twenty points of nDCG@10 from a reranker, which made trying the same component here an obvious move, and the experiment took two days.

Three measurements decide whether that component has anything to do, and all three can be taken before it is deployed anywhere. precision@5 is the fraction of the top five returned results that a human judged relevant, so a value near one means the first five slots are already full of correct answers. The oracle score is what a candidate list scores when it is reordered by the judgments themselves — relevant documents first, in descending grade — which is the best any reranker could ever manage over that list. Headroom is the oracle minus what the system scores now, and it is the entire quantity a second stage is competing for. On the system above, all three point the same way.

What a reranker is allowed to change about a list#

A reranker receives a set of candidates and returns the same set in a different order. It cannot add a document the first stage failed to retrieve, and it cannot delete one except by pushing it below wherever the list is cut. Every metric computed on the final top ten is therefore a function of two things: which of the candidates are relevant, which the first stage fixed, and where the second model puts them, which is the only variable left.

That makes the ceiling computable rather than estimable. nDCG@10 weights each position by a discount that falls as rank grows, so the arrangement that maximises it is the one placing the highest-graded documents in the earliest positions. Score that arrangement and the result is the oracle — not an approximation of the best case but the best case, because it is the maximum over every ordering of that particular list. Headroom is the distance from here to there, and no reranker, however good, converts more of it than exists.

Now take the case this chapter is about, where the first five slots are already correct. Suppose the top five candidates are all relevant and all carry the same relevance grade. Reordering them permutes five equal grades across five fixed discounts. The sum of grade times discount is the same five numbers multiplied by the same five weights in a different pairing, and when the grades are equal the pairing changes nothing at all. A perfect reordering of an already-correct list gains exactly zero — not a small amount, zero — and every position a first stage fills correctly is a position permanently removed from what a second stage can win.

precision@5 is the number that says how many of those positions are already gone. A value of 0.86 means that on average more than four of the five slots hold a relevant document before any reranking happens. What is left for the second model is the fifth slot, the ordering among documents whose grades genuinely differ, and the tail of queries where the top five are not full — which is a much smaller quantity than the raw accuracy difference between a bi-encoder and a cross-encoder suggests.

There was almost nothing left to reorder#

The oracle experiment puts a number on that before any model is involved. Oracle nDCG@10 over the depth-50 candidate lists was 0.89 against an actual 0.834, so total headroom was 0.056 and the reranker converted about a third of it. Compare that with a support corpus of 5.2 million passages, where headroom was 0.33 and the same model converted 0.20, and the difference is not in the model's competence; it is in how much disorder there was to fix.

The headroom row is worth pausing on, because it is not a measurement. It is one row of the table subtracted from another: 0.890 minus 0.834 is 0.056, and 0.840 minus 0.51 is 0.330, both exactly and in both columns. Nothing has to be instrumented to produce it and no second model has to exist, which is what makes it available before a purchase rather than after one. The same property is its limitation: an error in either input arrives in the headroom undiluted, so a judged set with unlabelled relevant documents deflates the oracle and understates the prize in the same motion.

Reading the table down its columns makes the API-docs system look like a smaller version of the support corpus. Reading it across, as gaps rather than levels, shows the two doing different things. Headroom differs by roughly a factor of six and the measured gain by roughly a factor of twelve, while the latency added barely differs at all: 71 to 187 is a hundred and sixteen milliseconds, 186 to 288 is a hundred and two, and for planning purposes those are the same number. The cost of a second stage is a property of the second stage. The benefit is a property of the first. That asymmetry is the whole reason one number cannot be carried between systems, and it is also why the budgets settle the argument: a hundred-odd milliseconds is roughly three quarters of a 150 ms budget and about a fifth of a 500 ms one.

There is a rule from the ceiling argument that needs a condition attached before it is carried here. When most queries have a single relevant document, a perfect reranker either puts it at rank 1 and scores 1.0 or never had it and scores 0, so oracle nDCG@10 converges on recall. The support column obeys that: recall@50 of 0.86 against an oracle of 0.840. The API-docs column does not, with recall@50 of 0.97 against an oracle of 0.890 — a gap four times as wide. The reason is that nDCG normalises against the ideal ordering of all the relevant documents for a query, including any the candidate list never retrieved, so queries with several relevant documents of which only some were retrieved cap the oracle below 1.0 however it is ordered. Where that holds, reading recall as though it were the ceiling overstates the ceiling.

Passage length is the mechanism behind the precision@5 row, and it is worth being precise about why. A bi-encoder compresses a passage into one fixed-size vector, and the loss in that compression scales with how much distinguishable content the passage contains. A 94-token chunk describing POST /v2/webhooks — its path, its two required fields, its rate limit — has one topic, and a 384-dimensional vector represents it with very little left over. A 220-token support passage covering trial expiry, downgrade behaviour, a card-on-file condition and an exception for annual plans has four topics competing for the same 384 dimensions, and the query only cares about one of them.

Joint attention earns its cost by resolving which part of a passage the query is about. When a passage has one part, there is nothing to resolve, and the cross-encoder's advantage collapses to whatever it knows about phrasing that the embedding model does not. Chunking strategy therefore partly determines whether a reranker will pay, which means the two decisions are not independent: a team that chunks aggressively is buying first-stage precision and simultaneously reducing the return on a second stage. Neither of those effects is a requirement imposed by any model — chunk size is a configuration somebody chose, and choosing it short is a decision to spend the reranker's budget in the first stage instead.

The typeahead budget and the console budget#

The latency condition deserves separating from the quality one, because a budget is not a single number across a product. A typeahead box updating on every keystroke needs a p95 under about 150 ms for the list to feel attached to the typing, and the same corpus queried from a support agent's console after the agent presses Enter has 800 ms before anyone notices. That gap is large enough to change the answer, and it suggests a design that was considered and rejected here: rerank asynchronously, showing first-stage results immediately and reordering them once the cross-encoder returns. It was rejected because a result list that reshuffles under the cursor is worse than a slightly worse order that holds still, which is a usability constraint rather than a latency one and does not appear anywhere in a latency budget.

Both of those numbers are product decisions rather than engineering limits, and labelling them as such is what keeps the conclusion portable. Nothing about the corpus, the index or the model requires 150 ms; the requirement comes from having put search behind a keystroke. Move the same index behind an Enter key and the budget multiplies by more than five, at which point a hundred-odd milliseconds of reranking stops being the binding constraint and the quality argument has to stand on its own.

Reading precision@5 as a distribution#

precision@5 is the cheapest of the four signals to read, and reading it correctly means looking at its distribution rather than its mean. A mean of 0.86 could be a system that returns four or five good results for nearly every query, or one that is perfect on 70% of queries and useless on the rest, and only the second has a reranker-shaped problem hiding inside a healthy average.

Splitting the judged set by precision@5 showed 148 queries above 0.8 and 32 below 0.4. Those two counts add to 180, which is the whole judged set, so the band between them holds nothing at all. That is a stronger statement than it looks. precision@5 over five slots can only take six values — none of the five relevant, one of five, two, three, four, or all five — and the empty band is the middle of that range. The population is not a spread around a mean of 0.86; it is two populations, one where retrieval works completely and one where it fails badly, with almost nobody in between. A mean computed over it describes no query in the set.

The consequence follows arithmetically. Running the reranker only on the second group produced a gain of 0.09 within it. That group is 32 of 180, a little under a fifth of the queries, and 0.09 applied to a little under a fifth of the set is 0.016 on the overall mean — against a measured whole-set gain of 0.017. The two figures agree to a thousandth, which says that essentially the entire headline gain came from those 32 queries and the other 148 contributed approximately nothing. A router that sends only low-confidence queries to the reranker is therefore a legitimate design rather than a clever trick — it is the shape the data already has — and it was priced: it needs a confidence signal the first stage does not currently emit, and it would have cut the added p95 by about 80% at the cost of a component nobody had built.

The reranker that made it worse#

One arm of the experiment moved backwards. A ms-marco-trained cross-encoder — trained on web passages answering natural-language questions — scored prose above code on this corpus with some consistency. The clearest case was the query webhook retry backoff, where the first stage returned the right chunk at rank 2 and the reranker moved it to rank 7.

query: webhook retry backoff

first stage (rank, doc)
  1  Webhooks overview: delivery guarantees and ordering
  2  POST /v2/webhooks — retry policy: exponential backoff, 2^n s, max 6 attempts
  3  Configuring webhook endpoints in the dashboard

after ms-marco cross-encoder
  1  Webhooks overview: delivery guarantees and ordering        (score  4.11)
  2  Configuring webhook endpoints in the dashboard             (score  2.88)
  ...
  7  POST /v2/webhooks — retry policy: exponential backoff…     (score -1.24)

The causal chain there has no mystery in it once the steps are written out. The correct chunk is mostly a parameter table and a formula, with the words "retry" and "backoff" appearing twice in a hundred tokens of otherwise symbol-heavy text. A cross-encoder learns what a relevant passage looks like from the pairs it was trained on. The training distribution for this model contained essentially no such documents, so its notion of a relevant passage is a paragraph of fluent prose on the query's topic. It then applied that notion faithfully, which is the uncomfortable part: the model is not malfunctioning, it is generalising correctly from a distribution that does not include the documents being served.

The average is what hides this, and the way it hides it is worth naming, because it is the same defect in a different costume from the mean latency of a tail incident. Across the judged set this pattern cost 0.03 of nDCG@10 on the code-heavy queries and was masked by gains elsewhere, leaving the headline at +0.017. A mean over a mixed population does not move when one subpopulation regresses, provided another gains by a comparable amount, so a positive average is compatible with a segment getting materially worse. It is a defence that stays constant while the property it would have to discriminate — document format — varies underneath it, which is a good argument for reporting rerank deltas by query segment rather than as a single mean.

Fine-tuning would fix it. Three thousand judged pairs from click logs and a few hours of training on the sentence-transformers cross-encoder recipes would produce a model that knows a parameter table is an answer, and the projected gain from doing so was estimated at 0.03 to 0.05 — still under the headroom ceiling of 0.056, and still over budget on latency. The fine-tuning would have made the reranker good and left the decision unchanged, which is the clearest available demonstration that the binding constraint was never model quality.

Where the same four numbers argue for shipping#

Nothing here says rerankers are overrated, and the same two-day experiment on the support corpus in the earlier chapters returned a large enough number to justify three GPU replicas. The four conditions that made this case negative are all measurable in advance — headroom from the oracle run, precision@5 from the judged set, the latency budget from the SLO document, and domain fit from a segment breakdown — so the question of whether to rerank has an empirical answer per system, and the disagreements that consume the most time are the ones conducted without any of those four numbers on the table.

It also does not say the gain was fake. Seventeen thousandths of nDCG@10 is a real improvement that some users experienced as a better ordering, and if the budget had been 300 ms it would have shipped. A negative result of this kind is a statement about a constraint, and it should be recorded with the constraint attached, because that is what makes it possible to change your mind later without re-litigating the whole thing.

The one case the record above does not cover is the system that has no judged pairs to compute any of this from, or one where relevance is not the criterion at all — where the requirement is "prefer documents from the last quarter unless the query names a period" or "rank the section that states a policy above one that references it". Neither of those can be read off a judged set, because neither is a statement about whether a passage answers a question. They are instructions rather than labels.

Everything a reranker can buy is the distance between a candidate list's current order and its best possible one, and a first stage that returns short, focused passages has usually travelled most of that distance already. Measure the distance before buying the vehicle: the oracle run costs no GPU time, the precision@5 distribution costs a group-by, and between them they answer the question that a benchmark comparing two model architectures cannot answer for any particular system.

Where they say no, the useful artefact is not the decision but the conditions under which it would flip, written next to the numbers that produced it. And where the criterion itself cannot be expressed as a judgment about relevance — states-a-rule above cites-a-rule, recency as a tie-break — none of these four measurements applies, because there is no judged set that encodes the preference in the first place. There is a class of reranker that takes instructions.