Recall@k and nDCG@k

The two metrics are computed from the same inputs and answer different questions. Recall@k is the fraction of a query's judged-relevant documents that appear anywhere in the top k, averaged over the golden set, and it discards position entirely. nDCG@k discounts each document's graded contribution by the position it was returned at and divides by the best ordering the judgments allow, so it is almost entirely about position. One run scored both ways produces two numbers that can move in opposite directions, and neither is wrong when it does.

The pathological case makes the arithmetic obvious. Suppose every query in a golden set has exactly one relevant document, and the retriever returns it at rank 9 every single time. Recall@10 counts presence in the top ten and reports 1.0, a perfect score. nDCG@10 divides the discounted gain at rank 9 by the discounted gain at rank 1, and since the gain cancels regardless of grade, it reports $1/\log_2(10) = 0.301$ for every query and therefore 0.301 for the run. The cancellation is worth seeing rather than taking on trust, because it is what makes the figure independent of the rubric. The numerator is the document's gain times the discount at rank 9. The denominator is the same document's gain — it is the only judged document, so the ideal ordering is that one document at rank 1 — times a discount of 1. The gain appears once above and once below and divides out, whatever grade the labeller assigned, leaving the discount at rank 9 on its own. That is why the score is exactly the discount and why it is the same number for a grade 3 and a grade 1. The same ranking, judged identically, scores 1.0 and 0.301 depending on which question you ask of it. That gap is not an artefact of the extreme case: for any query set with one relevant document per query, nDCG@10 is bounded above by recall@10 and equals it only when every hit lands at rank 1, so the difference between the two numbers is a direct read-out of how much ordering quality is being lost.

The failure recall cannot see, and the one nDCG cannot#

The inverse case is less obvious and does more damage, because it is produced by an evaluation that looks careful. Retrieval in one system is two stages: BM25 returns 50 candidates, a cross-encoder reranks them, and the top 10 go to the generator. A cross-encoder is a model that reads the query and one candidate document together and scores that pair directly, which is accurate and far too slow to run across a whole index, so it only ever sees a shortlist. The team working on the reranker evaluates the reranker, which means scoring the ordering it produces over the 50 candidates it was given. Their nDCG@10 was 0.88, which reads as respectable by any standard, and it was honestly computed — the ordering really was close to the best available ordering of those 50 documents.

Recall@50 over the same golden set was 0.63. Thirty-seven percent of the judged-relevant documents were never in the candidate list at all, so no ordering of those 50 could include them, and the reranker's 0.88 was a score on a curated subset of the problem. Every point of that 0.88 was real and none of it was reachable by a user, because the documents the user needed had already been discarded one stage earlier.

The mechanism is the IDCG convention from the previous chapter. When the ideal ordering is computed over the documents present in the run rather than over every judged-relevant document for the query, missing documents cannot lower the score — they are absent from both the numerator and the denominator. Even with the stricter convention, a fixed cutoff hides the ceiling: nDCG@10 on a query with six relevant documents where four are missing still divides by an ideal that only contains ten positions, and a run that nails the ordering of the two it has looks better than it deserves.

Stated as a property rather than as a case, that is the reason nDCG cannot be trusted alone at this boundary. Take a candidate stage that loses a third of the relevant documents and one that loses none, hand both to the same reranker, and let it order what it receives well in each case. Under a run-only ideal both score highly, and under a strict ideal with a fixed cutoff both still score highly, because the missing documents are outside a window the metric never looks through. So the number is nearly the same in the good case and the bad one — it does not vary with the thing it is being asked to detect. A measurement with that property is not a weak signal about candidate quality. It is no signal about candidate quality, and adding decimal places to it will not create one.

Recall at the candidate depth is the only metric in the set that reports this. It is why the depth you measure recall at should be the depth of the stage you are diagnosing, not the depth the generator consumes. Recall@10 answers "did the final list contain it"; recall@50 answers "could the reranker possibly have surfaced it"; recall@200 answers "is the index capable of it at all".

Four runs on one golden set#

The same 160 queries, the same judgments, four configurations. The numbers came out of a single evaluation script run against stored result lists, so nothing differs except the retrieval configuration. The two hybrid rows combine the lexical ranking with an embedding-based one using reciprocal rank fusion (RRF), which scores each document by summing $1/(60 + \text{rank})$ over the lists it appears in and therefore needs no calibration between the two scoring scales.

The second row is the load-bearing one. Reranking permutes a candidate list, so recall@50 is identical to the row above it — 0.81 in both cases, and it could not be otherwise. All nine points of recall@10 improvement came from documents moving from ranks 11–50 up into the top 10, which is the same movement the nDCG gain describes. If the only number on the dashboard had been nDCG@10, the reranker would have looked like a 10-point win and the hybrid retriever like a 5-point win, and the team would have prioritised the smaller of the two, because nDCG cannot tell you that the hybrid change raised by twelve points the ceiling the reranker is working underneath.

The reranker's contribution appears twice in that table and it is the same size both times, which is the check that the two stages are doing independent jobs. On BM25 the reranker moves recall@10 from 0.62 to 0.71. On the hybrid candidate list it moves recall@10 from 0.74 to 0.83. Nine points in both cases, on two different candidate lists with two different ceilings — and the recall@50 column does not move in either case, because a permutation of a fixed list cannot change what is in it. Read down that column and the two 0.81s and the two 0.93s are the arithmetic saying the same thing: the reranker owns the ordering and the retriever owns the ceiling, and each one's contribution can be read off without disentangling it from the other's.

Reading the pair as a pair gives a small diagnostic vocabulary. High recall@50 with low nDCG@10 is a ranking problem, and a reranker or better fusion weights will move it. Low recall@50 with high nDCG@10 on the candidate set is a candidate-generation problem, and no amount of reranking will touch it — the fix is in the analyzer, the chunking, the query expansion, or the addition of a dense retriever. Both low is usually an ingestion problem, and it is worth checking whether the documents are in the index at all before touching retrieval. Both high is the case where you stop.

def evaluate(runs, qrels, k_final=10, k_candidate=50):
    """runs: {query_id: [doc_id, ...]} in rank order, at least k_candidate long."""
    rows = {}
    for qid, ranked in runs.items():
        judged = qrels[qid]                       # {doc_id: grade}
        relevant = {d for d, g in judged.items() if g > 0}
        if not relevant:
            continue
        top_k = ranked[:k_final]
        rows[qid] = {
            "recall@k": len(relevant & set(top_k)) / len(relevant),
            "recall@cand": len(relevant & set(ranked[:k_candidate])) / len(relevant),
            "ndcg@k": ndcg_at_k([judged.get(d, 0) for d in top_k],
                                list(judged.values()), k=k_final),
        }
    return rows

The judged.get(d, 0) on the second-to-last line is where the incompleteness from the first chapter of this topic enters the arithmetic: an unjudged document contributes grade 0, so a pool that missed a genuinely correct document silently converts it into a penalty. Passing list(judged.values()) as the ideal set rather than the grades of the returned documents is the strict IDCG convention, and it is the one that lets a missing document cost you something.

Two averaging choices hide in that function and both have to be decided once. Averaging the per-query recalls, as written, is macro-averaging and gives every query equal weight regardless of how many relevant documents it has. Summing hits and relevant documents across the whole set first and dividing once is micro-averaging, which lets a query with nine relevant documents count nine times as much as a query with one. Macro is almost always what you want for a golden set, because the unit of user experience is a query and not a document, and the difference between the two on this set was 0.62 against 0.71 — large enough that two engineers reporting "recall" without saying which are simply not talking about the same quantity.

Which of the two comes out higher is a fact about the golden set rather than about the retriever, and knowing why makes it predictable. Micro-averaging is a weighted average of the per-query rates with the weights being each query's count of relevant documents. So micro exceeds macro exactly when the queries carrying many relevant documents are the easier ones, and falls below it when they are the harder ones. On a set built by pooling, queries with many judged-relevant documents tend to be broad questions with several adequate answers, and broad questions are usually the ones retrieval handles well — which is the mechanism that makes the two diverge here rather than a defect in either. Macro is the right default because a user issues a query and experiences its result, and they do not issue documents.

Choosing the two values of k#

Reporting recall and nDCG together implies picking a cutoff for each, and they are rarely the same number. The cutoff for nDCG should be the number of results the consumer actually sees, because positions beyond that are invisible and rewarding them measures something no user experiences. The cutoff for recall should be the depth of the stage you are diagnosing, which for a two-stage pipeline is the candidate depth rather than the final one.

Raising the candidate depth is not free, and the shape of the trade-off is worth measuring rather than guessing. On this stack the cross-encoder scored candidates at roughly 3.1 ms each on a warm GPU, batched, so moving the candidate depth from 50 to 200 took the reranking stage from about 155 ms to 620 ms and pushed the p95 of the whole retrieve-and-rerank path from 190 ms to 680 ms. Those two increases are worth comparing: the reranking stage grew by a little over four and a half hundred milliseconds and the end-to-end p95 grew by roughly the same, which says the extra half second is all reranking and none of it retrieval. When the two do not match, the depth change has moved something else as well — usually the candidate fetch itself — and the diagnosis is a subtraction rather than a profiler. What it bought was recall@200 of 0.96 against recall@50 of 0.93 — three points of ceiling for nearly half a second. Measuring recall at several depths in the same run costs nothing extra, since the ranked lists are already there, and the resulting curve tells you where the depth stops paying.

Every entry in the last column is the per-document reranking cost multiplied by the depth: 3.1 ms at ten candidates is 31, at fifty is 155, at a hundred is 310, at two hundred is 620. That the column is a straight multiple of the depth is the fact worth having, because it means the reranker's cost carries no economies of scale — doubling the depth doubles the bill, every time, with no point at which the curve bends in your favour. The recall column bends sharply and the cost column does not, and a decision about depth is entirely the collision of those two shapes.

The first two rows of the depth table are also the first two recall figures in the four-run table above, which is how you know both are describing the hybrid configuration rather than being two unrelated experiments. Reading them together prices the reranker's whole job: 0.74 is what the top ten contains before reranking, 0.93 is what the top fifty contains, and the nineteen points between them are the entire stock of documents any reordering could ever promote.

The interesting number in that table is the 0.19 between depths 10 and 50, because it is the size of the pool the reranker has to work with, and it is why a two-stage architecture exists at all. Beyond 100 the curve is flat, which says the remaining 4% of relevant documents are not deep in the candidate list; they are absent from it.

Recall gives you a work queue, nDCG does not#

The most practical property of a recall number is that its complement is enumerable. If recall@50 is 0.93, there is a specific list of query-document pairs where a judged-relevant document failed to appear, and reading twenty of them by hand takes an hour and produces a categorised backlog. Doing that on the BM25 baseline's 0.81 gave three groups. Twelve of the 29 misses were documents that were not in the index at all, because the ingestion job skipped PDFs over 10 MB and three of the architecture guides were over the limit. Nine were vocabulary mismatches, where the user wrote throttling and every document said rate limiting. Eight were chunk-boundary failures, where the question was answered by a sentence in one chunk and a table two chunks later, and neither chunk alone scored well enough to surface.

Twelve, nine and eight add to twenty-nine, which is every miss accounted for with nothing left in an "other" pile — and that closure is the property that makes the exercise worth an hour. A categorisation with a residue is a categorisation that has found some causes and is silent about how many remain; one that closes says the failure surface has been fully described at this depth, so the three fixes below are the whole of the work rather than the part that was easy to name.

Those three groups have three different owners and three different fixes, and none of them is a retrieval algorithm. A low nDCG cannot be decomposed this way, because a bad score is spread across every position in the list and there is no equivalent of "the document that did not appear" to go and look at. Ordering failures are diagnosed by reading rankings, which is slower and less conclusive.

Where the pairing stops helping#

Recall@k is not precision and does not constrain it. A run that returns 50 documents of which 3 are relevant and a run that returns 50 documents of which 30 are relevant score the same recall if both contain all the relevant ones, and a downstream generator with a fixed context budget cares enormously about the difference. Recall is also trivially inflatable by raising k, which is why a recall number without its k attached is not a number; recall@1000 approaching 1.0 tells you the corpus contains the documents and nothing about whether any system can find them. The inflation is guaranteed rather than likely, which is what makes it dangerous: a document inside the top fifty is inside the top two hundred, so raising the cutoff can add hits and can never remove one. Recall is therefore non-decreasing in k by construction, and any figure reported without its cutoff has one free parameter that only moves the number one way.

Neither metric knows anything about the question the user actually asked. A run can score recall@10 of 0.95 and nDCG@10 of 0.88 and still produce wrong answers, because the generator ignored the context, or because the relevant document is out of date, or because the passage that got chunked out was the one with the caveat in it. Retrieval evaluation bounds answer quality from above; it does not predict it.

The two metrics also do not settle what goes on the wall. They diagnose different failures, which means both have to be computed and stored, and it does not follow that both belong in front of the whole team every morning — a dashboard with nine numbers on it is a dashboard nobody reads.

The claim that nDCG supersedes recall, on the grounds that it accounts for both presence and position, is wrong for the reason this chapter has been about: nDCG at a fixed cutoff cannot express the candidate-stage ceiling, and a team that drops recall loses the ability to tell a reranking problem from an indexing one. The costs are asymmetric, too. Dropping nDCG means shipping an ordering regression and hearing about it from users; dropping recall means spending a quarter tuning a stage that is working correctly underneath a limit set by another stage entirely.

Which of them earns the headline slot, and what the answer depends on, is decided by something outside the metrics: what the thing consuming the ranked list actually does with it.