Retrieval recall and answer groundedness

The two failure modes a retrieval-augmented system has are a passage that never arrived and a passage that arrived and was not used. They belong to different stages, they are fixed by different work, and a single answer-quality score cannot tell them apart. This chapter defines the two metrics that can, states precisely what each one is silent about, and sets out the four cells a single failing question falls into once both numbers exist for it.

Both metrics are cheap. One needs no model call at all and the other needs one judge call per answer, which makes them affordable on every build rather than once a quarter — and a metric that runs only when someone remembers to run it cannot participate in a release decision. What they cost instead is a set of labelled relevant passages for the first and a calibrated judge for the second, both investments made once and reused.

Recall@k is a set comparison, and it needs no model#

For a question q with a labelled gold set G(q) of chunk IDs that contain the answer, and a retrieved list R(q) truncated at k, recall@k is the fraction of gold chunks that appear in the top k. Averaged over the eval set, that is the number reported as "recall@5 = 0.79".

def recall_at_k(retrieved: list[str], gold: set[str], k: int) -> float:
    if not gold:
        return float("nan")            # unanswerable question: exclude, do not score 0
    return len(set(retrieved[:k]) & gold) / len(gold)

def eval_retrieval(questions, retrieve, k=5):
    per_q = {q["qid"]: recall_at_k(retrieve(q["text"]), set(q["gold"]), k)
             for q in questions}
    return sum(per_q.values()) / len(per_q), per_q

One question, worked in full, is worth more than the definition on its own. Take a question whose gold set holds two chunk IDs, and whose retriever returns a ranked list in which the first of those two sits third and the second does not appear at all. Truncating at k = 5 keeps the opening five candidates and discards the rest, so the set being scored is those five. Intersecting them with the gold set leaves one member, the one that ranked third. Dividing that one by the two chunks in the gold set gives a half, and a half is what this question contributes to the average. Note what the truncation did and did not do: it is the reason a chunk sitting at position six counts for nothing, and it is also the reason recall@5 and recall@50 can be computed from the same retrieved list without re-running anything.

The nan on the empty gold set is not a defensive flourish, and the alternative is worth being explicit about. A question nobody can answer from the corpus has an empty gold set, so the intersection is empty and the division is zero over zero. Returning 0.0 there would be a claim that retrieval failed, which is false — retrieval was asked for evidence that does not exist. Averaging that false zero in drags the reported recall down by an amount that depends on how many unanswerable questions somebody happened to include, which makes the metric a function of the eval set's composition rather than of the retriever. Returning nan and excluding the question keeps the average over questions the retriever could have got right.

Two things about that function matter more than the arithmetic. It calls retrieve and nothing else — no generation, no judge, no tokens billed for output — so a full run over 300 questions takes about eleven seconds against a warm index and costs the price of 300 query embeddings, on the order of two cents. And it returns the per-question dictionary alongside the average, which is what lets you ask later which specific questions moved rather than only by how much the mean did.

Where teams get recall wrong is in the definition of the gold set when a question has several supporting passages. If G(q) has three chunks and two appear in the top 5, the function above returns 0.67, which treats the question as two-thirds answered. Frequently that is wrong, because one of the three was sufficient on its own and the other two were corroborating. The alternative — score 1.0 if any gold chunk appears — is the hit rate, and it is the honest metric when your labels mark "chunks that would let you answer" rather than "chunks that are jointly required". Pick one, write down which, and do not let the two drift between releases, since a team that quietly switches from mean recall to hit rate will report a jump of eight to twelve points that corresponds to no change in the system.

Where that jump comes from is visible in the same worked question. Under mean recall it scored a half, because one of its two gold chunks arrived. Under hit rate it scores a full one, because at least one arrived. Every question with more than one gold chunk and at least one hit scores strictly higher under hit rate, every question with exactly one gold chunk scores identically under both, and no question scores lower. So the two averages cannot cross: hit rate is greater than or equal to mean recall on any label set, and the size of the gap is set by how many of your questions carry multiple gold chunks and how thoroughly they are retrieved. That is why the switch shows up as a clean improvement with no release behind it, and why the number to write down alongside the metric name is the average size of the gold set.

Recall is one of a family, and it is worth being clear about why it is usually the right member of it for this job.

With k = 5 and all five chunks concatenated into the prompt, the generator sees the fifth chunk as well as the first, so the ranking metrics that discount by position are measuring something the downstream stage does not care much about. That said, position is not free — long-context models attend unevenly across a prompt, and a gold passage at position 5 is used less reliably than the same passage at position 1. The isolation test in chapter 5, on attributing a failure to a stage, is what turns that from a suspicion into a measurement on a specific question.

Groundedness is scored against the context, not against the world#

Groundedness — the same property is called faithfulness in several eval libraries — decomposes an answer into atomic claims and asks, for each one, whether the supplied context entails it, meaning whether the context states it outright or leaves no room for it to be false. The score is the fraction of claims that are entailed. It is computed by a judge model, since entailment over free text is not something a set operation can do.

GROUNDEDNESS_PROMPT = """You are checking whether an answer is supported by the
context it was given. Do not use outside knowledge. Do not judge correctness.

CONTEXT:
{context}

ANSWER:
{answer}

Split the answer into atomic factual claims. For each claim, decide whether the
CONTEXT alone entails it. Return JSON:
{{"claims": [{{"claim": "...", "entailed": true|false, "evidence": "<quote or null>"}}]}}
"""

def groundedness(answer: str, context: list[str], judge) -> float:
    result = judge(GROUNDEDNESS_PROMPT.format(context="\n\n".join(context), answer=answer))
    claims = result["claims"]
    return sum(c["entailed"] for c in claims) / len(claims) if claims else 1.0

Two instructions in that prompt are load-bearing. Do not use outside knowledge stops the judge from marking a claim entailed because the judge itself believes it, which is the single most common way a groundedness implementation degrades into an expensive correctness check with a lower ceiling. Do not judge correctness stops it from penalising an answer that is faithful and wrong, and you want such an answer to score 1.0, because the whole point is that the fault lies upstream.

The first of those two instructions is doing something more specific than tightening a rubric, and it is worth following through. The judge is a language model, so it arrives with opinions about audit logs and refund windows of its own. Entailment asks a question about a relationship between two texts: given only what this context says, is this claim forced. A judge consulting its own beliefs answers a different question — is this claim true — and the two come apart exactly on the cases the metric exists for. An answer that states a correct fact the context never mentions is entailed by nothing and true anyway; a judge grading on truth marks it supported and the metric stops distinguishing "used the evidence" from "happened to be right". Worse, the substituted question has a lower ceiling than the one you asked, because the judge's beliefs about your product are whatever a general model absorbed about products in general, whereas its ability to check one paragraph against another is a skill it actually has.

Requiring a verbatim quote in the evidence field is the enforcement for that instruction rather than a separate idea. A claim the judge believes but cannot quote produces an empty quote, and an empty quote is a visible failure. A claim the context genuinely states produces the sentence that states it. The check turns an instruction the judge may or may not follow into an artifact you can read, which is why it moves agreement rather than merely sounding stricter.

Here is what that looks like on a question drawn from a set where groundedness came back at 0.93 overall and the generator had been declared healthy:

Q: "How long are audit logs kept on the Team plan?"
context[0] (c-9930, docs/security/audit-logs.md):
  "Audit log retention on the Enterprise plan is 90 days. Logs older than
   the retention window are deleted nightly and cannot be restored."
answer: "Audit logs are retained for 90 days, after which they are deleted
         nightly and cannot be restored. [c-9930]"
groundedness: 1.00
correct: no

Every clause in that answer is supported by the passage in front of it. The passage is about a different plan. The generator did exactly what a generator is supposed to do and produced a wrong answer, and the metric that scores generation reported a perfect result, because groundedness is a claim about the relationship between an answer and its context and says nothing at all about whether the context was the right context. This is not a flaw to be patched out of the metric. It is the property that makes the metric useful: precisely because it holds the context fixed, it isolates generation from everything upstream of it.

Groundedness costs roughly one judge call per answer with the answer and the context in the prompt, so around 3,000–5,000 input tokens per question. Over 300 questions on a small judge model that is a few cents and two or three minutes; the generation calls being judged cost more than the judging does.

The metric answers exactly one question: given this context, did the model stay inside it. It will not tell you whether the answer addressed what was asked — an answer that faithfully summarises an irrelevant passage scores 1.0 and helps nobody, which is why answer relevancy is a separate score. It will not tell you whether an unsupported claim is false, only that the context does not support it, and a model that correctly answers from parametric knowledge when retrieval came back empty is penalised by this metric while helping the user. And it will not catch omission, since a claim that was never made cannot be unentailed.

Calibrating the judge that produces the number#

A groundedness score is the output of a model, so it inherits that model's errors, and a component metric nobody has checked is a component metric that will lose an argument the first time it says something inconvenient. The check is small: take 50 answers spanning the score range, have a person mark each claim as entailed or not by the supplied context, and compare against the judge's per-claim verdicts. On the pipeline in this chapter that produced agreement on 88% of claims, with the disagreements concentrated in one place — the judge marked claims entailed when the context implied them loosely, for example accepting "logs are deleted automatically" from a passage that says logs are removed at the end of the retention window. Tightening the prompt to require a verbatim supporting quote in the evidence field moved agreement to 94% and dropped the average score by 0.06, which is a real reduction in the number and an increase in what it means.

Run that check once before publishing the metric and again whenever you change the judge model, because a judge upgrade shifts every historical score and makes cross-release comparisons meaningless without a re-run of the baseline. Storing the judge model name and prompt hash alongside each score is what makes that re-run possible rather than a guess.

The metrics adjacent to these two#

Three scores get confused with groundedness often enough to be worth separating explicitly. Answer relevancy asks whether the answer addresses the question, independent of both context and truth, and it is what catches the faithful summary of an irrelevant passage. Answer correctness compares the answer to a reference answer written by a person, which is an end-to-end metric rather than a component one, since it is a judgement about the final output against the world. Context relevancy, sometimes called context precision, asks what fraction of the supplied chunks were actually useful, and it is a retrieval-side metric that behaves very differently from recall: adding three irrelevant chunks leaves recall unchanged and halves context relevancy.

That last claim is small enough to check by hand and it makes the difference between the two families concrete. Suppose three chunks are supplied and all three are useful. Context relevancy is the useful fraction of what was supplied, so it is three out of three, a perfect one. Add three irrelevant chunks and the useful count is still three while the supplied count is six, so the fraction is three out of six, a half. Recall meanwhile counts how many gold chunks appeared, and the three additions contain no gold and displace none, so its numerator and its denominator both sit exactly where they were. The two metrics have different denominators — one counts what you were looking for, the other counts what you sent — and that is the whole of why one is blind to padding and the other is measuring almost nothing else.

The reason to keep them apart is that they fail in opposite directions and a team that reports one under another's name will make the wrong call. A system that abstains on everything scores 1.0 on groundedness and near 0 on relevancy; one that answers everything from parametric knowledge scores well on correctness for common questions and poorly on groundedness. Recall and groundedness are the pair this topic leans on because between them they cover the two stages that carry defensible metrics.

The four cells a failing question falls into#

With both metrics in hand, a single failing question falls into one of four cells, and each cell has a different owner.

The two escalations from chapter 1, on end-to-end and component metrics, sit in opposite corners of that table: the audit-log question in the bottom-left, faithful to an Enterprise passage while the user asked about Team, and the /v1/exports question in the top-right, with the correct chunk at rank 1 and an answer that contradicted it. A single helpfulness score cannot separate those, and either metric alone gets one of them wrong — groundedness passes the first, recall passes the second.

What neither metric covers#

Both of these are stage metrics, and stage metrics carry an assumption that the stage boundary is real — that "retrieval" has an input you can specify, an output you can call right or wrong, and labels obtainable at that point. Retrieval and generation both clear that bar, which is why this chapter had two metrics to write down. Query rewriting, chunking and citation formatting sit in the same diagram and do not all clear it, and instrumenting a stage that cannot define its own correct output produces a number that moves without meaning anything.

Neither metric is a statement about the user, either. A question can score 1.0 on both and still leave the person who asked it no better off, because the answer addressed something adjacent, arrived in a format the surface cannot render, or omitted a caveat the context carried. That is why the end-to-end score stays on the dashboard alongside them.

What the pair buys is attribution. Recall is a set comparison against labels, costs no inference, and says whether the evidence arrived; groundedness is one judge call per answer, holds the context fixed on purpose, and says whether the evidence was used. Read together they place a failing question in one of four cells with a different owner in each. Read singly they mislead in opposite directions, and the 1.00 on a wrong answer above is the cleanest demonstration available that a stage passing its own metric says nothing about the stage next to it.

The test for which boundaries can carry a probe at all is next, because the diagram has more boxes than it has measurable seams.