Groundedness stays high when nineteen of twenty chunks do not belong

A retrieval-augmented generation system — one that answers a question by searching a body of documents, pasting the passages it finds into the prompt, and asking a language model to answer from those passages — is normally watched through two quality numbers. Recall reports whether the passages that hold the answer were among the ones retrieved. Groundedness, also called faithfulness, reports whether the sentences of the answer are supported by the passages that were retrieved. Both are standard, both were defined in topic 9.10, and both are worth having. Neither can express the complaint users actually make, which is that the system read a great deal of material with no business being read and answered a question adjacent to the one that was asked.

That gap is structural rather than a matter of calibration, and it is the same structure in both cases: each metric asks whether some retrieved passage does a job, and a question of that form only gets easier to satisfy as passages are added. The measurement that asks the opposite question — of each retrieved passage, was this one worth retrieving — is context precision. The figures below come from the forty-query fixture of the previous chapter, run against a small corpus assembled to reproduce this failure, so they are evidence about a mechanism rather than an estimate of anybody's production quality.

A dashboard that has been green for six weeks#

retrieval
  recall@20 .................... 0.94   (target 0.90)
generation
  groundedness ................. 0.91   (target 0.85)
  answer relevance ............. 0.72   (target 0.80)   ▼
serving
  p95 latency .................. 1.9 s
  tokens per query (context) ... 2,510

Two of the three quality numbers are above target and have been since launch. Read them against their targets rather than against each other, and the shape is on the dashboard as printed. Recall clears its target by four hundredths, groundedness clears its own by six, and answer relevance misses its own by eight — a larger distance than either of the others is clearing by. A system with a mild general quality problem does not look like that. A system whose two green numbers are insensitive to a defect its third number is picking up looks exactly like that.

Both green numbers are fully satisfied by the context the manifest printed in the previous chapter: twenty chunks, three of them relevant. Neither score is a mistake and no recalibration would move them. They are the correct scores for a system that read seventeen passages it had no reason to read.

Both metrics are indifferent to the other seventeen chunks#

Recall@20 is a question about membership. It asks whether the chunks that contain the answer appear anywhere in the twenty that were retrieved, and it is satisfied identically whether they arrived at rank 1 and rank 2 or at rank 17 and rank 19. Nothing in its definition refers to the other chunks. A retriever that returned the entire corpus for every query would score recall@N of 1.00, which is the reductio that shows what the metric measures: it bounds what generation could possibly use, and says nothing about what generation has to wade through. That reductio is a general property at its limit rather than a joke — adding candidates can never remove a gold chunk already in the list, so recall cannot fall as k rises, and a score whose maximum is reached by the worst retriever it is possible to build is measuring an upper bound on what was available.

Groundedness is a question about entailment — whether a passage in the context actually says what a sentence of the answer claims — and it was defined in topic 9.10 as the fraction of claims in the answer that are supported by the retrieved passages. Its right-hand side is the retrieved set, which means the metric grades the answer against whatever the retriever handed over, however unsuitable that was. If seventeen of the twenty chunks are about key rotation in some loose sense, and the model composes a plausible procedure out of three of them, every sentence it writes is entailed by something in the context. Groundedness of 0.91 is not a mistake in that situation; it is the honest score for an answer that is faithful to a context it should never have been given.

Take that definition apart and the indifference stops being a surprise. Groundedness splits the answer into the separate claims it makes, and for each claim asks whether some passage in the retrieved context supports it; the score is the count of claims that pass over the count of claims. The denominator is a count of claims, so it is a property of the answer and of nothing else. And the word carrying the weight on the other side is some — a claim passes when at least one passage supports it, and which passage, or how many, is nowhere in the score.

Now split the retrieved context in two. Call the passages that genuinely bear on the question the relevant set and everything else the distractors; the context is the two together. Take a claim that passes, and suppose the passage supporting it sits in the relevant set, which is what it means to say the model composed its answer out of the three chunks that belonged. Put the distractors back. The claim still passes, because the passage that supported it is still there and nothing about it changed. That step is allowed for one reason, worth stating plainly: an existential claim already satisfied by one member of a set stays satisfied when the set grows. Adding passages cannot take a witness away.

Run the argument over every claim and what comes out is an identity rather than a tendency. Groundedness scored against the relevant set alone, and groundedness scored against the relevant set plus any distractors whatsoever, are the same number. The count of distractors appears on neither side. It was not cancelled by an algebraic step; it never entered the expression, which is the stronger statement and the worse one for anybody relying on the metric.

What that licenses is narrow, and stating it exactly is the point of the derivation. Groundedness certifies the generator: it says the model did not assert things its evidence did not contain, which is a real guarantee against a real failure. It cannot certify the retriever, on any corpus, under any configuration, because the size of the retriever's output is a quantity it is provably blind to. A high groundedness score is not weak evidence that retrieval is healthy. It is no evidence at all, and the gap between those two readings is the gap between a team that goes looking and a team that stops.

The identity is also why this chapter's title states the case at nineteen of twenty rather than at the seventeen of twenty this system is running. Nothing in the argument depends on the count. Nineteen distractors, ninety-nine, or a whole corpus of them leave the score where it was, so long as the passages the model drew on are still in the pile.

An analogy, offered as one. Groundedness behaves like a proofreader who checks that every sentence of a report is supported by some document in the box on the desk, and never asks whether the right documents are in the box. It breaks in one place worth noticing: a real proofreader can see the box and would say something if it held the wrong files, whereas the metric is defined so that it cannot, because the box is its right-hand side by construction. There is no vantage point from which groundedness could notice, which is why the fix is a second number rather than a stricter judge.

The perverse consequence is that adding distractors can raise groundedness. More text in the window means more propositions available to entail whatever the model says, so an answer that drifts off-question finds support more easily, not less. The two metrics together therefore have a blind spot with a precise shape: they cannot express that a chunk which is in the context should not have been. Every anti-pattern in this topic lives inside that blind spot, which is why teams running these two numbers can watch them stay green for six weeks while complaints accumulate.

Those are two arguments pointing one way. The identity holds the answer fixed and shows the score cannot fall; the drift argument lets the answer respond to its context and shows the score can rise. Between them there is no room left for groundedness to register the defect, and what results is the cleanest example in this topic of a quantity that stays constant while the dangerous property varies. Hold the useful passages fixed, let the useless ones grow without limit, and the score does not move at all while the fraction of the context that deserved to be there falls towards zero.

The measurement that does move is the same question with the quantifiers the other way round. Instead of ranging over claims and asking whether some passage supports each one, it ranges over passages and asks of each whether it was worth retrieving. The count of passages is now the denominator, so distractors are in it, and each one added drives the score down: relevant over relevant-plus-distractors is a fraction whose bottom half grows while its top half stands still. One swap of what is quantified over, and that swap is the whole difference between a number that cannot see the defect and a number that measures it.

Context precision is the number nobody plotted#

The missing measurement is the fraction of the assembled context that is relevant. In its simplest form it is precision over the chunks that entered the prompt, and it needs the same gold labels the query set already carries.

def context_precision(context: list[Chunk], gold: set[str]) -> float:
    """Fraction of chunks in the prompt that are actually relevant."""
    return sum(c.id in gold for c in context) / len(context)

def context_precision_at_ranks(context, gold) -> float:
    """Rank-weighted variant: rewards relevant chunks appearing early."""
    hits, total = 0, 0.0
    for i, c in enumerate(context, 1):
        if c.id in gold:
            hits += 1
            total += hits / i          # precision at this rank
    return total / max(hits, 1)

The two divide by different things, and that is the reason to keep both. The plain one divides by the length of the context, so it counts a relevant chunk at rank 20 exactly as it counts one at rank 1, and it is the dependence on the distractor count that groundedness lacks. The rank-weighted one divides by the number of hits, so the length cancels and what survives is where in the ordering the hits fell. One is sensitive to how much was sent, the other to how it was ordered, and those are the two things the rest of this topic changes.

Run either over the forty-query fixture under the baseline configuration and the number is stark. Plain context precision comes out at 0.14 — around three relevant chunks in every twenty — and the rank-weighted variant reaches 0.41, because the relevant chunks do tend to land in the top half even when they are outnumbered. Both are worth plotting, and the plain one is the better alarm, since it is the one that tracks how much of the model's attention is being spent on material that cannot help.

Both of those are means over forty queries rather than properties of any one of them, and the difference matters when a reviewer starts reconciling figures. The single key rotation query in the previous chapter put three gold chunks into a context of twenty, a fraction slightly above the fixture mean, and its manifest printed a context token total slightly above the per-query mean on the dashboard. Neither pair contradicts itself. One query and the average of forty are different measurements, and a reviewer who reconciles them by assuming a typo has discarded the only evidence that the fixture and production are being measured the same way.

A context precision of 0.14 also explains why a bigger context window did not help when the team tried it. Distractors are not inert filler that the model steps over on its way to the relevant passage; they compete for attention, and the competition is worst when they are topically adjacent, which is precisely what a dense retriever returns. Running the fixture with the same three gold chunks and then padding the context to twenty with near-miss chunks costs about nine points of answer relevance, from 0.81 to 0.72, with the gold material identical in both arms. An arm, here and for the rest of the topic, is one configuration run end to end over the whole frozen query set, so that two arms differ in the setting under test and in nothing else. Position matters too, and material in the middle of a long context is used less reliably than material at either end, so a relevant chunk sitting at rank 11 out of 20 is being read under the least favourable conditions the assembly can produce.

The value that padding experiment lands on is worth pausing over. The padded arm comes out at 0.72 answer relevance, and the production dashboard — computed from live traffic, on a different corpus, by a different job — also reads 0.72. Those two instruments share no step. Landing on one number licenses a single claim and no more: the mechanism reproduced in the fixture is the mechanism operating in production, which is what a fixture exists to establish and what argument cannot. It licenses nothing about how large the effect is elsewhere, and a second coincidence would be needed before anybody should expect the agreement to survive a change of domain.

The distinction matters for how you talk about the failure. Retrieval is not broken in the sense of failing to find things; it found the drain step and ranked it second. Retrieval is broken in the sense that it also found seventeen other things and the system put all of them in the prompt, and the model, reading a context that is 14% signal, reconstructs a procedure that is mostly right and missing a step. This is the diagnosis to give when someone reports that groundedness looks healthy and answers are about the wrong thing: context precision is low, the candidates at ranks five through twenty are topically adjacent and irrelevant, and the model grounds itself faithfully in whichever of them it attended to.

Cutting k is not the same intervention as ranking better#

The obvious response to a context that is 14% signal is to put less in it, and dropping k from 20 to 5 takes one line of configuration and no new infrastructure. It is worth running on the fixture before adopting, because it moves two numbers in opposite directions.

The first row of that table is the dashboard. Recall, groundedness, context tokens and p95 latency all reproduce the four figures printed at the top of this chapter, so the baseline arm is pinned to production on four independent columns. That pinning is what makes the other two rows readable: a difference between the baseline row and another row is attributable to the switch that changed rather than to the fixture being some other system.

Truncating to five keeps the same ordering and simply throws away ranks six to twenty, so a third of the queries lose the chunk that held their answer. Precision roughly doubles because the denominator shrank, and the system is now worse: it answers fewer questions correctly and it answers them from a context that is still 71% noise. The failure has changed shape rather than gone away, and it has changed into a shape the dashboard is even less able to see, because recall was the one number that would have caught it and it is measured at k, so a team that cuts k and keeps reporting "recall@k" is comparing 0.61 against 0.94 while calling both metrics by the same name.

"The denominator shrank" is half of it, and the missing half is the interesting half. Truncating to five keeps a quarter of the slots. Had the relevant chunks been spread evenly across the twenty ranks, a quarter of them would have survived too, top and bottom of the fraction would have shrunk by the same factor, and precision would not have moved at all. It roughly doubles instead, so about half the relevant chunks are in the top five while only a quarter of the slots are. The factor by which precision rises under truncation is exactly the factor by which relevant chunks are over-represented near the top, which is the only thing that movement measures. It is a statement about the ordering, and the ordering is what a reranker changes.

The recall column closes on the same subtraction rather than adding a second fact. Recall in context falls by a third under truncation, which is the figure already reported above as a third of the queries losing the chunk that held their answer; since recall here is a fraction of queries, a difference of fractions of queries already is a count of queries and needs no further step.

The third arm does something different. It widens the candidate set to fifty, then applies a cross-encoder that scores each candidate jointly with the query rather than comparing two independently produced vectors, and keeps the top five of those. Recall stays close to the wide-candidate number because the reranker is choosing from fifty rather than from twenty, and precision quadruples because the ordering it produces is genuinely informative in a range where cosine similarity was compressed into five hundredths.

That arm changes two things at once and the table says which column each moved. Widening the pool protects recall: the third arm gives up three hundredths against the baseline where the truncated arm gives up thirty-three, and the only difference between those two five-chunk arms is the size of the pool the five came from. Reranking buys the precision, since both arms send five chunks and only one sends five chosen by a model that read the query and the chunk together. Calling the arm "the reranker" credits one mechanism with the work of two, and a team that adds a reranker without widening k gets the precision and keeps the recall loss.

The token column is the one to check before trusting the rest, because it is where an assembly step dropping or duplicating chunks would show. Divide each row's tokens by the number of chunks it kept and the three arms agree on a per-chunk cost to within a few per cent of the configured 128-token chunk size, which licenses reading a change in that column as a change in k rather than a change in what is being sent. The residual is small and real: the two arms that both send five chunks differ in tokens, and since nothing about the assembly differs between them, the difference is in which five were selected. A scorer that reads the chunk text prefers chunks with more of it.

What the reranker costs, stated in full#

Naming a fix without pricing it is how a design review produces a decision nobody can implement. A reranker is a second model on the request path, and the bill has four lines.

The second row checks the first rather than asserting a shape. Linear in k means added latency divided by k comes out about the same at every k, and the three quoted points do: twenty, fifty and a hundred candidates all land near two milliseconds a candidate. The two small departures are informative. The low end runs dearer per candidate because a batch has to be assembled and shipped whatever its size, and that fixed cost is spread over fewer candidates; the high end runs dearer in total than a straight line through the other two points predicts, which is what a batch of a hundred costs a six-layer model on one card that a batch of fifty does not. Neither breaks the linear reading, and that is what licenses pricing a k you have not measured by multiplying.

The arms table has a residual in its latency column that this one only partly accounts for. The reranked arm sits two tenths of a second above the baseline at p95, and the cross-encoder's own p95 contribution is 140 ms. The remainder is unattributed by anything printed here, and there are two candidates for it: the wider retrieval, fetching and scoring fifty vectors instead of twenty, and the change in prompt length, which pushes the other way. Report the reranker's share as the reranker's cost and leave the rest labelled as the cost of the arm's other change until somebody measures it.

Against that, the third arm sends 655 context tokens instead of 2,510. At $3 per million input tokens that is a saving of about $0.0056 per query, or $5.60 per thousand, which more than covers a hosted reranker at $1–2 per thousand and pays for the GPU node outright at any serious volume. The honest version of the trade is that you are spending 95 ms and one service to save 1,855 tokens and recover the precision, and whether that lands depends on your latency budget rather than your cost model.

That saving closes on itself with no term left over: the token figure is the subtraction between the two arms and nothing else, the per-query figure is that difference at the stated price per million, and the per-thousand figure is a thousand times the per-query one. Two conditions belong on it before it goes into a review. It is a saving on input tokens at one assumed price, so a cheaper generator shrinks it in proportion and a dearer one widens it. And the GPU option is a fixed monthly cost rather than a per-query one, so "pays for the GPU node outright at any serious volume" needs a volume attached before anybody can check it — above some query rate the saving covers the node, and below it the saving does not. Ask for the rate.

The fallback line deserves the attention it rarely gets. When the reranker times out, the system must decide between answering from the raw top five by embedding distance and failing the request, and if nobody decides, the framework decides, usually by silently returning the unranked order. That means the anti-pattern you removed reappears during exactly the incidents when people are watching the assistant most closely, and it will not show up in the aggregate metrics because reranker timeouts are a small fraction of traffic. Emit the reranker's status in the trace so the manifest records which path a given answer took.

Where precision stops being the right thing to chase#

Context precision is not answer quality, and optimising it alone leads somewhere silly. A system that returns exactly one chunk per query and gets it right two thirds of the time scores 0.67 precision and fails a third of the questions outright; the k=5 arm above is a mild version of the same trade. The pair of numbers is what carries meaning, and chapter 7 works through how to report a change that moves both of them at once so that a precision win concealing a recall loss cannot be read as success.

Pairing them is forced by the same asymmetry the derivation turned on. Precision has a denominator you control directly — send fewer chunks and it rises, whatever the retriever did — while recall has a denominator you do not, since the gold set for a query is a property of the corpus and the question. A number that can be improved by sending less has to be reported beside a number that gets worse when you send too little.

There is also a class of failure that no reranker touches, and the fixture manifest already showed one. Eight of the eleven chunks holding the key rotation procedure never appeared in the candidate set at all, at k=20 or k=50, so no amount of reordering can bring them into the context. A reranker is a permutation of the candidate list, and a permutation cannot introduce a member the list did not have, which is the precise sense in which reordering and retrieving are different operations. Those eight exist as separate chunks only because the ingest split the runbook at 128 tokens, and each of them mentions key rotation once in passing while the query's wording matches just one of them. That mechanism is the next thing to trace.

Both green numbers on that dashboard are checking one stage twice. Recall bounds what the generator could have used; groundedness confirms the generator did not go beyond what it was given. Both ask whether something in the retrieved set does a job, both are therefore satisfied more easily as that set grows, and neither carries a term counting what was retrieved and wasted. That is not a flaw in either definition — it is what they were built to measure — and the error lies entirely in reading them as coverage of the whole system.

The measurement that closes the gap ranges over retrieved passages instead of over claims, and it costs nothing new to compute: the gold labels are already written and the manifest already lists what entered the prompt. Plot it beside the other two and the six-week stretch of green becomes a six-week stretch of one number sitting at 0.14 while the two above it sit above target.