Cancellation in an aggregate score

Cancellation is what happens when two changes land in the same release and act on different terms of the same expression. It is not a rare coincidence — releases bundle work, and the work that lands together is usually owned by different people whose changes move different stages — and it is invisible in exactly the number most teams sign off on.

Take one release pair where it happened. The arithmetic reproduces both releases from four measurable conditional rates, and a single query either confirms the mechanism or kills it. Netting two component numbers against each other repeats the composite's error by hand, and end-to-end evaluation stays worth paying for once both stages have their own metrics.

Three numbers behind one sentence#

The release notes for build B say quality improved slightly. Behind that line, on the same 400-question eval set, are three numbers.

Fifteen points of recall is about 60 of the 400 questions that used to arrive at the generator with their evidence and now do not. Sixteen points of groundedness is a large, real change in how faithfully the model treats what it is given. The composite moved one point, and one point on a 400-item set is inside the noise. Signing off on that release on the strength of the composite means shipping a substantial retrieval regression with a note that says quality improved.

Two changes landed in the same window and they act on different terms of the same expression. The decomposition from chapter 7, on sequencing component evals — the end-to-end pass rate as the recall-weighted mix of the pass rate with gold evidence present and the pass rate without it — reproduces both releases from four measurable conditional rates.

# A: chunker at 800/128, reranker v1, generator prompt v4
project(recall=0.81, p_pass_given_gold=0.86, p_pass_without_gold=0.44)   # -> 0.780

# B: chunker at 1500/0, reranker v1, generator prompt v5 + abstention line
project(recall=0.66, p_pass_given_gold=0.93, p_pass_without_gold=0.51)   # -> 0.787

The retrieval change lowered the weight on the strong term from 0.81 to 0.66. The generation change raised both conditional rates, from 0.86 to 0.93 on questions with evidence and from 0.44 to 0.51 on questions without. The weighted mix comes out one point higher, which is the composite's entire story: a real regression averaged against a real improvement, with both quantities discarded in the process of taking the average.

Multiplying those out term by term is what makes "cancelling" a mechanism instead of a metaphor, and it takes four multiplications. In release A the strong term is the 0.81 share passing at 0.86, which contributes just under seven tenths of the final figure, and the weak term is the remaining share passing at 0.44, which contributes a little over eight hundredths. Those two add to 0.780. In release B the strong term is the 0.66 share passing at 0.93, which contributes a shade over six tenths, and the weak term is the remaining share passing at 0.51, which contributes a little over seventeen hundredths. Those two add to 0.787.

Now compare the terms rather than the totals, which is the step the composite makes impossible. The strong term fell by roughly eight hundredths between the releases. The weak term rose by roughly nine hundredths. Both movements are large — each is around a tenth of the whole score — and they point in opposite directions, so their sum is smaller than either of them. That is cancellation stated exactly: two quantities of comparable size and opposite sign, added, leaving a residue that resembles neither. Nothing was hidden by anybody. Addition discards the addends, and a release note that reports only the sum is reporting the one number from which the addends cannot be recovered.

The groundedness rise has a second contributor worth naming, because it makes the situation slightly worse than the numbers suggest. Groundedness is the fraction of an answer's claims that the supplied context entails, and an abstention makes no factual claims, so the implementation from chapter 3, on retrieval recall and answer groundedness, returns 1.0 for "I don't know" by way of the empty-claims branch. Build B added the abstention instruction, and it added it in the same release that stopped supplying evidence for 60 questions. A portion of the sixteen-point groundedness gain is therefore the model correctly declining to answer questions it can no longer answer, scoring a perfect faithfulness on each refusal. The metric is behaving as specified. The specification rewards a system that has become less useful.

That empty-claims branch is worth tracing back to its reason, because it is not a bug and the consequence is not an accident. Groundedness divides the entailed claims by the total claims, and an abstention makes no claims, so the expression is zero over zero and something has to be returned. Returning 1.0 is the defensible choice: an answer that asserts nothing has asserted nothing unsupported, and returning zero instead would penalise a system for declining to invent. The metric does exactly what its definition says. What the definition does not contain is any notion of usefulness, which is why a system that answers fewer questions and refuses more climbs the groundedness scale by refusing, and why "the metric is behaving as specified" is the beginning of the problem rather than the end of it.

The one slice that decides it#

The explanation above is a hypothesis, and it makes a prediction sharp enough to be killed by a single query. Take the questions where the gold chunk reached the model's context in A and did not in B, and look at nothing but their end-to-end scores across the two builds.

lost = [q for q in questions
        if gold_in_context(q, run_a) and not gold_in_context(q, run_b)]

score_a = mean(run_a.e2e[q.qid] for q in lost)
score_b = mean(run_b.e2e[q.qid] for q in lost)
held    = [q for q in questions if q not in lost]
print(len(lost), score_a, score_b,
      mean(run_a.e2e[q.qid] for q in held), mean(run_b.e2e[q.qid] for q in held))

Run on this pair of builds it returned 61 questions, scoring 84 in A and 49 in B, while the remaining 339 went from 77 to 84. That is the mechanism confirmed with numbers attached: a 35-point collapse on a sixth of the eval set, paid for by a 7-point gain on the rest, netting to plus one. The release notes describe the net.

Those four figures rebuild both headline scores, which is the check that the slice is the whole story rather than a part of it. In A, 61 questions at 84 together with 339 at 77 average to a shade over 78. In B, the same 61 at 49 together with the same 339 at 84 average to a shade under 79. And the reason the mean rose while the larger per-question movement was downward is a comparison of two totals: seven points gained on 339 questions is a bigger total than thirty-five points lost on 61, because 339 is more than five times 61 while 35 is only five times 7. The difference between those two totals, spread across the 400 questions, is about six tenths of a point, which is the +1 in the table once both figures are rounded to whole numbers.

The intuition worth naming now the arithmetic is done is that a mean keeps no record of how many questions each unit of score came from. Thirty-five points lost on one question and one point lost on thirty-five questions are the same contribution to an average and completely different events for the people who asked. Support hears about the first kind and never about the second.

The same slice is what would have killed the hypothesis. If those 61 questions had scored roughly the same in both builds, the recall drop would not be causing user-visible harm, and the most likely explanation is a measurement artifact — the chunking change in B re-cut the indexed documents, and gold labels stored as chunk IDs from the A-era index would point at chunks that no longer exist, manufacturing a recall drop out of a bookkeeping failure. That check costs one query and it is the difference between blocking a release and discovering that your labels expired. Where the labels are stored as document offsets and re-resolved per build, as chapter 2, on the stages of a RAG pipeline, recommends, this failure mode does not arise, and a recall drop can be read at face value.

Netting the two component numbers is the same mistake again#

The tempting sign-off is that sixteen points of groundedness is a bigger win than fifteen points of recall is a loss, which is why the composite went up, so ship it. That argument nets two quantities that are not on a common scale and that are not even about the same population of questions.

Recall@5 is a proportion of questions; groundedness is a proportion of claims within the answers that were produced. A fifteen-point recall loss removes evidence from a specific, identifiable set of 61 questions, which then get answered worse or not at all — the slice above prices that at 35 points of answer quality each. A sixteen-point groundedness gain is spread across the answers that still got produced, including the refusals, and improves them by an amount the composite already counted in the 7-point rise on the held set. Adding a per-question loss to a per-claim gain and calling the sum positive is the same operation the single composite performed, done by hand with extra steps. The reason for measuring stages separately is to stop performing it.

Spelling out the two units makes the mismatch hard to talk past. Fifteen points of recall counts questions: it is 60 identifiable rows of the eval set, each of which used to have its evidence and now does not, and each of which can be opened and read. Sixteen points of groundedness counts claims: it is a shift in the fraction of individual assertions, drawn from a pool whose size depends on how many sentences each answer happened to contain and which includes every refusal as a perfect score with nothing in it. There is no conversion between the two. Asking whether sixteen of the second outweighs fifteen of the first is asking whether sixteen of one thing is more than fifteen of a different thing, and the fact that both were written as two-digit numbers on the same slide is the only reason the question sounds answerable.

The correct posture on a release with this shape is that it contains one regression and one improvement, both of which are real, and that they are independent decisions. The generation change is worth keeping. The retrieval change is worth reverting or fixing, and the fact that its damage is currently masked in the aggregate makes it more urgent rather than less, because a masked regression accumulates: the next release's baseline is 0.66, and the recall loss becomes the new normal that nobody remembers deciding to accept.

The accumulation is worth following one step further, because it is what makes "more urgent rather than less" a claim about mechanism rather than about tone. Every release is compared against the one before it. Once 0.66 is the recorded value for build B, the comparison the next release runs is against 0.66, and a build that holds at 0.66 reports no change. The 0.81 is still in the history and it is no longer in any comparison anyone looks at. A regression that survives one sign-off therefore stops being a regression and becomes the baseline, not through anybody's decision but through the arithmetic of comparing consecutive builds. The only thing that reverses it is a gate with an absolute threshold or a comparison drawn against something older than last week.

Reporting that does not permit the same description twice#

The release note said "quality improved slightly" because the reporting format had one field for quality. Four changes to how results are published prevent the same sentence from being written again.

Report the stage metrics alongside the composite, always, with sample sizes, and never publish the composite alone. A release note that reads end-to-end 78 → 79 (n=400); recall@5 0.81 → 0.66; groundedness 0.72 → 0.88 cannot be summarised as a slight improvement by anyone who reads it.

Gate on each stage metric independently rather than on the composite. A drop of more than one standard error in recall@5 blocks the release regardless of what the composite did, because the composite is not evidence about retrieval. On 400 questions at a proportion near 0.8, that threshold is roughly 0.02, so a fifteen-point fall is not a marginal call.

Publish the movers, not only the means. The slice query above should run automatically on every eval comparison: the questions that changed state at each boundary, their count, and their end-to-end scores in both builds. Most of what a mean hides is recoverable by listing what moved.

Report the distribution alongside the average. The share of questions scoring below the product's acceptability threshold went from 9% to 17% across these two builds while the mean rose, and that share is closer to what support experiences than the mean is, because escalations come from the bottom of the distribution and not from its centre.

Running the two kinds of eval at different cadences#

The reporting changes above are affordable because the component evals and the end-to-end eval do not need to run at the same frequency, and pretending they do is what makes teams run neither. Retrieval recall over 400 questions costs about three cents and fourteen seconds, so it belongs in the pre-merge check on any pull request touching ingest, embedding, retrieval or reranking configuration, where it blocks a chunking change before it reaches a build. Citation structural checks are free and deterministic, so they run in the same check. Groundedness needs the answers, so it runs nightly with the full pipeline pass, at a few dollars a night. The full end-to-end judge run with reference comparison is the expensive one and runs per release candidate, plus on demand when a component gate trips and somebody needs to know how much the user actually lost.

Gates have a false-positive rate, and stages have blind spots#

There is a limit to how far the gating arrangement scales, and it is worth stating so the arrangement is not oversold. Every gate is a threshold, every threshold has a false-positive rate, and a team that adds five gates at one standard error each will trip one on roughly a third of releases through noise alone. The compounding is the part that surprises people, and it is one multiplication: the chance that no gate trips is the chance each individual gate holds, multiplied together once per gate, and a handful of numbers each slightly below one multiply to something a good deal further below one. Each gate looks reasonable in isolation and the set of them does not, which is a property of the set rather than a fault in any member of it.

Gates on stage metrics are worth their friction when they are few, when their thresholds are derived from the eval set's actual variance rather than picked round, and when tripping one triggers the slice report rather than an automatic block — the slice is what tells you whether a two-point recall movement cost any user anything.

With retrieval recall and generation groundedness both in place, the question of why to keep paying for full end-to-end runs is fair — they are the slowest and most expensive part of the harness, and the component metrics are the ones that produce actionable diagnoses. The answer is that both stages can pass their own metric while the assembled answer fails the person who asked. Every failure in the following list is invisible to both component metrics and visible only at the end.

The general reason is that component metrics are proxies you chose, and each one is a partial specification of what its stage should do. Anything you did not think to specify falls between the stages, and the only measurement positioned to catch it is the one taken where the user stands. The end-to-end score has no address, which was the complaint chapter 1, on end-to-end and component metrics, opened with, and it also has no blind spots of the kind above, which is the property none of the component metrics have.

That gives the working arrangement. The end-to-end score is the alarm and the number the product is judged on; the component metrics are the address, and they are what a failing alarm gets resolved with; the traces turn a rate into a specific request you can re-run. The gates run on all of them independently, so that a movement at any boundary has to be acknowledged rather than averaged away.

Where to take it next is a matter of applying it in the order this topic developed. Record the intermediate artifacts on every request, since nothing else is recoverable afterwards. Build labels as document offsets for a few hundred real questions, and from them the retrieval eval that runs for cents. Use the traces to attribute a batch of failures to boundaries, and let that distribution — not the diagram, not the stage that is easiest to change — decide where the next eval and the next quarter go.