Per-case deltas and cancellation

The aggregate score of an arm is a sum over per-case scores. It is the quantity the eval was built to report, the quantity every dashboard shows, and the quantity a well-run ablation is designed to make interpretable — and it is also the one statistic guaranteed not to show a block that helps one kind of case and harms another by the same amount. Doing everything right up to the last step and then reading the wrong number out of a correct experiment is the common failure here, rather than sloppiness in the experiment itself.

Four quite different systems produce a delta of approximately zero on a 500-case binary set, and only the first of them justifies deleting anything.

The last column is the one to derive rather than accept, because everything after it depends on seeing why two very different rows print the same number. On a binary set the aggregate is the count of passing cases divided by the size of the set, so the change in the aggregate is the change in the count of passes, divided by the size of the set. A case that scored the same in both arms contributes nothing to that change, whichever value it scored. A case that improved adds one pass; a case that worsened removes one. The delta is therefore improved minus worsened, over five hundred. Row two: six minus five is one, over five hundred, which is the two thousandths shown. Row three: forty-one minus thirty-nine is two, over five hundred, which is four thousandths. Row four: one hundred and twenty-one minus one hundred and nineteen is also two, so it prints the identical four thousandths while moving three times as many cases. The statistic contains no term at all for how many cases moved. It has a term only for the difference between the two directions.

The first row is the only one where "no measurable quality impact" is an accurate description of the system. The second row is the one the churn floor from the previous chapter teaches you to recognise and discard. The third and fourth rows are what this chapter is about: the block is doing a great deal, and the reported statistic cannot see it. A team that deletes on row three has not made a neutral change; it has made a change that moves 80 cases and does not know which 80.

The reading, in order#

Once the aggregate is unchanged, there is a fixed sequence of questions, each of which either terminates the investigation or hands you to the next one.

Row three of the table walks straight down the right-hand branch. Eighty discordant cases against a floor of eleven is a little over seven times the noise the harness produces against itself, so the first gate is cleared without argument. The join that produces those eighty has to be exact before any of it means anything: same case_id on both sides, the same repeat aggregation applied to each arm, and no case appearing in one arm and not the other. McNemar's test on 41 against 39 returns p = 0.91, which says the split is as even as a coin, so there is no directional claim to make about the block overall. That leaves the interesting question: are the improvements and the regressions spread evenly across the set, or do they land in different places?

Grouping the movers#

Answering that needs one query, provided the per-case rows carry a slice label — the named subset of the eval set a case belongs to, which is why the next chapter on instrumentation exists.

SELECT a.slice,
       SUM(CASE WHEN b.score > a.score THEN 1 ELSE 0 END) AS improved,
       SUM(CASE WHEN b.score < a.score THEN 1 ELSE 0 END) AS worsened,
       COUNT(*)                                            AS cases
FROM   results a
JOIN   results b USING (case_id)
WHERE  a.arm = 'with-fewshot' AND b.arm = 'without-fewshot'
GROUP  BY a.slice
HAVING improved + worsened > 0
ORDER  BY improved + worsened DESC;

That is a self-join, and it is worth reading as one. results appears twice under two aliases, and USING (case_id) pairs each case's row on one side with the same case's row on the other; the WHERE clause is the only thing making a the incumbent arm and b the variant rather than the reverse, which is why getting it backwards silently flips the sign of every number in the output. The two SUM(CASE WHEN …) expressions are counters rather than sums of scores: a row adds one to improved when the variant scored higher, one to worsened when it scored lower, and nothing to either when the two scores match. That is the discordant-pairs rule from the previous chapter, written in SQL. Grouping by slice runs both counters separately inside each named subset, and the HAVING clause drops the slices where nothing moved, since a slice with no discordant cases has nothing to contribute.

On a ticket classifier whose few-shot block had produced exactly this cancellation, the answer was not evenly spread at all:

Check the table against the totals before reading anything into it, because that check is what turns the cancellation from a surprise into an accounting identity. Sum the improved column down the five rows — thirty-three, one, nothing, five and two — and it comes to the forty-one improvements the join found. Sum the worsened column — two, nineteen, twelve, four and two — and it comes to the thirty-nine regressions. The cases column sums to the five hundred in the set, which confirms every case landed in exactly one slice. Now take each slice's net: billing gains thirty-one, out-of-window refunds lose eighteen, warranty loses twelve, tracking gains one, and the remainder is flat. Those five nets add back to the same two-case net the aggregate reported. Nothing was concealed by the sum. The sum had one slot and five facts to fit into it.

The size of the slices is what allows the cancellation to be so close, and it comes off the same column. A hundred and seventy-four billing cases out of five hundred is about a third of the set; the thirty-one refund cases and twenty-two warranty cases together are about a ninth of it. A slice three times the size needs only a third of the per-case effect to offset the other two, and the effects here are the other way round — small and broad against large and narrow — which is exactly the ratio that lands a delta near zero.

Read as a whole, the block is doing something specific and legible: it is holding the classifier to the right category on the two slices where the category boundaries are subtle, and it is distracting the classifier on the largest, simplest slice, where a plain question about an invoice gets over-thought into a neighbouring category because one of the twelve worked examples looks superficially similar. Both effects are real. They sum to nothing because billing/plain-question is 35% of the frozen set and refunds/out-of-window plus warranty/edge-case together are 11%.

That last sentence is what decides the ticket, because the set's slice mix is an artefact of how the set was built rather than a claim about production. An eval score treats every case as worth the same, which is a modelling assumption and usually a false one: a misrouted billing question costs one extra queue hop, and a mishandled out-of-window refund costs a human twenty minutes plus whatever goodwill the customer had left. If you can attach even a crude per-slice cost — minutes of agent time, escalation probability, refund exposure — the 41-versus-39 split becomes a weighted sum that reports a direction instead of a tie. Crude is fine here; the difference between "these slices cost about the same" and "these slices differ by an order of magnitude" is what changes the decision, and nobody needs three significant figures to establish that.

Cherry-picking, graded scores and genuinely inert blocks#

Reading individual cases is a licence to look, not a licence to conclude, and there is a failure mode on this side too. With 500 cases and two arms you can always find an example that supports whatever you already believed; picking three sympathetic cases out of the 39 that got worse and presenting them as the finding is cherry-picking with extra steps. What makes the classifier analysis above evidence rather than anecdote is that the grouping was decided by a pre-existing slice taxonomy, every discordant case was assigned to exactly one group, and the counts are reported in full including the slices where nothing happened.

Binary scores also make this easier to see than graded ones, because a case either flipped or it did not. Graded scores hide cancellation better and are more common in support evaluation, where a judge returns 1 to 5 on helpfulness. A one-point drop on 40 cases and a one-point rise on 40 others produces the identical mean, and the per-case diff is a distribution rather than a flip count, so "did this case change" needs a threshold.

That threshold is derived rather than chosen, and the direction of the inequality is the whole of the reasoning. If the judge already moves a typical case by some amount when nothing at all has changed, then any delta at or below that amount is fully accounted for by the judge, and counting it as movement imports the churn floor into the analysis disguised as signal. The cut therefore has to sit above the measured spread, and the value to take is the first one above that spread which the scoring scale can actually express. Set it lower and a graded run reports hundreds of movers, most of them the judge; set it much higher and genuine half-scale regressions are filed as no change. The workable convention is to treat a change smaller than the judge's own repeat-to-repeat spread as no change: measure that spread during the A/A run, and if the judge moves a typical case by up to 0.4 points between identical runs, count only deltas of 0.5 or more as movement. Everything else here then applies unchanged, with the histogram of deltas standing in for the improved and worsened counts.

Cancellation is also not automatically a reason to keep a block. Row four of the first table — 240 cases moving, evenly split — describes a system where the block is churning behaviour without a stable reason, and the right response there is to distrust both arms and look at whether the block is interacting with something else in the prompt. And row one is a genuine possibility that this chapter should not talk you out of: some blocks really are inert, the per-case diff really does come back all zeroes, and deleting them is free. Establishing that a block is inert on your set is a different measurement with its own trap, which the last chapter of this topic takes apart.

An unchanged aggregate is the beginning of a reading rather than the end of one. The sequence is fixed and short: count the discordant cases and compare them with the floor, test the split for direction, group the movers by slice, and weight the groups by what an error in each actually costs. A block that trades one slice for another survives that reading as a routing decision, which is a better outcome than either of the two things a net-zero delta invites — deleting a block that is holding the hard cases together, or keeping one that is costing you errors on the easy ones.

The whole analysis rested on two columns that were assumed rather than earned: a slice label on every row and a matching case_id across arms. Neither exists unless the harness writes them, and the same is true of the token counts that priced the trade and the retrieval provenance that would have told you whether the block was displacing a passage rather than being read.