Auditing false fails

An audit here means reading failures by hand against the criterion they were supposed to be graded on, and counting how many of them were not failures at all. The count divides into an estimate of a rate, the rate scales back across every failure the suite produced, and the product is a corrected pass rate with an interval attached. None of that is difficult. What makes it worth a chapter is that the arithmetic is easy to do badly in ways that all point the same direction, and that the corrected number is easy to publish in a form nobody can reproduce.

The setting is a text-to-SQL suite that runs over 500 items and reports 68% pass: 340 passes and 160 failures, with the failure list the only place anybody looks when the number moves. The criterion anybody actually cares about is whether the query returns the right rows, and the suite's rubric had been failing candidates that returned exactly the right rows in a different formulation. That is the defect the previous chapter diagnosed — a rubric that let the reference's particular implementation stand in for the requirement, so that a query reaching the reference's rows by another route was recorded as a failure.

Two properties of that arrangement decide the whole method, and both are easy to slide past. The first is that 340 and 160 are not two separate measurements. The suite ran 500 items and reported 68% pass, so the passes are sixty-eight hundredths of 500 and the failures are whatever is left over. The two counts are one fact written twice, which is why moving items from one column to the other later on leaves the total untouched and leaves every later division by 500 untouched with it.

The second is that a false fail can only be found among the failures. An item the suite passed is not an item the suite wrongly failed, whatever else may be wrong with it. The 160 are therefore not a convenient place to start looking; they are the entire population in which the quantity being estimated lives, and their number is known exactly rather than sampled. That is what makes the scaling step below legitimate. It is also the reason the correction only ever moves the number upward, which is a property of where the sample was drawn rather than a finding about the system.

The correction#

Fifty of the 160 failures were drawn at random and read by hand — about four minutes each, opening the candidate query, the reference, and the row sets side by side. Eleven of the 50 returned exactly the rows the reference returned. They were not failures of the system; they were failures of the rubric. Eleven out of 50 is 22%, and the useful move is to stop treating that as an anecdote about eleven queries and start treating it as an estimate of a rate that applies to all 160. The random draw is what licenses that:

false-fail rate among failures    11 / 50            = 0.220
estimated false fails, all 160    160 × 0.220        = 35.2
corrected passes                  340 + 35.2         = 375.2
corrected pass rate               375.2 / 500        = 75.0%

Each of those four lines is worth being able to defend on its own. The first divides the false fails found by the number read, which is what a rate is: eleven in fifty, a little over a fifth. The second carries that rate onto the whole failure set, and it is the only one of the four that is an inference rather than a calculation. What licenses it is that the draw was random. Every one of the 160 had the same chance of being read, so whether an item was read is independent of whether it is a false fail, and the fifty that were read stand in for the hundred and ten that were not. Had the fifty instead been the ones that looked most suspicious, or the first fifty in a list sorted by cause, the rate among them would say nothing whatever about the rest.

The third line moves items between columns rather than changing what was measured. About thirty-five of the recorded failures were not failures, so they belong with the passes, and the fourth line divides by 500 because the suite still ran 500 items. Nothing here claims that a fraction of a query exists. The figure is an expected count — the number that would come out on average if the draw were repeated many times — and it is left unrounded so that the rounding happens once, at the end, rather than twice.

The step most often got wrong is the conversion between the two rates, because both are percentages and percentages invite being added to one another. A fifth of the failures is not a fifth of the suite. Failures are 160 of 500, a bit under a third of the items, so a rate measured among failures has to be multiplied by that share before it becomes points of pass rate. A little over a fifth of a bit under a third is around seven parts in a hundred. Adding 22 to 68 instead gives a number nothing in the audit supports.

Seven points. The suite says 68%; the system is doing about 75% on the criterion that matters.

Seven points is enough to change decisions. It is the difference between a feature that misses a 70% launch bar and one that clears it, and — more often the live question — it is the difference between "the model is the problem, let's try a bigger one" and "the eval is the problem, let's fix the rubric". Teams spend weeks on the first when the answer is the second, because a pass rate arrives with an authority that a rubric defect does not.

The belief to block before going any further is that the corrected figure is now the true one. It is not a measurement of the system. It is a measurement of the suite, plus a model of one defect in the suite, and its accuracy is bounded above by the accuracy of the audit that produced the model. The audit is people reading queries against a criterion, which is the same species of instrument the suite was just faulted for being, operating one level up. What the correction buys is not a trustworthy number but a defensible statement about the size of a known defect, which is a smaller claim and a far more useful one.

What 50 audited items actually buys#

The point estimate of 0.220 is not the rate. It is one draw from a sampling distribution, and the interval around it is wide enough to matter. A Wilson score interval, which is a confidence interval for a proportion and is better behaved than the textbook normal approximation at proportions this far from 0.5 with n this small, gives:

from statistics import NormalDist

def wilson(successes: int, n: int, conf: float = 0.95) -> tuple[float, float]:
    z = NormalDist().inv_cdf(1 - (1 - conf) / 2)
    p = successes / n
    d = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / d
    half = (z / d) * ((p * (1 - p) / n + z**2 / (4 * n**2)) ** 0.5)
    return centre - half, centre + half


lo, hi = wilson(11, 50)                      # (0.128, 0.352)
n_fail, n_total, n_pass = 160, 500, 340
print((n_pass + n_fail * lo) / n_total,      # 0.721
      (n_pass + n_fail * hi) / n_total)      # 0.793

Two properties of that function can be read off the code rather than taken on trust. The centre line does not return the observed proportion: it adds a term to the numerator and divides by a quantity slightly larger than one, which pulls the centre toward one-half by an amount that shrinks as the sample grows. That shift is exactly what the textbook interval leaves out. The normal approximation places its interval symmetrically around the observed proportion, and at a proportion this far from one-half with a sample this small that interval comes out both too narrow and capable of running off the end of the scale, since nothing in it stops a lower bound landing below zero. Wilson can do neither, which is why it is the one to reach for at these sizes rather than a matter of preference.

The two print lines then push each end of the rate interval through the same arithmetic the point estimate went through — multiply by 160, add to 340, divide by 500 — because that is the only way the three numbers stay comparable with one another. It also accounts for the interval on the pass rate being so much narrower than the interval on the rate. The rate interval spans a little over two tenths; multiplying by the share of the suite that failures make up shrinks it by that same factor, and a bit under a third of a little over two tenths is the seven points the pass-rate interval covers.

So the corrected pass rate is about 75%, with a 95% interval running from roughly 72% to 79%. The interval is seven points wide — the same size as the correction itself. That is the honest summary: the audit establishes that the suite understates the pass rate by something in the neighbourhood of four to eleven points, and pins it no more tightly than that.

Narrowing it costs linearly in the interval and quadratically in effort, which is the usual and unwelcome arithmetic:

The general law is visible in that third column. Doubling the audit from 25 items to 50 narrows the interval on the false-fail rate by not quite a third. Quadrupling it to 100 narrows it by about a half. Halving an interval costs four times the sample, and because the hours column is linear in items at four minutes apiece, it costs four times the hours as well. That is why an audit that came out too loose is almost never rescued by reading a few more items, and why the argument at this point is between a sample and a census rather than between a sample and a slightly bigger one.

Auditing all 160 removes the sampling uncertainty entirely for eleven hours of work, which on a decision worth a launch is cheap. The reason to sample at all is that this is not a one-off: the audit has to be repeated whenever the rubric or the model changes, and a ten-hour recurring cost gets skipped in a way that a three-hour one does not.

Why it has to be repeated, rather than measured once and carried forward, is that a false-fail rate is not a property of the judge on its own. It is a property of one rubric, applied by one judge, to the particular distribution of candidate answers that one generator produces. Change the rubric and the mapping from candidate to verdict moves. Change the generator and the candidates move: a model that stops writing correlated subqueries removes most of the population the rate was measured on, and the same rubric defect then costs a quite different number of points. Quoting one release's seven points against the next release's failure list is the commonest way a correct measurement becomes a wrong one, and it leaves no trace, because the arithmetic is still valid arithmetic performed on numbers that no longer describe the same thing.

Stratify, because failures are not interchangeable#

A judge constrained to name a cause for each failure makes the sample allocable, and the false-fail rate turns out to be wildly different across those causes. Breaking the 160 down and sampling within each stratum — each group of failures that shares a cause — rather than uniformly, buys a tighter estimate for the same fifty items:

Proportional allocation is what makes the two totals agree, and it is worth checking rather than believing. The approach-mismatch stratum holds 48 of the 160 failures, which is three-tenths of them, and 15 of the 50 audited items went there — three-tenths again. The other two rows match their own shares as closely as whole items allow. When that holds, the estimate built stratum by stratum and the estimate built from the pooled rate are the same arithmetic performed in a different order, because the pooled rate is already the average of the three stratum rates weighted by exactly those shares. The three stratum estimates sum to 35.2, and so does 160 times eleven over fifty.

The point estimate is unchanged at 35.2, because the sample was allocated proportionally. What changes is the variance: the estimate is now a sum of three stratum estimates, two of which are close to certain — a query that does not execute is not a false fail under any reading, and the twelve audited confirmed it — while all the uncertainty lives in the 48-item approach-mismatch stratum. Concentrating the next twenty audit items there, rather than spreading them, shrinks the interval faster than doubling the sample uniformly would.

That last claim follows from two properties of a stratum, and neither of them needs a formula. A stratum contributes uncertainty in proportion to its size, because whatever rate is measured inside it gets multiplied by that size on the way into the total. And a rate is hardest to pin down when it sits near one-half, because that is where the items within a stratum disagree with one another most; a stratum whose audited items all came back the same way has very little left to be uncertain about. The non-executing stratum fails both tests for mattering. It is the smallest of the three, and its rate is not merely measured at zero but held there by the criterion, since a query that does not run returned no rows at all and so cannot have returned the reference's rows. The twelve read there confirm a definition rather than estimate a proportion. The approach-mismatch stratum is the reverse on both counts: it accounts for most of the estimated false fails, and its rate sits at six-tenths, about as close to the worst case for a proportion as this table gets. Items read there move the total, and items read in the other two mostly reconfirm what is already known.

The table also localises the bug. Sixty percent of "approach differs" failures are false, which is a precise statement about which rubric clause is broken and roughly how badly. That is a more actionable output than the corrected pass rate, and it is what actually gets fixed.

Who audits, and under what instructions#

An audit is itself a measurement, made by people, against a criterion — which is the situation a rubric disagreement session describes, and it has the same failure mode.

The instruction matters most. The auditor's question is narrow: given this candidate query, this reference, and both row sets on every fixture, do they return the same rows? That is not the same question as "should this have passed the rubric", and handing the auditor the rubric invites them to re-derive the judge's reasoning rather than check the fact. Fifty row-set comparisons is a mechanical task that runs at four minutes an item. Fifty rubric adjudications is a judgment task that runs at fifteen and produces a rate that depends on who was doing it.

The auditor should not see the judge's stated reason. Reading "fails because the approach differs from the reference" before looking at the query anchors the auditor toward agreeing, and the effect is large enough to matter on a rate this small — eleven items out of fifty is the difference between a seven-point correction and a four-point one.

Two auditors overlapping on part of the sample is what makes the number defensible. Fifteen of the fifty went to both, and they agreed on fourteen; the one disagreement was a query returning the reference's rows in a different column order, where one auditor treated column order as part of the row set and the other did not. That is a specification gap in the audit instruction, fixed by a sentence, and finding it is the point of the overlap. Had they disagreed on four of fifteen, the audit would have been as unreliable as the judge it was measuring, and the corrected pass rate would not be reportable at all. For the same reason the rubric's author should not be the sole auditor: they know what the criteria were meant to mean, which is exactly the knowledge that makes them lenient toward the judge's readings of it.

Reporting the corrected figure without laundering it#

The wrong move at this point is to replace 68% with 75% on the dashboard. The 75% is an estimate produced by a manual audit with a stated method and a seven-point interval; the 68% is a direct measurement of a defective instrument. Swapping one for the other buries both facts and leaves a number nobody can reproduce. What goes in the report is the whole chain:

Suite reports 68.0% pass (340/500). A random sample of 50 of the 160 failures found 11 (22%) that return exactly the reference's row set, using a different formulation — a rubric defect, not a system failure. Correcting for it gives an estimated 75.0% pass rate, 95% CI 72.1%–79.3% (Wilson, on the audit sample). The defect is concentrated in the "approach differs from reference" stratum (60% false-fail rate, 48 items). The rubric clause is being fixed; the suite will be re-run and this estimate retired.

Three properties make that reportable rather than persuasive. The corrected figure carries its interval, so nobody quotes it as 75% flat. It carries its method, so a sceptic can re-draw the sample. And it announces its own expiry: the estimate exists to bridge the gap until the rubric fix lands, after which the re-run produces a directly measured number and the estimate has no further use.

When to skip the arithmetic and just re-run#

All of the above is worth doing when the estimate is the deliverable. Often it is not. Re-running this suite costs $6 and twenty minutes of wall clock, while auditing fifty failures costs 3.3 analyst-hours. Once the first three hand-read failures established that the rubric clause about alternative implementations was missing, the cheapest path to a correct number was to write the clause, re-run, and read the new failure list — not to estimate what the number would have been.

The estimate earns its keep in three situations. When the re-run is genuinely expensive: a human-graded eval, or one that executes against production data under a change window. When the decision is due before the fix can land, and somebody has to know whether the gap between a suite's number and a launch bar is a rubric artefact or a real shortfall. And when the fix is contested — an estimate that the defect is worth seven points is what gets rubric work prioritised against feature work, and "we think the eval is a bit pessimistic" is not. Outside those, the audit's product is not the corrected number but the stratum table, and that comes from reading fifteen items rather than fifty.

The deeper limitation is that the correction is one-sided by default. Everything above audits failures, and the same logic applies in the other direction: some of the 340 passes are queries the judge waved through that do not return the right rows. Auditing only the failures and publishing the resulting seven-point improvement is the most common way this arithmetic gets misused, and it is not usually deliberate. Failures are where people look, the correction they produce always moves in the flattering direction, and a team that stops there has performed a real measurement and reported half of it.

Both halves also share a limitation worth stating plainly: the auditors were the people who wrote the rubric. They agreed with themselves about what "returns the right rows" means far more readily than two independent graders would have, which is a rubric reliability problem reappearing one level up. The audit is a measurement made with an instrument — human judgment applied to a criterion — that has not itself been calibrated.

A corrected pass rate, then, is an estimate with a method, an interval and an expiry date, and it is worth computing precisely when the thing it changes is a decision rather than a dashboard. Eleven false fails in fifty moved 68% to about 75%, three false passes in fifty moved it back toward 71%, and the number that survives both is still less useful than the finding underneath them — that one rubric clause was generating 60% of one failure category.

Which is the question the whole topic has been circling. The judge has been probed, its protocol settled, its references corrected and its failures audited, and every one of those operations compared the judge to some other artefact rather than to the thing it is supposed to approximate. The remaining question is how well the judge agrees with people, and the first number anyone computes for that turns out to be nearly uninformative.