Rare defects and sampled aggregates

Once an eval runs the whole path end to end, a reasonable-sounding proposal follows: delete the deterministic tests. They cost maintenance every time a prompt template changes, they exercise pieces of a system the eval exercises together, and if the answers coming out are good then the plumbing producing them must be fine.

Part of that is correct and worth conceding before arguing with the rest. An end-to-end check does subsume unit coverage of the happy path, so a test asserting that an assembler produces a well-formed prompt for the ordinary case is genuinely redundant once a live run produces good answers on the ordinary case. Some tests are worse than redundant: a test asserting the exact wording of a system-prompt sentence fails on every copy edit and never catches anything, because it mirrors the implementation rather than constraining a behaviour, and that is an argument for deleting it which has nothing to do with evals.

The rest of the proposal fails on what an aggregate over a sample can resolve, and that limit is arithmetic rather than a matter of opinion. It also has to be answered on its own terms, because underneath the proposal is a real complaint: a deterministic suite that nobody is allowed to delete from does grow forever, and a team drowning in brittle tests is not wrong to want something done. An answer that only defends the tests concedes that point and loses the argument in six months.

A defect at 1 in 200 is 0.5 points on a 400-case score#

Take a parser bug of a shape that actually ships: when a completion contains a trailing prose paragraph after the closing brace and the answer references two orders, the repair path drops carrier_ref, so the customer gets a status update with no tracking link. The combination occurs on roughly 1 request in 200.

Over a frozen 400-case set, the expected number of affected cases is 2, and that figure comes from adding one probability per case. Each case is one request, and each request trips the combination with probability one in two hundred, so the expected count is 400 multiplied by one two-hundredth, which is 400 divided by 200. Adding per-case probabilities up like that is legitimate whatever the cases do to one another, because expectations add whether or not the events are independent, so the 2 survives even in a set whose cases resemble each other. What it does not carry is the reading most people give it. Two is an average over many hypothetical 400-case sets, not a statement about the one set on disk.

The question a gate turns on is not the average but whether the set on disk contains an instance at all, and that is a separate calculation. A single case fails to contain the combination with probability 199 in 200, which is one minus the one-in-two-hundred rate. For the whole set to contain none, all 400 cases have to miss. Multiplying those per-case miss probabilities together is the step that needs justifying, and what justifies it is independence: whether one case happens to reference two orders and end in a trailing prose paragraph tells you nothing about whether the next one does, because each case was drawn separately. Under that condition the probability of a run of events is the product of their individual probabilities, so the probability that all 400 miss is 199/200 multiplied by itself 400 times.

The probability of at least one appearing is 1 - (199/200)^400, about 86%, so the eval does usually contain an instance. Read the other way round, the same arithmetic says roughly one 400-case draw in seven contains no instance at all. How big that miss probability is depends almost entirely on the expected count rather than on the rate and the case count separately, which is why a set ten times as large, run against a defect a tenth as common, misses just as often.

Two conditions sit underneath that 86%, and both are worth establishing before anybody quotes it in a design review. The first is that the cases are drawn from the traffic the defect occurs in. A set assembled from hand-written examples and individual customer complaints is not a sample of production, and a rate measured on production says nothing about what such a set contains. The second is that the set is frozen, which the arithmetic quietly cuts against: the 86% is the chance that a fresh draw of 400 cases contains an instance, and a frozen set is drawn once. If that single draw came up empty, it is empty every week for as long as the set stays frozen, and re-running it weekly does not convert a one-off miss into a second chance.

Containing an instance and detecting it are different things, and it is the second of those that the proposal founders on rather than the sampling. Two failing cases out of 400 moves the pass rate by 2 divided by 400, which is 0.005, or 0.5 points on a score reported out of a hundred. Set that against a paired detectable difference of 4.4 points for a suite of that size at a measured 9.8% flip rate between runs. That flip rate, or discordance, is the share of cases whose pass-or-fail verdict changes between two runs of the same unchanged commit, so it is this instrument's own noise floor rather than a textbook assumption. Dividing 4.4 by 0.5 puts the defect's entire footprint at a ninth of the smallest move the instrument can resolve. The eval registers 84.2 one week and 83.7 the next, and no procedure, statistical or otherwise, distinguishes that from the judge sampling differently.

A green run is therefore not evidence that the defect is absent, and this is the step worth slowing down on, because a reader who has followed the arithmetic can still walk away believing the eval cleared the build. A null reading counts as evidence of absence only to the extent that the instrument would have reported presence, and this instrument would not have. The four scores in the scenario below were taken over nine days during which the defect was live on every one of them, and the highest and the lowest of the four differ by more than the defect contributed to any of them. A flat series under those conditions is what this defect looks like, and it is also what no defect at all looks like, which is the whole of the difficulty.

Making the sample larger does not rescue the measurement. Detecting a 0.5-point difference at the same power needs roughly 7.84 × 0.098 / 0.005² cases, which is about 30,700 — a $370 run taking 32 hours, on every merge.

That expression is the paired sizing formula from chapter 5, on sizing an eval set, with this defect's figures substituted into it, and its three parts are worth naming individually. The 7.84 is the squared significance-and-power multiplier chapter 5 assembles, and it is a constant that does not move with the suite. The 0.098 is the measured discordance, the same flip rate as above written as a proportion instead of a percentage. The 0.005 is the difference to be detected, half a point written the same way. Multiplying the top gives a little over three quarters; squaring the bottom gives twenty-five parts in a million; dividing the first by the second is where 30,700 comes from.

The same formula run backwards produces the 4.4 points quoted earlier, and doing that once is what shows the two figures to be one figure seen from two sides. Fix the case count at 400 and solve for the difference instead: 7.84 times 0.098 divided by 400, then square-rooted, gives the difference as a proportion, and read on the hundred-point scale the eval reports on, that proportion is the 4.4 points. Resolution and case count are two ends of the same expression, and reading it in either direction hands you the other.

The shape of that expression carries more than either figure, because the difference sits in the denominator and is squared. That is what turns a modest gap in resolution into an enormous gap in cases. Divide 4.4 by 0.5 and the gap in resolution is a little under nine; square that, because the difference enters squared, and the gap in cases is a factor of roughly seventy-seven. Applied to the frozen 400 it gives about 30,700, which is where the case count comes from and why it is not the ninefold increase a first reading suggests. The wall clock follows the same multiplier, since the cases run at the same rate either way: seventy-seven runs of twenty-five minutes each is the 32 hours.

The deterministic test that catches that same parser bug is a fixture file and four lines of assertion, runs in eleven milliseconds, and names the defect in its failure message.

def test_carrier_ref_survives_trailing_prose_on_multi_order_answers():
    raw = (FIXTURES / "two_orders_trailing_prose.txt").read_text()

    parsed = parse_order_answer(raw)

    assert len(parsed.orders) == 2
    assert [o.carrier_ref for o in parsed.orders] == [
        "DHL-3391847722",
        "DHL-3391847905",
    ]

What lets eleven milliseconds do what thirty thousand cases could not is that the test is not sampling anything. The fixture is one recorded response held still on disk, the parser is the code under test, and the assertion is a function of those two and of nothing else, so the verdict is the same on every run and one instance of the defect is enough to produce it. A sampled aggregate needs enough instances to lift a mean clear of its own noise, and how many that takes is the arithmetic above. A fixture needs one, because it is not measuring a rate at all — it is walking a path and checking what came back.

The asymmetry runs both ways, which is what makes this a division of labour rather than a ranking. Rare deterministic defects are cheap for a test and structurally invisible to a sampled aggregate. Broad distributional degradations — answers getting vaguer, citations getting thinner, a tone drift across every response — are invisible to fixtures and are exactly what an eval is for, because no fixture set contains the thing that changed and the change shows up only as a level.

Three regressions and what each instrument says#

The parser bug is one of a family. Two more from the same feature make the shape clearer:

The second row is the one people find most surprising, and it generalises: an eval measures the system under eval conditions, which are not production conditions. Concurrency, rate limits, cold caches, retry storms, timeouts under load, and every failure that depends on traffic shape rather than on content sit outside what a 400-case sequential run can express. An eval set has no burst behaviour because it was never asked to have any.

The third row is where the eval works and still leaves a diagnosis problem, and the cost is worth being precise about. A drop from 80.6 to 68.4 is a real signal, and what arrives in hand is a scalar.

Six candidate commits walked in the worst order is 2.5 hours and $36, and that is the well-behaved case, where exactly one commit is responsible and the effect is large enough to reproduce on the first re-run. When the effect is 5 points rather than 12, each hypothesis needs a confirmation run to separate it from noise, which doubles both numbers. A failing assertion arrives with a function name, a line number and the input that broke it.

The distinction is the one between a thermometer and a continuity tester. A thermometer says the room is four degrees colder than yesterday, which is real information about a real property, and no reading of it says which window is open. A continuity tester says which wire is broken and says nothing whatsoever about whether the room is comfortable. Wiring a house with only one of them is a choice nobody makes deliberately.

That comparison is a picture rather than a claim about laboratory instruments, and the place it breaks is the place this chapter has been working in. A thermometer's error is tiny compared with the changes anybody points one at, which is why a reading that has not moved is read as a room that has not changed. The eval's noise floor is several times the size of the defect in question, so a reading that has not moved licenses no such inference. Everything in the comparison about which question each instrument answers holds; nothing in it about how much a null reading is worth does.

Keeping the deterministic suite from growing without bound#

The maintenance objection survives that ratchet and deserves an answer, because a suite that only grows eventually costs more than it returns. The rule that works is to attach every deterministic test to a fault class rather than to a feature, and to review it when the code it constrains is deleted rather than when the prompt is edited.

The eleven wording tests failed that rule because their fault class was "somebody changed a sentence", which is not a fault. The parser tests pass it because their fault class is "the repair path mishandles a response shape the model actually produces", which stays real for as long as the parser exists. Applied over a quarter, that rule took one team's suite from 74 tests to 61 while adding nine new ones from eval diagnoses, and the wall clock went from 90 seconds to 71.

The ratchet has a direction worth noticing. It converts a fault class from the expensive instrument to the cheap one exactly once, at the moment it is diagnosed, and it never converts anything back. A team running it for a year ends up with an eval that catches progressively fewer things. The cause is migration rather than improving quality: each fault class the eval catches moves into the fast suite after its first occurrence, which is the intended outcome and reads as a declining return on the eval if nobody says so out loud.

What neither instrument covers#

Being honest about the remaining gap matters more than the tidiness of the split. Between them, a fast suite and a sampled eval leave two things uncovered.

Failures that depend on production traffic shape — the retry row above, and everything downstream of it — belong to load testing and production monitoring, and no amount of eval budget substitutes for a p99 latency alert and a 429 rate dashboard. That is the same boundary the classification exercise drew at the provider's edge: what happens inside the provider is measured through your prompt, and what happens to your service under load is measured in production.

Failures on inputs nobody has thought of belong to neither, because a fixture set contains what somebody imagined and a frozen eval set contains what somebody sampled, and the questions customers will ask next month are in neither. That gap closes only by feeding production traffic back into both sets on a schedule, which is why a weekly run is worth pointing at unlabelled traffic cases and why a frozen set carries a version number that is expected to increment.

The proposal that opens this chapter is therefore right in its premise and wrong in its conclusion. An eval does exercise the whole path, and a deterministic suite does contain tests worth deleting. What it cannot do is see a defect whose rate is small enough to hide inside its own noise, or name a location when it does see something, and both of those are the ordinary shape of a bug in code you wrote. Keeping both instruments is not redundancy; it is the recognition that one reports a level and the other reports a location, and that a team with only the first spends afternoons bisecting while a team with only the second ships answers nobody is measuring.

What this topic has built, taken together, is a sorting criterion, a classification of the checks around one feature, a set of triggers with an owner for each path, an audit for a gate that may have quietly stopped protecting anything, a case count derived from the smallest regression worth catching, and a spend plan whose cuts are stated rather than absorbed. Applying it starts with the checks already sitting in your repository: take each one, ask whether its verdict repeats on the same commit within minutes, write the answer and the reason beside it, and count how many are currently running in the wrong place.