Run-to-run noise in a pass rate

The output of a property-checked suite is a fraction: the share of cases that satisfied every requirement on that run. Two such fractions, one per prompt version, look like a comparison, and the sentence that writes itself is that the new prompt cost the suite three points. Everything upstream of that sentence can be done correctly — cases drawn from real traffic, checks that assert properties rather than strings, a sample in the hundreds rather than a hand-typed dozen — and the sentence can still be premature.

What makes it premature is that a single run's rate is one draw of a quantity that moves on its own. How far it moves is calculable — the standard error of a rate measured over n cases sets the smallest gap two single runs can resolve, and any release threshold set below that gap is a mechanism for blocking releases at random. But the formula is the weaker of the two available instruments, because it assumes each case is an independent draw and your case set, drawn from real traffic with its duplicates and its families of near-identical requests, is not. The stronger instrument is to repeat both arms over the same cases and read the spread off the runs, which costs an afternoon of compute and assumes nothing.

How far a rate wanders when nothing changed#

Take 200 cases, each passing independently with some true probability p, and count the passes. That is a binomial variable — a count of independent yes-or-no outcomes that each carry the same probability — and the standard error of the resulting proportion, meaning the typical distance between the rate a run reports and the true rate underneath it, is √(p(1−p)/n). At a pass rate near 0.85 with 200 cases:

√(0.85 × 0.15 / 200) = √0.0006375 = 0.0252

Two and a half points. That is one standard error, so runs land within ±2.5 points of the truth about two thirds of the time and within ±5 points about 95% of the time. A single 200-case run reporting 0.82 is consistent with a true rate anywhere from roughly 0.77 to 0.87.

Three things happen inside that square root and each is doing a separate job, so none of them should be taken on trust. The p(1−p) on top is how much a single case bounces around. A case that passes with probability p contributes a one or a zero, and the spread of that single outcome is largest when p is a half — the case is a coin flip and genuinely unpredictable — and shrinks to nothing as p approaches either end, since a case that always passes never surprises anybody. Multiplying p by 1−p is the quantity with exactly that shape: it peaks in the middle and vanishes at both ends. At the pass rate this suite sits near, eighty-five hundredths times fifteen hundredths is a bit under thirteen hundredths, which is already well down from its maximum. Cases you nearly always pass are cheap to measure.

Dividing by n is the averaging step, and it is where the intuition usually goes wrong. The rate a run reports is the average of n case outcomes, and averaging independent things cancels some of their movement: when one case happens to fail, another happening to pass pulls the average back. That cancellation improves in proportion to n, so the bounce in the average is the bounce in one case divided by n. Note carefully that it is the bounce and not the standard error that divides by n — the two differ by the square root, which is the third step.

The square root at the end puts the answer back into the units of a pass rate. p(1−p)/n is measured in squared rate, because it came from squared distances, and a spread in squared rate cannot be compared against a threshold expressed in points. Taking the root converts it. The consequence is the one people get wrong: because the root is taken after the division, quadrupling the number of cases halves the standard error rather than quartering it. The table below is that fact laid out. Going from 50 cases to 200 is four times the work and takes the standard error from 5.0 points to 2.5 — a halving, not a quartering — and every further halving costs another factor of four.

A comparison is worse than a single measurement, because both figures carry that error. The standard error of the difference between two independent runs is √(SE₁² + SE₂²), which for two runs of the same size is SE × √2 ≈ 3.6 points. Ninety-five percent of the time, two runs measuring genuinely identical systems land within 1.96 × 3.6 ≈ 7 points of each other.

Two details in those lines decide whether the rest of the chapter is believable, and both are short. The first is why the two errors combine by squaring, adding and rooting rather than simply adding. Squared standard errors add when the two quantities are independent, and these are: separate runs, separate draws, nothing shared. Adding the standard errors directly would describe a world where the two runs always err in opposite directions, which would be the worst case rather than the typical one. Squaring first credits the fact that some of the time both runs land high together, or both low, and the difference between them is then small. With SE₁ and SE₂ the same size, the sum of their squares is twice one square, and the root of twice a square is the square root of two times that quantity — which is where the roughly 3.6 comes from: two and a half points multiplied by a little over one and four tenths.

The second is 1.96, which is the number of standard errors you have to go out in each direction to enclose 95% of a normal distribution. It is a property of that distribution rather than of this suite, and the reason a normal distribution applies to a count of passes over 200 cases is that a sum of many small independent contributions takes that shape whatever the individual contributions look like. So multiplying it by the 3.6 gives the width inside which the gap between two runs of an unchanged system falls nineteen times in twenty: the seven points that the rest of this chapter compares every proposed threshold against.

Laying that out against sample size shows where the resolution actually lives:

This is the arithmetic a two-point release threshold on 200 cases has to survive, and it does not. The threshold sits at roughly a quarter of the gap that a single 200-case run can resolve, which means a gate written that way fires on noise most times it fires.

That claim has a number attached, and it is the number that persuades people. Against a system that genuinely did not change, the difference between two runs has a standard error of 3.6 points and a mean of zero. A drop of two points or more is 2 / 3.6 = 0.56 standard errors below zero, and the normal tail beyond 0.56 standard errors is about 0.29. A gate blocking on a two-point drop, run against an unchanged system, therefore blocks roughly 29 releases in 100. Nearly a third of the time it stops a release for a change that did nothing, and a team learns within a month that the correct response to the gate firing is to run it again.

Three assumptions went into that, and each is worth seeing, because the same shape of argument prices any gate anybody proposes. The mean is zero because the system did not change: both runs are measuring one true rate, so the differences they report scatter around no difference at all. The division by 3.6 converts a distance in points into a distance in standard errors, which is the step that makes it comparable to anything — two points means nothing until you know how noisy the measurement is, and two points against a noise of 3.6 points is a bit over half of one typical wobble. And the tail is read on one side only, because the gate blocks on drops and waves rises through. A run landing three points above the last one is exactly as much noise and stops nothing, so only half the distribution is being counted.

Read the last step in reverse and the weakness of the threshold becomes something you can feel rather than compute. Half a standard error is not far into a normal distribution at all; most of the probability sits within one standard error of the centre. A boundary drawn barely past halfway therefore leaves a large slice of the distribution beyond it, and twenty-nine in a hundred is that slice.

That failure mode is more corrosive than a gate which is too loose. A gate nobody trusts is a gate that gets overridden, and once overriding is routine the one time it fires for a real six-point regression looks exactly like the twenty-nine times it did not.

The intuition that defends the two-point threshold is usually a counting argument: 200 cases means each case is worth half a point, so a two-point move is four cases changing, and four cases changing is obviously something real. The premise is right and the conclusion does not follow. Four cases flipping is exactly what a stochastic generator does between two runs of the identical prompt — five draws over one support ticket produced four distinct outputs in the first chapter of this topic, and any case sitting near a check's boundary will land on either side of it depending on the draw. Counting cases tells you the granularity of the measurement. It says nothing about its noise floor, and the noise floor here is several times coarser than the grain.

The drill#

Before reporting a regression, measure the noise instead of reasoning about it. Run both prompts over the same 200 cases several times each, and look at the two sets of numbers:

import statistics

def repeat(prompt: str, cases: list[Case], runs: int = 5) -> list[float]:
    return [pass_rate(evaluate(prompt, cases)) for _ in range(runs)]

cases = load_cases("cases/*.json")          # the same 200 for both arms
a = repeat("prompts/v6.txt", cases)
b = repeat("prompts/v7.txt", cases)

for name, xs in (("v6", a), ("v7", b)):
    print(f"{name}  {[f'{x:.3f}' for x in xs]}  "
          f"mean={statistics.mean(xs):.3f}  sd={statistics.stdev(xs):.4f}")

v6  ['0.820', '0.835', '0.815', '0.845', '0.825']  mean=0.828  sd=0.0120
v7  ['0.790', '0.810', '0.800', '0.785', '0.805']  mean=0.798  sd=0.0104

Two things fall out of that output, and they point in opposite directions.

The first is that v6 alone spans 0.815 to 0.845 across five runs of identical code against identical cases — three points of movement with no change of any kind. Any single pair of nights could have posted 0.845 against 0.810 and made a drop look like three and a half points, or 0.815 against 0.810 and made it look like nothing happened.

The second is that the two sets of five do not overlap. Every v6 run is at least 0.815; every v7 run is at most 0.810. The difference in means is 3.0 points, and the standard error of that difference — √(sd_a²/5 + sd_b²/5) — is about 0.7 points. Four standard errors. On these 200 cases, v7 really is worse than v6, and the drill that was going to demote the finding to noise confirmed it with room to spare.

That expression is the same machinery as before with measured numbers in place of assumed ones. Each arm's five runs have a spread you can read off directly — sd, printed by the script — and the quantity being compared is not one run but the mean of five, which bounces less than any single run does for the averaging reason already given: dividing the squared spread by five is the same division by n that appeared in the binomial formula, applied to repeats instead of cases. The two arms' contributions are then combined by adding squares and taking a root, because the arms are independent of each other. What comes out is 0.7 points against a gap of 3.0, which is where the four comes from — the observed difference is four times the size of the noise in the observed difference, and no formula about binomial variance was used anywhere in that sentence.

That is the outcome to expect more often than people assume. Repeating the runs does not exist to talk anyone out of regressions; it exists to say which side of the noise floor a result is on, and it will sometimes say the effect is larger and cleaner than one run suggested. The 3.6-point figure from the table is what one run of each arm buys. Five runs of each shrinks the standard error of each arm's mean by √5, and repeats are cheap in a way that new cases are not.

Repeats do not shrink case-sampling noise#

Two different uncertainties are in play, and the drill only shrinks one of them.

Generator noise is the run-to-run variation the repeats measure — the same prompt over the same cases giving 0.815 one night and 0.845 the next, because each case is a fresh draw. Repeats shrink it, at a cost of exactly one more evaluation run per repeat.

Case-sampling noise is different. Those 200 cases are themselves a sample from the traffic you care about, and a different 200 would give a different answer. Running the same 200 cases fifty times drives the generator noise toward zero and does nothing whatsoever to this one. The √(p(1−p)/n) figure from the table is the right order of magnitude for it, which means "v7 is three points worse" is well established on these cases, while "v7 is three points worse on traffic" carries an uncertainty of a couple of points no matter how many repeats are run. Both statements are useful; they are not the same statement, and a release note should say which one it is making.

There is one more caveat on the binomial figure, and it cuts in the direction of the drill's result. √(p(1−p)/n) assumes 200 independent cases sharing a single pass probability, and neither half of that holds. Real cases have wildly different difficulty — many pass essentially every time, a few fail essentially every time, and only a handful sit near the boundary where the draw decides. Those boundary cases are the only ones that move between runs, so the observed run-to-run spread on a fixed case set is normally narrower than the binomial formula suggests, which is why v6's five runs spanned three points rather than the ten the formula would allow. Failures also cluster: if six cases in the set are long-context conversations and a prompt change hurts long contexts, those six move together rather than independently, and a formula assuming independence understates how far the total can swing.

The practical reading is that √(p(1−p)/n) is a useful ceiling for how much a single run can move and a poor estimate of how much it actually will — which is reason enough to stop deriving the noise and start measuring it, since the repeats give the real spread for your cases, your prompt and your checks with no assumptions in it at all.

When the drill says the effect is inside the noise#

The v6-versus-v7 comparison came out clean, and plenty do not. Five repeats of each will often produce two clouds that overlap heavily — say v6 at 0.820, 0.835, 0.815, 0.845, 0.825 against v7 at 0.810, 0.840, 0.820, 0.830, 0.815. The difference in means is under a point and the standard error of that difference is comparable, so the experiment has resolved nothing.

The conclusion to draw is narrow, and it is easy to overstate in either direction. The result does not say the two prompts are equivalent; it says this experiment could not tell them apart at this resolution. A one-point regression could be sitting there undetected, and if a point matters for the feature — a percentage point on a support assistant answering 40,000 tickets a month is 400 wrong answers — then the honest response is to build an experiment that can see one point, not to declare the change safe. Nor does the result license shipping on the point estimate, the single measured number with no spread attached to it: "v7 came out a fraction ahead, so it's an improvement" reads a difference smaller than the noise as if it were a finding.

Three levers change the resolution and they cost different things. More repeats shrink generator noise and cost one evaluation run each, which is the cheapest option and the one with a floor — past ten or so repeats the remaining uncertainty is nearly all case-sampling, and more repeats stop helping. More cases shrink both sources and cost the work of sampling and validating new inputs, which is days rather than minutes. Better checks are the underrated third option: much of the run-to-run movement comes from a handful of boundary cases where a check's verdict flips on wording, and a check written to be less phrasing-sensitive removes that movement at its source rather than averaging over it. Choosing between them starts from what size of change would alter a decision. If nothing below three points would change what the team ships, five repeats over 200 cases is already adequate. If a single point matters, the current setup was never going to be enough, and no amount of re-reading the numbers will fix it.

The discipline this chapter asks for is small and it changes what a suite is for. A rate from one run is a draw with a spread around it; the spread on 200 cases near 0.85 is two and a half points for one arm and three and a half for a comparison; and the cheapest way to know your own spread is to run an unchanged prompt several times and write the range down. With that band in hand, a three-point drop is either four standard errors of real regression or a night at the edge of the usual range, and the difference is visible in the same glance that raises the question.

None of which yet says what a release gate should assert. Five repeats of two prompts is 2,000 model calls, and a release job cannot run that on every push without somebody asking what it costs. Comparing two arms over the same pinned cases is also throwing information away — the two sets of five were treated as independent samples when in fact each case was scored under both prompts, and pairing them up gives a sharper answer for the same money. And a real gate has to survive the ways a pass rate moves that have nothing to do with either noise or quality: cases added or relabelled between releases, harness errors scored as failures, a model alias that resolves somewhere new. The next chapter turns the measurement into a decision.