Placing an eval gate threshold

The noise floor of an eval suite is how much the score moves when nothing else does. Take a pipeline with its dataset digest pinned, its model version pinned and its grader committed, run it three times against the same unchanged commit on main, and the three numbers are not the same number. One team doing this on a Tuesday morning got 0.87, 0.83 and 0.86 against a gate set at 0.85. Nothing between those runs differed: same SHA, same 120 cases, same claude-sonnet-4-5-20250929, temperature zero on both the system under test and the judge.

One of those three runs would have blocked a merge, and since the commit had not changed, whatever that run was objecting to was not a regression. A developer who hit it would have re-run and got a green tick, and the check would have been called flaky and forgotten. The more useful reading is that the gate is sitting inside the spread of its own measurement, which is a property you can measure in an afternoon and which decides every threshold question this chapter is about.

The re-run is where the first wrong belief forms. It is tempting to read the green tick as a correction: the first run got something wrong, the second got it right, and the true state of the commit is whatever the passing run reported. Nothing in the setup supports that reading. The commit, the case file and the model version are identical across the two runs, so the second run holds no information about the change that the first one lacked. The two runs are two draws from one distribution, and a habit of re-running until the check goes green reports the best of several draws rather than the score.

Four sources of movement at temperature zero#

The obvious first question is where the movement comes from when everything is pinned, and there are four sources, only one of which most teams think about.

Provider inference is not bitwise reproducible even at temperature zero. Batching changes which requests are grouped together, floating-point reduction order changes with batch composition, and on mixture-of-experts models — ones that send each token through a small subset of specialist sub-networks chosen at inference time — the routing decision can flip on a near-tie. The effect on any individual token is tiny and the effect on a long generation is not, because one different token early can send the rest of the response down a different path.

The judge amplifies whatever the system under test did. An LLM grader asked for a pass/fail verdict on a borderline answer is itself sampling, and borderline answers are exactly the ones a well-constructed eval set is full of, since cases everybody passes carry no information. In practice the aggregate movement is concentrated: on that 120-case suite, 7 cases accounted for all of the variation between the three runs, and the other 113 returned the same verdict every time.

Retries and timeouts change the population being scored. A case that times out and is retried gets a fresh generation; a case that times out and is recorded as a failure contributes a zero. Whether your harness does the first or the second is worth knowing, because a 2% timeout rate handled as failures puts a floor of two points of noise under everything.

Sample size sets how much any of that matters. If cases flip verdict between runs with probability f, the standard deviation of the aggregate score is approximately sqrt(f(1-f)/n). That expression is worth building rather than quoting, since every threshold in this chapter is arithmetic on it. Write n for the number of cases in the suite and f for the probability that any one case comes back with a different verdict than it returned last time. Each case is then a two-outcome draw — flipped or not — which is a coin weighted to come up flipped a fraction f of the time, and the suite score is the average of n such coins.

Start with one case, because the whole expression is visible at that size. The variance of a quantity is the average of its squared distance from its own mean, and a case that counts as one when it flips and zero when it does not has mean f, since that is what it averages to over many runs. When it flips its distance from that mean is one minus f, and when it does not the distance is f. Averaging the squares of those two distances, each weighted by how often it happens, gives f times the square of one minus f, plus one minus f times the square of f. Both terms carry a factor of f multiplied by one minus f, and pulling that factor out leaves one minus f plus f inside, which is one — so a single case has variance f times one minus f, the f(1-f) in the numerator.

The score is the sum of those n per-case quantities divided by n, and two steps carry the single-case variance to the variance of that average. Variances of independent quantities add, so the sum carries n times the single-case variance. Dividing a quantity by n then divides its variance by n squared rather than by n, because variance is measured in the square of the score's units and a factor applied to a quantity lands twice inside a square. That leaves n times f times one minus f over n squared, and one factor of n cancels top and bottom, which is the /n. The square root at the end only converts a variance back into a standard deviation in the score's own units, and it is why quadrupling a suite does not quarter its spread: the variance falls by the full factor of four and its square root falls by a factor of two.

Now put this suite's own figures through it. The 7-in-120 flip rate gives f ≈ 0.06, about six in a hundred. Six hundredths multiplied by ninety-four hundredths is a little under six hundredths again; dividing that by the hundred and twenty cases gives about five ten-thousandths; and the square root of five ten-thousandths is a bit over two hundredths of the score, which is the predicted standard deviation of 2.2 points, a point being a hundredth of a score throughout. The three Tuesday runs span a little over four points from top to bottom, roughly what three draws around a 2.2-point standard deviation look like — a consistency check rather than a confirmation, since three numbers cannot pin down a spread.

One step there needs its condition stated, because the rest of the chapter rests on it. Variances add only when the quantities are independent, and per-case flips are not fully independent in a real suite: a provider-side change in batching moves several cases at once, an edit to a shared system prompt moves every case that exercises it, and cases written from one source document tend to flip together. Correlated flips make the true spread wider than sqrt(f(1-f)/n) predicts and never narrower, so the expression gives a best case rather than the number. Finding out how far off it is on a particular suite means measuring the spread directly rather than arguing about the assumption.

Measuring the floor before arguing about the number#

The measurement is a loop, and it costs one run of the suite times however many repetitions you can afford. Ten is enough to be useful and twenty is comfortable.

# evals/noise_floor.py — run the unchanged suite k times, report the spread
import statistics
from evals.run import run_suite  # returns {"score": float, "per_case": {case_id: bool}}

def noise_floor(commit: str, k: int = 20) -> dict:
    runs = [run_suite(commit=commit, cases="evals/cases.jsonl") for _ in range(k)]
    scores = [r["score"] for r in runs]
    case_ids = runs[0]["per_case"].keys()
    flips = [c for c in case_ids if len({r["per_case"][c] for r in runs}) > 1]
    sd = statistics.stdev(scores)
    return {
        "mean": statistics.fmean(scores),
        "sd": sd,
        "min": min(scores),
        "max": max(scores),
        # threshold placing false blocks at ~1 in 100 runs, one-sided
        "gate_for_1pct_false_block": statistics.fmean(scores) - 2.33 * sd,
        "unstable_cases": flips,
    }

Run against that Tuesday commit with k=20, it returned a mean of 0.856, a standard deviation of 0.018, a minimum of 0.821 and a maximum of 0.889. The gate at 0.85 sits six thousandths of a score below that mean, and six thousandths divided by a standard deviation of eighteen thousandths is a third: the gate stands 0.33 standard deviations under the centre of the distribution it is judging. Turning a distance into a probability needs a shape for the distribution, and the shape available is the normal curve, because an average of many independent per-case verdicts is approximately normal whatever the individual verdicts look like — and these verdicts are about as far from normal as a quantity gets, being pass or fail and nothing in between. The mass of a normal distribution below a point a third of a standard deviation under its centre is roughly a third, which is the 37%: roughly one run in three blocks a change that did nothing at all to quality.

Those three Tuesday scores are consistent with that rate, one of them having fallen under the gate, but they are not the evidence for it. Three runs produce one block about as unremarkably at a true rate of a tenth as at a true rate of two thirds, so a rate estimated from them carries almost no precision. The 37% comes from the twenty-run measurement and the three runs are a check on it, which is the reverse of the order most teams work in: three runs and one red check are nothing to argue a threshold from.

Placing a gate is then arithmetic on that distribution, read in the other direction: choose the false-block rate you can live with and ask where the threshold has to go to deliver it. If you are willing to falsely block one pull request in a hundred, the threshold goes 2.33 standard deviations below the mean, at 0.814. The 2.33 is where the normal curve puts one part in a hundred of its mass below a point, read one-sided, counting only the tail on the low side. One-sided is the right reading because only a score falling short blocks a merge: a run that comes back unusually high sails through and no threshold ever objects to it, so spending half the budget on that tail would answer a question nobody asked. Working it through, 2.33 standard deviations of eighteen thousandths each is a little over four hundredths of a score, and taking four hundredths off the mean of 0.856 leaves the 0.814.

That choice has an immediate consequence for detection: a real regression only fails reliably if it pushes the mean below 0.814, which is a drop of a little over 4 points — the same gap between 0.856 and 0.814, read from the other end. Anything smaller will pass most of the time, because a true mean sitting above the gate only crosses it on an unlucky draw. The floor sets both numbers at once, and you cannot improve one without either accepting the other or shrinking the standard deviation.

Those two numbers are really one number, which is worth saying plainly now the arithmetic is done. The gap between the mean and the gate is a single distance read from its two ends. Read downward from the mean, it is how far an unchanged commit must fall on a bad draw before the gate objects, so the further out the gate sits the rarer a false block is. Read upward from the gate, it is how far the true score must drop before the gate objects reliably, so the further out the gate sits the larger a regression has to be before it is seen. Moving the threshold slides one down by sliding the other up, and only a smaller standard deviation shortens the distance itself.

Running the same gate arithmetic across suite sizes shows what the case count actually buys. The spreads below are in percentage points of the score, written pp to keep them distinct from percentages of a percentage: 2.2 pp on a score of 0.856 means the rerun typically lands between 0.834 and 0.878.

Those figures assume the same 6% per-case flip rate and independent flips, so treat them as the shape of the relationship rather than as your numbers. The normal reading behind the third column has a limit too: it improves as the case count grows, and the top row works against it, since six cases in a hundred flipping across sixty cases is a handful of flips per run and a handful of coins is lumpy rather than smoothly bell-shaped. That does not make the row wrong; it makes its tail figure the least trustworthy number in the table, a tail being where a lumpy distribution and a smooth one differ most.

The shape is the point, and it prices a decision rather than describing one. Sensitivity improves with the square root of the case count, and the working goes like this. The smallest regression a gate reliably catches is a fixed multiple of the standard deviation — the same 2.33 of them, since the detection distance and the gate distance are one distance — so shrinking the detectable regression means shrinking the standard deviation by exactly that factor. Going from a 5-point regression to a 2-point one is a shrink by a factor of two and a half. The standard deviation carries a square root over the case count, so buying a factor of two and a half on it costs that factor squared in cases, a bit over six.

The table's own rows check that working. A hundred and twenty cases against eight hundred is a jump of a bit under seven, and across it the smallest detectable regression falls from about 5 pp to about 2 pp. The smaller rows behave the same way: doubling from sixty cases to a hundred and twenty divides the standard deviation by the square root of two, about 1.41, and 3.1 pp divided by 1.41 is the 2.2 pp on the row below. So catching a 2-point regression instead of a 5-point one costs you roughly seven times the cases and seven times the money. Reducing f is the cheaper lever and is the subject of the next chapter, on what a re-run habit does to a gate.

The unstable case list that noise_floor returns is worth more than its summary statistics. Seven named cases that cannot make up their mind are a concrete, fixable object: usually an ambiguous expected output, a grading rubric that does not discriminate, or a case whose correct answer genuinely depends on something nondeterministic. Fixing or retiring them moves the whole distribution.

Two shapes of gate, and why the appealing one is harder#

With the floor measured, the choice between the two common gate designs stops being a matter of taste.

The relative form is the one engineers reach for, because it sounds like it measures the thing you actually care about — did this change make things worse — and because it adapts as the product improves rather than sitting there going slack. Both of those are true. There are two independent objections, and neither is about the idea being wrong so much as about it being harder to implement correctly than it looks.

The first is statistical, and it turns on a step that is easy to get backwards. A relative gate does not test a score; it tests a difference between two scores, and a difference of two noisy measurements is noisier than either. Subtracting feels as though it ought to cancel noise, on the reasoning that whatever made the baseline run unlucky made the candidate run unlucky too. It does not, because the two runs are independent draws rather than one draw seen twice: an unlucky baseline paired with a lucky candidate is exactly as likely as two unlucky ones, so the two errors are as apt to pull the difference apart as to pull it together.

What adds under that independence is the variance rather than the standard deviation, and that is the whole of sqrt(sd_a² + sd_b²). Variance is an average of squared distances from a mean, and squaring throws a sign away, so a quantity and its negative have the same variance — which is why subtracting a measurement contributes what adding it would. The variance of the difference is the baseline's plus the candidate's, and with both runs drawn from the same distribution that is twice one run's variance. Converting back to a standard deviation means taking a square root, and the square root of twice a variance is the square root of two times one standard deviation: about 1.41 of them rather than 2. At 2.2 points each, 1.41 times 2.2 is the 3.1 points of noise the delta carries.

A band of "no more than 2 points below main" is then narrower than one standard deviation of the quantity it is testing. A pull request that changed nothing has a true delta of zero and a measured delta scattered around zero with 3.1 points of spread, so the band sits about two thirds of a standard deviation below the centre, and the mass of a normal distribution below two thirds of a standard deviation under its centre is roughly a quarter: it blocks roughly a quarter of pull requests that changed nothing. The intuition that a relative gate can be tighter than an absolute one is exactly backwards, and the band has to be wider — 2.33 standard deviations of 3.1 points each, around 7 points, for the same 1% false-block rate — unless you do something about the variance.

The something is pairing, and it is worth the effort. Run the baseline in the same CI job as the candidate, over the same case file at the same digest, and compare per case rather than in aggregate.

Why that helps is the step worth following slowly, because it is what makes a relative gate practical at all. A large share of a suite's spread comes from which cases are in it: some cases are simply harder than others, and a suite holding more of the hard ones scores lower for reasons that have nothing to do with any commit. That between-case variance is the largest component of the score's spread, and comparing two aggregate numbers inherits it from both. Comparing per case does not, because the same hard case now appears in both arms, so its difficulty contributes the same amount to each — and a quantity contributing the same amount to both sides of a subtraction contributes nothing to the difference. The cancellation is exact rather than approximate: a hard case is hard for both arms.

What remains is only the cases that flipped between the two arms. A case that passed in both contributes zero to the paired difference however hard it was, and a case that failed in both contributes zero for the same reason. The paired delta is therefore not a difference of averages but a count of flips one way minus a count the other way, and its spread is set by the small unstable subset rather than by the whole suite. The paired delta on that suite had a standard deviation of 1.3 points rather than 3.1, and the useful summary is a count rather than a mean.

# paired comparison: same cases, both arms in one job
base = run_suite(commit=BASE_SHA, cases=CASES)["per_case"]
head = run_suite(commit=HEAD_SHA, cases=CASES)["per_case"]

regressed = [c for c in base if base[c] and not head[c]]   # pass -> fail
fixed     = [c for c in base if not base[c] and head[c]]   # fail -> pass

print(f"regressed={len(regressed)} fixed={len(fixed)} net={len(fixed)-len(regressed):+d}")
# regressed=5 fixed=1 net=-4   -> 5 named cases to read, not a 3-point delta to argue about

Comparing against a stored number from last week's nightly run is the version of this that fails quietly, because that baseline was computed on a different dataset version, possibly a different model snapshot, and certainly a different draw of execution noise. Recomputing the baseline costs a second full run of the suite on every pull request, which doubles the bill and is a real constraint the chapter on tiering the suite has to deal with.

The sentence a reader is most likely to write after meeting these two designs is that a two-point band is a stricter gate than a five-point one. That is tempting because on an ordinary tolerance a smaller number is a tighter requirement, and two points does sound like a finer instrument than five. What it misses is that a gate's strictness is measured not in points but in standard deviations of the quantity compared against it, since that is what decides how often it fires on nothing. Two points against an unpaired delta carrying 3.1 points of noise is two thirds of a standard deviation, which is mostly testing the noise; the same two points against a paired delta carrying 1.3 points is more than one and a half standard deviations, which is a genuinely strict gate. The band is identical in both sentences and only the measurement under it changed.

The second objection has nothing to do with statistics. A relative gate permits a floor that walks downhill. Fifteen pull requests a day, each landing 1 point below the main it was measured against and each re-baselining main on merge, is a suite losing 15 points a day with no single change ever tripping a 2-point band. Real drift is slower than that, since most changes are neutral, but a systematic bias of even −0.3 points per merge is 4.5 points a week and 18 points a month, and the gate is green throughout. This is the same failure as a test suite where each commit adds 40 ms to startup: no commit is the problem, and the aggregate is.

The weekly and monthly figures are worth walking, because they agree with the daily merge rate only under an assumption. Fifteen pull requests a day is about seventy-five across a working week, and three tenths of a point off each of seventy-five merges would be more than twenty points a week rather than four and a half. Four and a half is what comes out if roughly one merge in five touches the behaviour the suite measures at all — fifteen such merges a week, three tenths of a point each — while the rest move nothing. Four and a half points a week across four weeks is the eighteen points a month, so the two figures are one rate at two zoom levels.

The assumption doing the real work is that the bias is systematic rather than symmetric. If each of those merges moved the score by three tenths of a point in a direction as often up as down, the movements would behave like the independent per-case flips earlier in this chapter: the variances would add, and the total after many merges would grow with the square root of their count rather than in proportion to it. Sixty merges of symmetric noise leave a wander of a couple of points as likely to be upward as downward, and no ratchet at all. A relative gate is safe against symmetric noise and defenceless against a small consistent lean, which is why the question to ask of a merge queue is not how large its typical change is but whether its typical change has a direction.

The regression sizes your history actually contains#

Before any of that, three things have to be measured: the standard deviation of the score on an unchanged commit, the standard deviation of the paired delta, and the size distribution of the regressions you have actually caught in the past. That last one decides whether the arrangement is worth having at all, and it is the one nobody computes, because it requires going back through incidents rather than running a script.

If the real regressions in your history are 10 to 15 points, a gate with a 5-point detection floor catches them all and the noise argument is academic — place the threshold from the floor, take the 1% false-block rate, and stop thinking about it. If they are 2 to 3 points, no threshold on a 120-case suite will find them, and the honest conclusion is that you need more cases, a better grader, or a different control entirely. The table above prices the first of those: 300 cases to see a 3-point regression, 800 to see a 2-point one.

What that leaves is a fairly plain instruction. The spread of the score on an unchanged commit is a measurement, not a nuisance, and measuring it is the prerequisite for the whole discussion rather than a refinement of it, because it fixes both of the numbers a threshold argument is really about: how often correct work is blocked, and what size of regression can be seen at all. Twenty runs of a suite you already have gives you both, and any threshold chosen without them is a number somebody liked.

This chapter has assumed you are placing a threshold on a gate nobody has adapted to yet. The harder situation is a gate that has been sitting below its noise floor for six months, where the team has already built a working practice around the flakiness and everyone reports being reasonably happy with it, which is where the next chapter starts.