Confidence intervals and per-language gates

Two different questions get built into the same per-language gate and then blame each other when it is switched off. An absolute floor asks whether a locale is good enough to be sold to customers at all. A regression check asks whether this particular release made this locale worse than the last one. Only the second belongs in a deploy pipeline, and only the second can be made to work on the sample sizes a small locale can afford.

Both rest on the same statistical fact, which is that a pass rate computed over a small number of cases is a very blurry reading of the thing you care about. A confidence interval is the band of true underlying rates that could plausibly have produced the reading in front of you, and a 95% interval is the band you would expect to contain the true rate in nineteen runs out of twenty. Everything a per-language gate can and cannot do follows from how wide that band is at the sample sizes on offer.

Eighty cases buy a fifteen-point band#

Wilson is one of several formulas for computing that band from a count of passes. For 80 cases with 68 passes, the 95% Wilson interval runs from 75.6% to 91.2%:

from math import sqrt

def wilson(passes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = passes / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    margin = (z / denom) * sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
    return centre - margin, centre + margin

lo, hi = wilson(68, 80)
print(f"{lo:.3f} {hi:.3f}")   # 0.756 0.912

A locale whose true quality is 76% and a locale whose true quality is 91% produce the same 85% reading on 80 cases often enough that you cannot tell them apart, so a threshold drawn anywhere inside that band is not measuring quality, it is sampling noise. The Wilson form is worth using rather than the textbook normal approximation because the normal approximation misbehaves exactly where multilingual work lives — small samples and rates near the edges — and will happily hand you an upper bound above 100% for a locale scoring 96% on 50 cases.

Where the fifteen points come from is worth seeing, because its shape governs everything below. The heart of the calculation is the quantity p * (1 - p) / n inside the square root: the pass rate multiplied by the fail rate, divided by the number of cases. The numerator is how much a single case varies — one case is one draw that lands pass or fail, and its variability is largest when the rate sits near a half and shrinks toward either edge. The division by n is there because averaging n independent draws divides their variance by n. The square root converts a variance back into the units of the rate itself, and the 1.96 turns one of those units into the span expected to cover nineteen runs in twenty.

Put the figures in and the quantity under the root comes to about four hundredths, which is just under eight points once the multiplier is applied. Eight points either side is a band a little over fifteen wide. The consequence to carry away lives in that division by n sitting under a square root: halving the width of the band takes four times the cases, not twice. That one fact is why a small locale cannot be rescued by a slightly larger sample, and why the rest of this chapter changes the question rather than buying annotation.

The Wilson form earns its place at the edges. The textbook normal approximation centres the band on the observed rate and extends the same distance both ways, which is harmless in the middle and wrong near a boundary — for a locale scoring 96% on 50 cases the margin works out at over five points, on a rate with only four points of room left above it. Wilson shifts the centre of its band toward one half by an amount that grows as the sample shrinks, which is what keeps both ends inside the range a rate can actually occupy.

How many cases a gate needs before it can see anything#

Sample size is not a property of the locale, it is a property of the regression you want to catch. Detecting a 5-point slip from 85% to 80% with 80% power — the chance the test notices a regression that is genuinely there — at the conventional 5% significance level, which is the chance it fires when nothing has regressed, takes roughly 900 cases in each arm, meaning 900 scored under the old configuration and 900 under the new one. Detecting a collapse to 65% takes about 70.

Those four rows are one relationship. What a test has to do is tell two hypotheses apart, and the evidence that takes depends on how far apart they are — so the cases needed fall as the gap grows, and they fall faster than the gap grows. Quadrupling the effect, from five points to twenty, divides the requirement by roughly sixteen: about 900 becomes about 70. The rule underneath is that the cases required scale with one over the square of the effect you mean to see, which is the same square root as before, read from the other end.

That table is the honest reply to "how big does the Vietnamese eval set need to be." It needs to be big enough for the size of failure you are trying to stop, and the size of failure that actually happens in small locales is the bottom two rows. Locales at the tail do not drift down three points; they break, because the thing that breaks them is a prompt change, a model swap, a tokenizer boundary, or a retrieval path that behaves categorically differently there. Sizing a small-locale gate for a 20-point collapse and saying out loud that it cannot see a 5-point drift is a defensible engineering position. Claiming an 85% floor on 80 cases is not.

The asymmetry between drifting and breaking has a cause worth naming. A slip of a few points is what many small independent influences produce when they move a little in the same direction, and a large locale has many such influences — thousands of conversations across dozens of intents, each nudged slightly by a prompt edit. A locale at the tail runs through a small number of shared code paths, and when one of them changes it changes for everything in that locale at once. The distribution of failures is therefore not the same shape at the two ends of the traffic table, and a gate sized for the shape English produces is sized for a failure Vietnamese does not have.

Two adjustments buy real power before you buy a single extra annotation. Stop sampling the eval set fresh from production each week, because a fresh sample changes both the model's behaviour and the difficulty of the questions at the same time, and you cannot attribute the difference. A fixed set of cases, scored the same way every release, holds difficulty constant. Then compare per case rather than per rate, which turns the analysis into a paired one and is where most of the power comes from.

Run the same 120 fixed Vietnamese cases against last week's prompt and this week's. Ten cases change outcome: nine that passed now fail, one that failed now passes. McNemar's exact test, which discards every case that behaved the same way in both runs and asks only whether the cases that flipped flipped mostly downward, gives a two-sided p of 0.021 on those ten discordant pairs, so the gate fires. The unpaired view of the identical data — 102 of 120 against 94 of 120, 85% against 78.3% — gives a two-proportion p of 0.18 and fires nothing. Same cases, same model, same annotations, and the paired comparison sees a regression that the rate comparison cannot, because it never had to pay for the variance in which questions happened to be sampled.

That p-value is reachable by hand, which is rare enough to be worth doing. Discard the hundred and ten cases that behaved identically in both runs; they carry no information about direction, since they would look the same whichever way the release had gone. Ten cases flipped. If the release had no effect, each flip is a coin, as likely to have gone up as down. Ten coins produce a fixed number of equally likely sequences, and only eleven of them have nine or more landing on the same side — the ten with exactly nine, plus the one with all ten. Divide, and the answer is about one in a hundred, which is the one-sided figure the code prints. Doubling it for the two-sided test gives the 0.021, and nothing along the way is an approximation.

Why the same data reads 0.18 unpaired is the more useful half. A two-proportion test is told only that one set of 120 cases produced 102 passes and another set of 120 produced 94, and it has to allow for the possibility that those were two different sets of questions of different difficulty. That allowance is variance, and it is variance being paid for out of a budget of 120 cases. The paired test is told the two runs saw the same 120 questions, so question difficulty cancels exactly, and every case that behaved identically contributes nothing to the uncertainty. The extra power did not come from more data. It came from removing a source of variation that was never there.

from scipy.stats import binomtest

def regression_gate(pass_to_fail: int, fail_to_pass: int, alpha: float = 0.05):
    n = pass_to_fail + fail_to_pass
    if n == 0:
        return "pass", 1.0
    p = binomtest(pass_to_fail, n, 0.5, alternative="greater").pvalue
    return ("block" if p < alpha else "pass"), p

print(regression_gate(9, 1))    # ('block', 0.0107)
print(regression_gate(6, 4))    # ('pass',  0.3770)

The floor belongs on a roadmap, not in a deploy#

The two questions differ in every dimension that matters operationally, which is why building them into one check produces a check that answers neither.

A locale sitting at the level it has occupied since launch is not a regression, and blocking every deploy on it means every engineer shipping any change is being asked to fix that locale first. That is not a quality gate, it is a tax on unrelated work, and taxes on unrelated work get repealed. The floor still matters — a locale eight points below the rest of the product is a bad number and someone should own it — and it belongs on a roadmap with a name against it rather than in the path of a deploy.

A block does not have to be a block on everything#

The objection that a small-locale gate holds releases hostage assumes the release is one atom, and for most assistant deployments it is not. The prompt version, the model choice, and the retrieval configuration are per-request settings, so a gate that fails for Vietnamese can pin Vietnamese to the previous configuration and let the other eleven locales ship. The release goes out, the regression is contained to the locale where it was detected, and the argument about whether it is real happens on a normal working timescale instead of in a deploy window with people waiting.

That arrangement changes what the gate has to be right about. Blocking a global release on a noisy 120-case signal is a costly false positive, so you want the test conservative; pinning one locale to last week's prompt is cheap and reversible, so you can afford to fire on weaker evidence — a p-value threshold of 0.10 rather than 0.05 roughly doubles the power at the sizes in the table above. Loosening a threshold does exactly one thing, in both directions at once: a test fires when the evidence clears a bar, and lowering the bar lets more real regressions through and more false ones with them, in whatever ratio the two happen to exist. Nothing about the evidence improved. What changed is the price of being wrong, and the price is what a threshold should be set from. The cost of being wrong is a small locale running a week-old configuration, which is a state it can sit in indefinitely without anyone being harmed.

Tiering, so twelve locales do not cost twelve times one locale#

Not every locale deserves the same machinery, and pretending otherwise is how teams end up with nothing. Two axes decide the tier. Traffic share sets how much aggregate damage a failure does, and risk sets how much damage a single failure does — a regulated market, a locale where the assistant quotes statutory rights, a market with a named enterprise customer, or one where a regional press cycle is a live possibility. The two are independent, which is the point: a locale at 1% of traffic in a market where the assistant gives tax-adjacent answers belongs in the top tier, and a locale at 9% of traffic serving a low-stakes consumer feature does not necessarily.

The third tier is doing something specific and it is not statistics. Forty cases cannot detect a regression at any respectable confidence, and they can detect a locale that has stopped working altogether — empty responses, answers in the wrong language, a systematic refusal, output that is not in the right script. Those are the failures a tiny locale actually suffers, they show up in the first five cases when they show up at all, and a smoke set catches them at a cost of about an hour of annotation per quarter.

The override in tier 2 is load-bearing. A gate with no legitimate way past it is a gate that gets disabled the first time it is wrong at an inconvenient moment, and disabled gates catch nothing. A gate that can be passed by a named person typing a reason into the release record survives, because the cost of overriding it is a sentence rather than a fight, and the record of who overrode what for which locale is itself the evidence you need six weeks later when the regional manager escalates.

Twelve tests fire more often than one#

Running twelve independent tests at α = 0.05, where α is the false-fire rate each test is allowed on its own, gives roughly a 46% chance that at least one fires in a release where nothing regressed anywhere, and a 46% false-block rate is precisely how a gate loses its credibility. That figure is one multiplication repeated. A test allowed to fire falsely one time in twenty stays quiet nineteen times in twenty, so its quiet probability is nineteen twentieths. Twelve independent tests are all quiet together with that fraction multiplied by itself twelve times, which comes to a little over half, and at least one firing is everything else — a little under half, the 46%. The word carrying weight there is "independent", and it is close enough to true in this setting, because the locales are scored on different cases by different annotators, so one test firing spuriously says nothing about whether another will. Holm's correction over the twelve locales tightens the threshold each test has to clear so that the chance of any false block across the whole family stays near 5%, and it pays for that family-wise guarantee with power in exactly the small locales that had little to begin with. The practical compromise most teams land on is to correct across tier-1 and tier-2 locales, where a block has teeth, and to leave tier 3 uncorrected and non-blocking, since a noisy notification costs a glance.

Pinning carries its own operational tax. Locales pinned to different prompt versions means more than one configuration is live, which complicates every subsequent debugging session and means the trace for a conversation has to record which version served it. That is a real cost, and it is smaller than the cost of a global gate nobody trusts.

There is also the question the intervals cannot answer. Everything above treats a case as pass or fail against a rubric, and the rubric was written for the English set and translated. A gate can be perfectly powered, correctly tiered, properly corrected, and still be measuring whether the Vietnamese answers satisfy an English idea of a good answer. That is a real limitation and it is not a statistical one; it comes back when the eval sets themselves are on the table.

A per-language gate is worth building only in the form the sample sizes support: a paired regression check on a fixed set, sized for a collapse rather than a drift, with the shippability question moved somewhere it can be argued about slowly. What that buys you is a release that stops when a locale breaks, and the next question is what breaking looks like. The failures a gate catches in Vietnamese or Swahili do not resemble English failures — the answers do not come back garbled or obviously wrong, which is why humans reading them casually sign them off, and why the specific shape of low-resource degradation is worth knowing before you write a single case.