Bounding a failure rate

A pre-launch check normally consists of somebody sitting down with a staging environment and typing questions at the feature until they are satisfied. It is a reasonable thing to do and it catches real problems. What it cannot do is tell you the rate at which the feature fails.

This chapter works out what a clean run of n cases actually establishes, using a support assistant that answers customer questions from retrieved policy documents. There are two separate defects to keep apart: twelve draws cannot resolve a rate in the single digits, no matter how carefully those twelve are chosen, because the bound comes from the count; and questions a person invents while testing are systematically unlike the questions customers send, so a hand-written suite can grow to hundreds of cases and still contain none of the inputs that fail.

What twelve consecutive passes rule out#

If each question fails independently with probability f, the probability that twelve in a row pass is (1 - f)¹².

Build that expression rather than accepting it, because the rest of the chapter is arithmetic on top of it. Let f be the probability that any one question comes back wrong — the quantity nobody knows and everybody wants. A question either fails or passes and there is no third outcome, so the probability it passes is 1 - f. Two questions both passing is (1 - f) times (1 - f), and the multiplication is allowed because the two are independent: each is a separate call to the model, nothing carries over between them, and the first answer being wrong tells you nothing about the second. Twelve of them is that same factor multiplied in twelve times, which is what the superscript means.

Sanity-check it at the ends before trusting the middle. If f is zero the expression is one multiplied by itself twelve times, which is one — a feature that never fails always produces a clean twelve. If f is one, every factor is zero and the product is zero — a feature that always fails never produces a clean twelve. Both of those are obviously right, which is the point of checking them: an expression that gets the extremes wrong is wrong everywhere, and one that gets them right is at least the correct shape.

Put some numbers through it:

Read the third row. At a failure rate of one call in twelve — bad enough that a support team notices within a fortnight — a twelve-question check comes back completely clean about a third of the time. A clean twelve is not a lucky result for such a system; it is that system's single most probable outcome.

Turning the table around gives the bound. The largest failure rate that twelve passes rule out at 95% confidence — meaning the largest f that would have produced a clean twelve less than one time in twenty, so that ruling it out is wrong at most one time in twenty — solves (1 - f)¹² = 0.05, which is f = 1 - 0.05^(1/12) = 0.221. Twelve clean answers are consistent with a true failure rate anywhere from zero to about 22%. Stated honestly, that is the entire content of such a check: the assistant fails somewhere between never and one call in five.

Each move in that rearrangement is worth naming, because the same three steps produce every bound in this chapter. The starting point is the definition of the confidence level: 0.05 is one time in twenty, and it is the largest probability of a clean twelve you are willing to call implausible. Setting (1 - f)¹² equal to it asks which failure rate would make a clean twelve exactly that implausible, and every rate worse than that one is ruled out because it would make a clean twelve even less likely than the thing you already decided not to believe. Then 1 - f is recovered by taking the twelfth root of both sides, which is allowed because raising to the twelfth power and taking the twelfth root undo each other on positive numbers, and 1 - f is positive since f is a probability. That leaves 1 - f = 0.05^(1/12), and subtracting both sides from one gives f itself.

Check that against the case small enough to verify without a calculator. With one question instead of twelve, the twelfth root is no root at all, and the bound collapses to one minus the confidence level: a single clean answer rules out only a failure rate above about ninety-five per cent. That is obviously correct — one good answer tells you the feature is not broken on literally every call — and it is the right anchor for the intuition, because it says the bound starts near-useless and improves with each additional clean observation.

def upper_bound(n: int, confidence: float = 0.95) -> float:
    """Largest failure rate consistent with n consecutive passes."""
    return 1 - (1 - confidence) ** (1 / n)


for n in (12, 15, 20, 30, 50, 100, 300, 1000):
    print(f"{n:>5}  {upper_bound(n):.3f}")

   12  0.221
   15  0.181
   20  0.139
   30  0.095
   50  0.058
  100  0.030
  300  0.010
 1000  0.003

The shape of that column is the thing to carry away. The bound falls roughly as 3/n — a rule of thumb old enough to have a name, the rule of three, which arrived from clinical safety work in the early 1980s asking exactly this question about drug trials: if nothing went wrong in n patients, what rate of harm have you actually excluded? Same arithmetic, same answer. Ruling out a 1% failure rate takes around 300 clean observations. Ruling out 3% takes around 100. Twelve buys you 22%, and no amount of care in writing those twelve questions changes the number, because the number comes from the count.

Where the three comes from is worth a paragraph, because knowing it tells you when the rule of thumb stops applying. The exact statement is that the product of the failure rate and the sample size comes out at the natural logarithm of twenty — twenty being one divided by the one-in-twenty you chose as the confidence level — and that logarithm is just under three. So the three is not a fudge factor. It is the price of demanding 95% confidence, and demanding a stricter level — being wrong one time in a hundred rather than one time in twenty — replaces it with the logarithm of a hundred, nearer five, which loosens every bound in the column above by that ratio.

The approximation behind it holds only while the failure rate is small, which is worth stating rather than discovering. It rests on multiplying 1 - f by itself n times being very close to a fixed proportional shrinkage applied n times, and that closeness is what turns an awkward power into a division. Where the bound is a fifth or more the approximation is visibly loose — three divided by twelve is a quarter, against the exact 0.221 the code prints — and where the numbers actually carry a launch decision, in the low hundreds of cases, it agrees with the exact figure to the digit anybody would act on.

That is also why the intuition fails so reliably. Going from twelve cases to twenty-four feels like doubling your confidence, and it does halve the bound — from 22% to 12% — but 12% is still a rate that would have every support lead in the building on a call. The interesting region, where the bound is small enough to carry a launch decision, starts in the low hundreds of cases and is not reachable by anyone typing questions by hand on a Tuesday afternoon.

A sample drawn from the wrong distribution#

Suppose the pre-launch check had been 300 questions rather than twelve. The bound falls to about 1%, and the check can still miss the failure entirely, because hand-written questions and real conversations are not samples from the same distribution.

Nothing in the arithmetic above was ever a statement about your users, and this is the moment that becomes expensive. Every step of the derivation took f to be the failure rate on questions drawn the way these questions were drawn. Three hundred clean hand-written questions therefore bound the failure rate on hand-written questions, which is a real quantity and not the one anybody cares about. The count fixes how precisely you have measured whatever you sampled; it has no opinion about what you sampled. A perfectly executed measurement of the wrong population is still a perfectly executed measurement, and it will carry the same confident 1% into the launch review.

Here is what those two populations look like side by side, for one assistant whose pre-launch twelve were compared against a fortnight of live traffic:

The pre-launch twelve are all well-formed single questions of about a dozen words, because that is what a person writes when they sit down to test a thing. They know what the feature does, so they ask it what it does. Nobody typing test questions writes 200 words of frustration with an order confirmation pasted in the middle, then follows up three messages later with "so what about the other one", because that is not a question — it is what a customer does.

The failing conversations in that fortnight clustered exactly where the twelve had no coverage: most of them were long, and most contained a false premise the assistant accepted and built on. A hand-written suite could reach 300 cases and still contain zero of those, because the property that makes them fail is the property that makes them unnatural to invent.

Two distinct defects, then, in the same check, and they call for different fixes. Too few draws is fixed by taking more draws. A sample from the wrong distribution is fixed by changing where the sample comes from — real logged traffic, sampled to match the mix in the right-hand column, with the rare shapes deliberately over-represented so that a 3% category is not represented by two cases. Neither fix substitutes for the other, and a suite of 500 hand-written questions has fixed the easy half.

Patching against four known failures#

The natural response to a production failure is a pull request that adds the failing conversations to the suite. Support attaches four specific transcripts where the assistant stated a refund policy the company does not have; somebody adds those four prompts with the corrected expected answers, titles it fix hallucinated refund policy, patches the prompt until the four pass, and the build goes green.

That pull request does something, and what it does is worth stating precisely, because it is neither nothing nor what the title claims. It fixes four strings. A prompt patch that makes four cases produce the corrected answer has been optimised against four examples, which is enough to change the wording of a specific instruction and not remotely enough to establish that the underlying behaviour changed. The same failure with a different policy, a different phrasing, or a longer context is untouched, invisible, and now has a green build vouching for it.

The arithmetic makes the coverage argument concrete. Suppose the true rate of hallucinated policy statements is 4%. Fifteen hand-picked demo prompts come back clean with probability 0.96¹⁵ = 0.54 — a coin flip, which is to say the demo carried no information about a 4% failure rate at all. Having a 95% chance of catching at least one instance needs n ≥ ln(0.05) / ln(0.96) ≈ 74 cases drawn from traffic.

That last line is the same equation as before with the unknown moved. Earlier the sample size was given and the failure rate was solved for, so both sides were raised to a fractional power; here the failure rate is given and the sample size is solved for, which is what the logarithms are doing. A logarithm brings an exponent down to where ordinary division can reach it, and the ratio of the two is the number of times 0.96 has to be multiplied by itself to fall to 0.05.

Adding four takes such a suite from fifteen to nineteen and moves the 95% bound from 18% to 15%, against a failure rate of 4%.

There is a subtler problem with those four. They were selected because they failed, which means they are drawn from the failure distribution rather than from traffic. Once they are in the suite and the prompt has been patched to satisfy them, they are guaranteed to pass and will keep passing for as long as nothing touches that phrasing. They contribute no information to the measured rate ever again, while inflating the denominator. A suite of nineteen cases where four are pinned known-failures reports a higher pass rate than the same nineteen would have before the patch, and the improvement is an artifact of how the four were chosen.

Keeping them is still right. Four reproductions of a known bug are worth having as regression cases, clearly labelled as such and excluded from the rate. What they cannot be is the response to an incident.

What twelve questions are good for#

Reading any of this as an argument against manual checking would throw away a genuinely efficient tool. Twelve hand-written questions detect a whole class of failure with near-certainty in five minutes: the feature is wired to the wrong prompt, the retriever returns nothing, the response never parses, the deployment is pointing at last week's index, the assistant answers in the wrong language. Those failures happen on every case, so a sample of twelve finds them with probability essentially one, and a sample of five hundred adds nothing except cost. Any failure mode with a rate near 100% is fully characterised by a handful of examples.

The trouble is entirely with rates in the single digits and low tens, which is where every failure mode that survives to production lives — the ones that fire on every call get caught by whoever first opens the staging environment. So a manual pass keeps its place at the front of the process, as the check that the thing is plugged in, and the number it produces is a yes-or-no about catastrophe rather than an estimate of quality.

The rule of three has its own limits, and two are worth naming before it gets used as a launch calculator. It assumes independence between cases, which fails the moment a sample contains near-duplicate tickets from the same customer or several turns of one conversation — a set of near-duplicates carries the information of roughly one case, so the effective count is smaller than the number of files on disk and the true bound is looser than the formula says. And it bounds one rate over one population, so a bound computed on a mixed sample says nothing about a slice inside it: 300 clean cases rule out a 1% overall failure rate while leaving a far higher rate on the 6% of traffic that is not in English entirely possible.

The number that comes out of a clean run is a ceiling and not an estimate. Twelve passes say the failure rate is below 22%; three hundred say it is below 1%; neither says what the rate is, and neither says anything about the failure modes no hand-written question was ever going to trigger. Fixing the first defect is arithmetic; fixing the second means drawing from logged traffic rather than from imagination, which is the harder half, because it needs a sampling scheme and somebody willing to read real conversations.

Once both are fixed, the check reports a rate rather than a verdict — the fraction of sampled conversations containing an unsupported policy statement, before a fix and after it. Comparing those two numbers is where the next difficulty starts. A rate measured over a few hundred cases is itself an estimate with a spread, two runs of the identical prompt over the identical cases will not report the identical rate, and a drop from 4% to 3% may be a genuine improvement or the same system measured twice. The next chapter works out how wide that spread is.