Turning 90% into failures somebody has to handle on Tuesday

The message arrives at 09:40 on a Wednesday and it is from Priya, who runs the operations team that works downstream of the classifier you shipped three weeks ago. "This thing is broken. We had 16 failures yesterday. You told us 90%." She is not being unreasonable, and she is not being innumerate. She was told a number, she counted a different number, and nobody ever showed her how the two are related. The classifier handles about 100 requests a day. At 90% it produces about 10 failures a day. Sixteen is a Tuesday that will happen roughly once a month, and Priya has no way of knowing that, because the only thing she was given was a percentage.

The classifier is built on a large language model, which makes it a different kind of dependency from the services Priya's team has worked alongside for a decade, and two of its properties do most of the damage in conversations like this one. The first is that the model is wrong on some fraction of entirely ordinary inputs, so its quality is a rate — the share of requests it handles correctly across some population of them — rather than something that either works or is broken. The second is non-determinism: the same input sent twice can come back with different output, because the model draws each piece of its answer from a probability distribution over the possible continuations, and because the serving machinery underneath it varies in ways that never appear in a version number. Between them, those two properties mean there is no such thing as the answer the classifier gives to a particular request. There is a spread of answers, and a percentage is a summary of that spread.

Setting expectations on non-determinism is the work of making both of those facts visible to the people who will live with the system, in the units they already use, before they meet the consequences on their own. It is needed because a percentage does not survive the trip across an organisation. The same "90% accurate" is heard by one person as a promise the system keeps and by another as a system that fails one time in ten and is therefore broken, and each reading produces its own bad decision: the first writes the number into a requirements document as though it were a guarantee, and the second escalates an ordinary Tuesday as an outage. The situations where this lands are recurring and recognisable — a demo that becomes a requirement overnight, a launch criterion nobody can pass or fail, a message like Priya's, a Monday when the measured quality drops after a model update and somebody has to decide who hears about it, and a request that cannot be met at any rate worth promising. Each of those is a conversation rather than a piece of code, and each has a technique that works better than the one most people reach for by instinct.

Converting a rate into a count of wrong outputs is the arithmetic from designing around an imperfect component, and it is the floor here rather than the point. What that earlier work does not give you is the second number, which is how much the count moves from one day to the next when nothing at all has changed. A percentage sounds like a fixed property, so a count derived from it sounds like a quota — ten failures, every day, ten. Real days do not behave that way, and the gap between the quota people imagine and the spread they observe is where trust goes.

Ten a day means four to sixteen#

If each request succeeds independently with probability 0.9, the number of failures in a day of 100 requests follows a binomial distribution — the arithmetic of counting how many out of n independent yes-or-no trials land on one side — with mean 10 and standard deviation sqrt(100 × 0.9 × 0.1), which is 3. Two standard deviations either side of the mean covers roughly 95% of days, so ordinary days run from 4 failures to 16. Sixteen is not a signal. It is the top of the ordinary range, and the probability of seeing 16 or more on any given day is about 4%, which over twenty working days makes it more likely than not that at least one day in a month looks like Priya's Tuesday.

from scipy.stats import binom

n, p_fail = 100, 0.10
dist = binom(n, p_fail)

print(dist.mean(), dist.std())          # 10.0 3.0
print(dist.ppf(0.025), dist.ppf(0.975)) # 5.0 16.0  -> ordinary range
print(1 - dist.cdf(15))                 # 0.0399  -> P(16 or more) on a given day
print(1 - (1 - 0.0399) ** 20)           # 0.558   -> P(at least one such day in a month)
print(dist.cdf(4))                      # 0.0237  -> P(4 or fewer), the "it got better" day

The same arithmetic produces the other half of the conversation, which is the day somebody reports that the classifier has improved because there were only four failures. That happens about once every two months, and it means nothing either. Both directions matter, because a team that only hears from you when the number is bad learns that the number is a verdict rather than a measurement.

Volume changes the shape of this considerably, and it is worth doing on a whiteboard with the actual numbers rather than in the abstract.

The first row is the one people find most surprising. At 20 requests a day, a day with zero failures and a day with five failures are both entirely ordinary, so a single day carries almost no information about the underlying rate and there is no threshold worth setting on it. Small-volume features have to be watched over a week or a fortnight, and telling an operations lead that up front prevents a fortnight of Monday-morning post-mortems about noise. The last column is the number you actually want them to have, because it is the only one that should generate a phone call. Everything below it is the system working as described.

Doing it with them, not at them#

The calibration is an exercise you run in a room with the people who will absorb the failures, and it takes about forty minutes. Sending them a document with the same numbers in it does not work, for the ordinary reason that a number somebody derives with you is a number they can defend to their own team, and a number that arrives by email is a number they will quote back at you inaccurately.

The sequence is fixed and each step uses their units rather than yours. Start with their volume, which they know and you probably only estimated, and let them correct it — a feature you sized at 100 requests a day is frequently 140 on Mondays and 60 on Fridays, and that alone changes the thresholds. Then state the rate along with how it was measured and on what, because a rate with no provenance invites the response that it was measured on easy cases, which is often fair. Multiply to get expected failures per day, per week, and per person if the work is divided. Compute the ordinary range and write both ends on the whiteboard, because the lower end is what stops the first good week being read as evidence that the feature is better than you said. Set the number that triggers a call, and agree explicitly that everything below it is not an incident. Finally, multiply the expected count by the minutes a correction takes to get a staffing figure, and ask whether that capacity exists.

The output of the meeting is four numbers and a name: expected failures per period, the ordinary range, the escalation threshold, the correction capacity in hours, and the person who owns the failure path. Those go somewhere both teams can find them, which in practice means the same page as the runbook rather than a slide deck. When the number moves later — and it will move, because rates drift and volumes grow — you update that page rather than re-running the whole conversation from scratch.

The version that has to survive contact with twelve people#

Take the case that comes up most often, which is a team of twelve processing 3,000 items a day against a feature measured at 92%. The percentage is the input; nothing else in the following paragraph is, and all of it is what the operations manager actually needs.

Three thousand items at 92% means about 240 items a day come out wrong, which is 20 per person per shift. The standard deviation is sqrt(3000 × 0.92 × 0.08), just under 15, so an ordinary day lands between 210 and 270 and a day at 250 is not a bad day. A day at 300 is outside anything the model predicts and means the input mix or the system changed. If a correction takes four minutes on average — and that four minutes is an estimate that should be measured within the first fortnight rather than assumed — then 240 corrections is 16 person-hours a day, which is two of the twelve people, full time, doing nothing else. That is the real content of "92% accurate": it is a headcount, and it is a headcount before you have argued about whether 92% is good.

Errors also do not distribute evenly across the twelve. If one person owns the queue with the unusual document types and that queue runs at 78% rather than 92%, their personal experience of the feature is one wrong item in five, and they will describe the system as broken in a way that is accurate for them and misleading for everyone else. Doing the arithmetic per queue, per document type, or per customer segment — whichever slice the team already organises itself around — costs an extra ten minutes in the calibration meeting and prevents six weeks of contradictory anecdotes. The Randomness You Didn't Ask For is a serviceable plain-language piece to hand to a non-engineer who wants to understand why the output varies at all, but the slice-level arithmetic is the part you have to do yourself, because it depends on their queues.

What you ask for in return#

The calibration is half a transaction. Having spent forty minutes converting your rate into their workload, the thing to do next is state plainly what the feature needs from the operations side in order to stay at the number you quoted, and there are two items.

The first is a correction path with capacity attached. Somebody has to fix the 240, and "the team will absorb it" is not a plan, because the team is already sized for the work they had before. Two full-time equivalents of correction capacity, named, with a fallback for the days that run at 270, is a plan. The second is a person who reports what went wrong rather than that something went wrong. A count of failures tells you the feature is behaving as measured; it does not tell you which 240, and without that you cannot improve anything. What you want is a weekly handful of item IDs with a one-line category — wrong queue, hallucinated field, refused a valid input, formatting broke the downstream import — from a named person whose job description includes it.

The shallow version of this conversation is worth naming, because it is the version that gets had by default. It goes: "92% accurate means 8% will need fixing, which is much better than the current manual process, so just flag anything that looks wrong." Every clause is defensible and the whole thing is useless. It gives no count, so nobody can staff it. It gives no per-person load, so the twelve people discover their new workload by experiencing it. It gives no spread, so the first 270-day gets escalated as an outage. It sets no threshold, so nothing distinguishes a real regression from Tuesday. And "flag anything that looks wrong" is an unbounded request that produces either nothing or an unsorted pile, both of which are unusable.

Where this arithmetic stops being true#

The binomial model assumes each request fails independently of the others, and production violates that assumption in the two situations you most need to detect. A model or prompt change moves the underlying rate, so the failures are not drawn from the same distribution as yesterday's; a shift in the input mix, such as a large customer onboarding with a document format nobody has seen, does the same thing. Both produce a spike that the independence model calls essentially impossible, and that is exactly what makes the model useful as a detector. A 400-failure day at 3,000 items is not bad luck at any plausible reading of the arithmetic, so it is a change, and it should be investigated as one.

The other boundary is what the calibration does not decide. Converting 92% into 240 items and 16 person-hours tells you the size of the consequence; it says nothing about whether that consequence is worth accepting, which depends on what one wrong item costs and who absorbs it. Those are separate questions and they were the subject of the earlier work on imperfect components. Keep them separate in the meeting too, because mixing them lets a stakeholder respond to an arithmetic fact with a value judgement — "240 sounds like a lot" — and the conversation stops being about counts.

There is also a question this chapter has quietly taken for granted, which is where the 90% came from. In Priya's case the number came from a spreadsheet somebody built during a pilot, and it was reasonably well measured. Very often it did not. Very often the number entered the organisation because a feature worked in a meeting, someone counted the attempts, and a rate was born. What twenty attempts can and cannot establish about a rate turns out to be a much sharper question than most people expect, and it is the one that decides whether any of this arithmetic was built on sand.