Nineteen successes out of twenty tells you almost nothing

The demo runs for eighteen minutes. Somebody in the room asks for a twentieth question at the end, off-script, and the assistant gets it slightly wrong — close enough that the presenter says "that one's a formatting thing" and everybody nods. Nineteen out of twenty. The next morning the product requirement document has a line in it that reads the assistant answers policy questions correctly at least 95% of the time, and three people have already quoted the 95% in messages to other teams. Nobody lied. Somebody counted, divided, and wrote down what they got.

Twenty attempts is not enough attempts to distinguish a feature you should ship from a feature that will generate a complaint every fourth interaction, and the size of that gap is worth computing rather than asserting.

What twenty attempts can actually pin down#

An exact binomial confidence interval — the Clopper–Pearson interval — answers the question directly: given 19 successes out of 20, what range of true success rates would produce that observation without it being surprising? At the conventional 95% level — meaning the procedure that builds the interval captures the true rate in about 95 uses out of 100 — the answer runs from about 75.1% to 99.9%.

from scipy.stats import beta

def clopper_pearson(k, n, alpha=0.05):
    lo = beta.ppf(alpha / 2, k, n - k + 1) if k > 0 else 0.0
    hi = beta.ppf(1 - alpha / 2, k + 1, n - k) if k < n else 1.0
    return lo, hi

for k, n in [(19, 20), (47, 50), (95, 100), (285, 300), (475, 500), (950, 1000)]:
    lo, hi = clopper_pearson(k, n)
    print(f"{k}/{n}: {lo:.1%} - {hi:.1%}  (width {100 * (hi - lo):.1f} points)")

Every row reports the same point estimate — the single number you get by dividing successes by attempts — and they are not the same claim. A true rate of 75% on a feature running 3,000 times a day, at the volumes from the last chapter, produces 750 failures a day rather than 150, which is a different product and a different staffing plan. The interval at twenty samples spans both worlds, so quoting 95% from a demo is quoting the midpoint of a range you have not narrowed.

Narrowing it is a matter of arithmetic and someone's afternoon. Pinning a rate near 95% to within ±2 points needs roughly 460 labelled examples; to within ±1 point, roughly 1,800. Those are not enormous numbers when the labels come from traffic you already have — 460 tickets sampled from last quarter, with a rubric that states what counts as a correct answer and two afternoons of a support lead's time, is a real budget line and a small one. The useful thing to carry into a meeting is that the cost of turning a demo into a rate is measured in days rather than months, which removes the excuse that proper measurement is a research project.

Repeatability is a separate axis from coverage#

Even a perfectly sampled 460-item evaluation only measures the population it sampled from. A demo does something weaker along several axes at once, and separating them makes the refusal easier to state without sounding like an objection to demos.

Two entries in that table use terms worth pinning down before the rows are read. A shadow run sends real production traffic to the feature in parallel with whatever handles it today, records what came back and shows it to nobody, which buys the genuine input distribution without exposing anyone to the failures. Red-teaming is the deliberate hunt for inputs that break the system — hostile phrasing, instructions smuggled into a document, questions built to elicit a confident answer where the source material contains none — and it stays a separate exercise from sampling, because those inputs are rare in ordinary traffic and common in the events that turn into incidents.

The demo row is blank almost everywhere, and the coverage column is where it fails first. The twenty inputs were chosen by the person who built the thing, which is not a criticism of them — of course you demo the cases you have made work. It does mean the sample is drawn from a different distribution than production, so the Clopper–Pearson interval is the generous reading. That interval assumes the twenty attempts were independent draws from the population you care about. They were drawn from the presenter's shortlist, run once each, in a single session, against one model version, on one afternoon, and graded by the person who wanted them to pass.

Repeatability deserves its own attention because it is the axis engineers most reliably forget. The same input can produce different output across runs even with temperature set to zero. Temperature is the setting that controls how much randomness enters the choice of each next token, and zero is the value people assume removes the variation altogether, which is why the residual movement surprises nearly everyone who meets it. The causes have nothing to do with sampling: batching effects in the serving stack and floating-point non-associativity in accumulation both make the result depend on what else happened to be in the batch alongside your request. The paper Non-Determinism of "Deterministic" LLM Settings measures this directly and is the citation to end the argument when somebody insists temperature zero makes the system deterministic. A demo run once per input measures nothing about this dimension, so a question that answered correctly at 14:20 in the meeting is not established to answer correctly at 14:20 tomorrow.

Measuring it costs one extra pass over the evaluation set. Running the same 460 items three times and counting how many changed verdict between runs gives a flip rate, and on a retrieval-backed assistant with temperature at zero it is common to see something in the region of 4% to 8% of items flip at least once. That number matters for two separate reasons. It tells the operations side that a question a user asks twice may genuinely get two different answers, which is a support script rather than a bug report. It also tells you how much of a later movement in the headline rate is noise from the serving stack rather than a change in the system, which is the difference between an investigation and a shrug. Barry Zhang's Making Peace with LLM Non-determinism separates the contributing causes — sampling, batching, hardware, and the model's sensitivity to trivial wording changes — more carefully than most treatments, and is worth reading before you try to explain any of it to a stakeholder.

Why the instinct is reasonable and still wrong#

In deterministic software, a demo is close to a spec, and treating it as one is a habit built on solid ground. If a payment flow completes correctly in a demonstration, and the code is unchanged, it will complete correctly again — modulo environment differences that engineers already know how to reason about. A test suite that passed 19 times out of 20 would not be described as 95% accurate; it would be described as having a flaky test, and somebody would be assigned to fix it before the release. One failure in twenty is an incident in a deterministic system and a data point in a probabilistic one, and stakeholders who have spent fifteen years in the first world carry the first world's instincts into the second. César Soto Valero's Why Software Engineers Struggle With AI Systems is a decent thing to send a colleague who is feeling this dissonance without having named it.

The consequence is a specific and very common conversion error. Somebody sees a working demo and concludes the capability exists, then treats the remaining work as engineering polish — error handling, retries, a nicer interface. What the demo established is that the capability exists sometimes, and the distance between sometimes and reliably is not polish. It is the whole quality problem, and it may not close at all for the input population that matters.

Refusing it without refusing the demo#

Rejecting the demo outright is a bad move for reasons that are political and also technical: the demo did establish things, and dismissing it teaches people to stop showing you their work. The sequence that works is to credit precisely what it proved, name the missing quantity, and price the work to get it.

Crediting is specific rather than polite. A twenty-question demo establishes that the integration is wired end to end, that latency in the interactive path is acceptable, that the output format parses, that the retrieval step surfaces the right documents on at least some real questions, and that the one failure you saw is a genuine failure mode worth chasing. That is a substantial list and stating it out loud costs nothing.

Then the missing quantity, in one sentence: twenty attempts is consistent with a true rate between about 75% and 99%, and at our volume those two ends differ by several hundred failures a day. Then the price: a sample of 460 real questions from last quarter's tickets, a rubric, and two people for two afternoons gets us to plus or minus two points, and we can have it before the date you care about. Almost nobody argues with that, because it is a proposal rather than a veto, and because the alternative on offer is finding out the rate from users.

Where the 460 questions come from decides whether the whole exercise was worth doing. Pulling them at random from last quarter's tickets is the default and it is usually right, because it reproduces the mix of easy and hard that production will actually send. Random sampling does under-represent the categories that are rare and expensive, so the common adjustment is to stratify: sample proportionally within each policy area, then top up the two or three areas with fewer than thirty examples so that each one has an interval you can read on its own. Grading needs a written rubric fixed before anybody looks at outputs — correct, partially correct with the specific defect named, wrong — and two independent raters on a subset of about eighty items so you can report how often they disagreed. A 12% disagreement rate is worth knowing, because it puts a floor under how precisely the headline number can mean anything, and it usually reveals that the definition of "correct" was doing more work than anybody realised.

One phrasing to avoid is the counter-offer that keeps the shape of the mistake, which is running the demo again with more questions and reporting 47 out of 50. Fifty scripted questions chosen by the same person has the same coverage problem as twenty and a fifteen-point interval instead of a twenty-five-point one. The sample has to come from somewhere other than the person who wants it to pass.

What a tight interval still does not buy#

An interval is a statement about sampling error alone, so a narrow one on the wrong population is more dangerous than a wide one on the right population, because it carries an air of rigour. Measuring 950 out of 1,000 on a benchmark assembled two years ago gives you ±1.6 points on a question nobody is asking. The interval also says nothing about how the failures are distributed: 5% failures spread evenly across all users is a nuisance, and the same 5% concentrated in one customer's document type is an account at risk. Slicing the evaluation the way the last chapter sliced the workload — by queue, by segment, by document type — matters more than the third decimal place on the headline.

The demo's most durable damage is not the number people repeat in corridors, which decays. It is the sentence it leaves behind in the launch criteria document, where it sits with the authority of a written requirement and cannot be passed or failed by anyone. That sentence has a specific set of things missing from it, and rewriting it is the easier half of the job.