Choosing a confidence signal

Every method in this topic produces a signal with a cost and a coverage: the single-token probability from chapter 2 is free and applies only to label outputs; the mean over free text from chapter 3 is free and mostly measures phrasing; agreement across samples from chapter 5 costs N generations and the latency of the slowest; a second model acting as a verifier costs one extra generation and inherits its own blind spots. Choosing between them looks like a design argument and is not one. It is a model evaluation problem, answerable by scoring each candidate against the same labelled cases and reading off how much of the error each one catches at a review volume somebody can staff.

Two constraints usually settle most of it before any measurement happens, and both are arithmetic. The first is the inference budget, which rules out any method whose cost multiplier exceeds it — worth checking in the first two minutes of a planning meeting rather than after a week of benchmarking. The second is review capacity, which fixes the operating point every candidate has to be compared at, because a signal that catches 90% of errors at 30% review volume is not an option that exists when reviewers can cover 6.5%.

The evaluation set decides everything downstream#

The comparison needs one fixed labelled set that every candidate is scored against, and two properties of that set matter more than its size.

It has to be real traffic, because the failures worth catching are the ones a model produces on inputs nobody anticipated, and a curated set contains the inputs someone anticipated. And it has to be stratified, meaning sampled deliberately by case type instead of at the rates traffic happens to produce. Where the cases that justify the whole review process are 8% of volume, a 1,500-case sample drawn naturally contains roughly 120 of them and a handful of errors among those, which is not enough to distinguish any two methods. Oversampling the important stratum and reweighting when reporting overall figures costs nothing and is the difference between a measurement and a coin flip.

The set also needs an owner and a refresh cadence, because it decays. Traffic composition changes, reference documents get revised, an intake form adds a field, and a set assembled in March answers March's question. A workable arrangement is to keep the original cases frozen as a regression baseline and add a tenth of that number in freshly adjudicated cases each month — 150 a month on a set of 1,500 — retiring the oldest, so the set drifts with traffic while remaining comparable quarter to quarter. Adjudication is the expensive part — expert time, at a few minutes a case — so budgeting it as an ongoing line item rather than a one-off project cost is what keeps the measurement alive past the launch.

Reading the signals at one operating point#

Scored on 1,500 adjudicated cases from a clinical triage assistant, at a 6.5% review volume in every row, with the added latency reported at the median (p50):

The bottom row is the baseline that makes the rest legible. Reviewing a random 6.5% of cases catches 6.5% of errors, so the mean-logprob row at 11% is doing almost nothing — consistent with chapter 3, where the average over free text turned out to be mostly a length and phrasing measurement.

That equality is worth deriving rather than taking on trust, because it is the one row whose value is fixed before any measurement happens. Selecting cases for review without consulting a score gives every case the same chance of being picked. An error is a case like any other under that rule, since nothing about a case being wrong changes how likely it is to be drawn, so errors are drawn at the same rate as everything else. The share of errors reaching a reviewer is therefore the share of cases reaching one, whatever the model is and whatever the task, which is why the row reads the same figure twice.

Reading every other row against that line is what stops a number from flattering itself. Eleven of every hundred errors caught, at a review volume of 6.5%, is a factor of under two over an unsorted queue. The single-token logprob at 47% is a factor of about seven over the same baseline, and that ratio rather than the raw percentage is the quantity worth quoting to whoever is deciding what to build.

The two rows worth arguing about are the first and the fourth. Three-sample agreement catches four more percentage points of error than the free logprob signal, and costs 97% of the inference budget to do it. That is an entire budget spent on a four-point improvement, and it is a defensible purchase only if someone can say what four points of additional error detection is worth on this system. Putting that trade in front of a product owner as two numbers, rather than as an argument about which method is more principled, is most of the value of running the comparison.

Errors that are not independent#

The verifier row needs a second measurement that teams routinely skip. A second model reading the case notes and the draft answer, and judging whether the answer is right, is only useful to the extent its mistakes are different from the primary model's. If both were trained on similar data with similar objectives, they will tend to be wrong about the same cases, and the verifier's aggregate 39% conceals where those catches fall.

The step inside "trained on similar data with similar objectives" is the one that decides what a verifier is worth, and it is short enough to follow all the way down. Both models were fitted by being penalised whenever they gave low probability to the continuation that actually appeared in a training corpus, so both ended up as estimates of what text is typical. Two estimates of the same quantity, built from overlapping corpora, agree about what is typical in most places, and that agreement is what makes either of them useful at all. Where the typical continuation happens to be the wrong one, both models find it typical, and the verifier confirms the primary model's answer for exactly the reason the primary model produced it.

That is why calling a second model a second opinion is a borrowed phrase worth resisting. The authority in the phrase comes from clinical practice, where the independence is the point: a different person, with different training, looking at the patient rather than at the first clinician's note. A second model brings a second pass over the same evidence, processed by the same kind of estimate. It is a real check on the failures that are idiosyncratic to one generation — a slip, a rationale that contradicts its own conclusion, a mis-transcribed level — and no check at all on the failures that follow from what the training text made ordinary.

On this evaluation set the primary model made 62 errors on cases where the intake notes described chest discomfort in non-classical language — "pressure", "heaviness", "indigestion after climbing stairs" — and under-triaged them. The verifier flagged 4 of the 62. It caught plenty of other errors, mostly clerical ones where the rationale contradicted the assigned level, and it was nearly blind on the failure the clinicians care most about.

The mechanism behind those 62 sits in both models, which is why the second one did not catch them. Text describing discomfort as pressure, heaviness or indigestion after climbing stairs is, across the writing either model was fitted on, most often followed by something low-acuity, because most of the text in which those words appear is not about a cardiac event. The primary model assigns a low acuity because a low acuity is what typically follows that language. The verifier reads the same notes, forms the same estimate from the same statistics, and finds the low acuity unremarkable. Neither model is consulting a fact about what those words can indicate, and both are reporting what usually follows them.

So the aggregate figure and the useful figure came apart for a structural reason rather than a measurement one. The verifier caught the errors that came from one generation going wrong on its own, which are the errors a second attempt at anything tends to catch, and it missed the errors that came from a belief the two models share.

Measuring the overlap between what each signal catches, on the strata that matter rather than in aggregate, is the difference between "the verifier catches 39%" and "the verifier catches 39% of errors, almost none of them the dangerous kind."

The measurement itself is simple enough to specify in one sentence: for each pair of signals, compute the overlap between the error sets each one catches at the shared operating point. Two signals catching 47% and 39% could between them catch anything from 47% — if the second is a strict subset of the first — to 86% if they are disjoint. On this set the union was 55%, so the overlap is substantial and the verifier's independent contribution is eight points rather than thirty-nine. That single number is what justifies or kills the combined row in the table, and it cannot be inferred from the two individual figures.

Both of those bounds are worth deriving rather than accepting, because the space between them is the entire reason to run the measurement. Compare the two error sets rather than the two percentages. A union of two sets can never be smaller than the larger of them, so 47 of every hundred errors is the floor, and it is reached when every error the verifier finds is one the logprob signal had already found. A union can never be larger than the sum of the two, so 86 is the ceiling, and it is reached only when no error at all is caught twice. The tempting move is to add the two figures, and addition gives that ceiling, but addition is correct only in the case where the two signals share no errors — which is the one thing two models fitted on similar text do not deliver.

Landing at 55 puts this pair near the bottom of that range, and subtracting the logprob signal's 47 is what leaves the eight points above. The reason the range cannot be narrowed without the extra measurement is that a percentage counts how many errors a signal catches and says nothing about which ones, so two such percentages fix the bounds and never the answer between them.

The same check applies to agreement. Five samples from one model share a training distribution completely, so a systematic error is reproduced five times and scores 1.0, which chapter 5 flagged and which is worth re-measuring on the high-acuity stratum specifically rather than assuming.

Once the numbers are in, the general shape of the decision is small enough to draw:

Diagram

The limits, in writing#

The last deliverable of that meeting is a plain statement of the signal's limits, given to the product owner in writing, because the original ask will otherwise be remembered as a stronger claim than anything in the table supports.

The score is the model's probability over the five acuity levels at one token position, calibrated against 1,500 cases adjudicated last quarter. It is not a probability that the triage decision is safe, and it says nothing about whether the correct level was among the five in the first place. Nor can it flag a failure of omission: if the model never considers escalating, there is no competing candidate to split probability with, and the result is a confident label — chapter 4's mechanism, in a setting where it matters.

The gap between the ask and the deliverable is worth naming precisely, because "the model should tell us when it's unsure" is a reasonable sentence the machinery cannot satisfy. What the number reports is competition: how much of the model's probability was split between the options it was offered at that one position. A case where the mass genuinely spreads across acuity levels comes back low, and there the ask is met. A case where the correct level was never among the five, or where escalating was never a candidate, has nothing to split its mass between and comes back high — and the same high number that means "the options did not compete" is read as "the answer is safe". The signal is quietest in exactly the situation the question was asked about.

Self-agreement is not evidence that the shared answer is safe, and this is the one worth stating even though the budget ruled sampling out. Had the budget stretched to five samples, five samples agreeing would mean the model reliably produces that answer, and a model that reliably under-triages a particular presentation agrees with itself perfectly on every one of those cases. In a clinical setting the distinction between "reproducible" and "correct" is the whole distinction, and the number that would have gone on the dashboard measures the first one.

The reason those two come apart is the one already established for the verifier. Five samples are five draws from one distribution, and a systematic under-triage is a property of that distribution rather than of any single draw, so every draw carries it. Disagreement between samples detects the errors that one unlucky draw introduced, which is a real category and not the category that matters here. It cannot detect an error the model would make on every draw, and under-triage from non-classical language is exactly that kind of error.

Coverage, finally, is fixed by staffing rather than by the model. At 6.5% review volume, 53% of the errors reach a patient no matter which row of the table is chosen. The confidence signal changes which errors get caught; increasing the fraction caught means hiring reviewers, and that is a budget conversation rather than an engineering one.

That figure is read off the table rather than estimated. The operating point fixes how many cases a human sees, the best signal available at that operating point catches a bit under half the errors, and every error outside that half is one that reaches a patient. Among the rows the budget left standing the number barely moves, because those rows sit a handful of points apart while the distance to complete coverage is the size of the whole review budget. The lever is the number of reviewers, and the reason to put this in writing is that a signal described as catching 47% of errors is easily heard as leaving very little uncaught.

Comparing confidence methods is therefore a measurement exercise with three inputs: one labelled set drawn from real traffic and stratified toward the cases that matter, one operating point fixed by review capacity, and one baseline of random selection at the same volume. Everything else — cost multipliers, latency, correlated errors, the independent contribution of a second signal — reads off that comparison as numbers a product owner can trade against each other. The method that wins is frequently the cheapest one, and the honest report frequently includes an unspent budget and a rule layer that outperforms every probabilistic row on the stratum nobody can afford to get wrong.

The whole topic reduces to an operating procedure that fits on a page. Classify each decision as a label decision or a free-form one, and convert the free-form ones where a cheap second classification call can do it. Read the logprob where the output shape earns it, and use agreement only where it does not and where the budget genuinely covers N inferences plus the latency of the slowest. Define agreement over the field the downstream system consumes, with sampling verifiably switched on. Derive every threshold from a labelled set drawn from production traffic, record it with the model version and the date, and re-derive it on migration. Put the frozen canary set in CI so a silent revision surfaces as a failing check rather than as a quiet drift in review volume. And write down, next to the threshold, what the number does not mean.

Two resources are worth the time when you build the first version. The OpenAI cookbook's Using logprobs notebook is runnable and covers the request flags, top_logprobs, and threshold construction end to end. For making the signal legible to people who are not going to read an array, Visualizing logprobs from OpenAI responses shows how to render per-token confidence over the generated text, which turns "the model was unsure somewhere in this paragraph" into a highlighted token a reviewer can look at — and which, more than any dashboard number, is what makes the limits in this chapter obvious to the people who need to understand them.