Confidence thresholds and calibration

A threshold is the point where a confidence score stops being a number and starts being a routing decision: below it a case goes to a human, above it the system acts on the answer. Everything in this topic so far produces scores — the renormalised single-token probability from chapter 2, the agreement fraction from chapter 5 — and none of them is usable until a cutoff is chosen. The cutoff is where most of the remaining error lives, because it looks like a tuning parameter and behaves like a measurement.

Two properties have to be separated to see why. Ranking is whether the cases the score puts at the bottom are in fact the cases the model gets wrong most often. Calibration is whether the number means what it says: among the cases scored 0.80, whether 80% are correct. A model can rank perfectly while being uniformly overconfident, or be beautifully calibrated and rank poorly. A threshold depends on ranking; a number shown to a human depends on calibration; and a score carried across a model migration depends on both staying put, which they do not.

The two properties are independent, and the quickest way to believe that is to build a score that has one and not the other. Take a router whose score is the true chance of being right with a fixed amount added to every case, so that everything it would honestly have called six in ten it reports as nine in ten. Adding the same amount everywhere moves nothing past anything else, so the cases it puts at the bottom are still the cases it gets wrong most often. Its ranking is perfect, its number is a lie in every band, and a cutoff on it still routes correctly. Reverse it and take a score that is honest in aggregate — across everything it labels seven in ten, seven in ten really are right — but which gives that same value to correct and incorrect cases alike. Sorting by that one produces an arbitrary order, so no cutoff picks out the cases worth reviewing any better than drawing them at random.

Which of the two a system needs is settled by what consumes the score. A threshold asks one question of each case — which side of the line it falls on — so it consumes order and nothing else, and any rescaling that preserves order moves the threshold without changing which cases end up on which side of it. A number displayed to a person is consumed as a frequency, because "94% confident" is read as a rate of being right, so there the value itself has to be honest.

Two models, two distributions#

Run two models over the same 2,000 labelled tickets from chapter 2 and the confidence distributions barely resemble each other:

Model B is the better classifier by a point and a bit, and its 0.95 is worth much less than model A's. Where A puts 61% of its mass above 0.95 and is right 98.7% of the time there, B puts 88% of its mass in the same band and is right 94.2% of the time. Read the last row: to get the same 95% precision on the auto-routed stream you need a cutoff of 0.83 on A and 0.991 on B. Those are not adjacent numbers, and no amount of squinting at model B's 0.95 will tell you it means what model A's 0.83 meant.

The two accuracy figures in that top band are easier to hold as error rates. Being right 98.7% of the time inside a band means being wrong about one time in eighty inside it, and being right 94.2% of the time means being wrong about one time in seventeen. A case carrying a score of 0.95 therefore fails roughly one time in eighty under model A and roughly one time in seventeen under model B — the same printed number, on the same scale, attached to error rates more than four times apart. A threshold is a promise about the cases sitting above it, and that is the promise that changes when the model does.

Comparing the two models' confidence numbers directly is therefore not a comparison of reliability. It is a comparison of how peaked their output distributions happen to be, which is a consequence of vocabulary, of training data, of how much preference optimisation was applied after pretraining — the later training stage that teaches a base model to follow instructions and to prefer the kind of answer people rated well, which reliably sharpens distributions, which is why instruction-tuned models are usually more overconfident than their base models — and of whatever the serving stack does between the logits and the response body.

The belief that gets in the way here is that a more confident model is a better one. It is tempting for a sound reason: inside a single model on a single prompt, confidence and accuracy do move together, which is the entire content of chapter 2's table and the reason a threshold works at all. What fails is carrying that relationship across the boundary between two models, because how peaked a model's output distribution is depends on how it was trained and how it is served, and neither of those is a fact about how often it is right. Model B is more confident than model A everywhere and is right less often inside the band the two share.

Separating ranking from calibration is what makes model B's behaviour readable. Its ranking is fine: within its own compressed range, the cases it scores lowest are still the cases it gets wrong most often, which is why a threshold at 0.991 recovers a working router. What collapsed is the spread. Eighty-eight percent of its mass piled into a band a hundredth of a unit wide, so a threshold placed anywhere in the region where model A's thresholds lived now selects almost nothing. The signal did not disappear; it moved into the fourth decimal place, where floating-point noise and the provider's rounding start to matter and where a threshold is uncomfortable to defend. That is worth knowing before a migration, because "the new model needs a threshold of 0.991" is a sentence that should prompt a look at whether the score has enough usable resolution left to route with at all.

The reason ranking survives that compression and selection does not is worth separating out. Ranking is a statement about order, and squeezing a set of numbers into a narrow band leaves their order exactly as it was. Selection is a statement about value: a cutoff takes everything on one side of a line, so a line drawn outside the band takes all of the cases or none of them. The information is still in model B's scores, and the only place left to reach it from is inside the band.

What that costs is resolution rather than signal, and resolution can be measured. A cutoff three decimal places in separates cases by differences of the same order as the rounding a provider applies to the value it returns. The check worth running before a migration is how many distinct scores the labelled set produces in the region the cutoff has to sit in: collapsed onto a handful of values, the model has stopped being routable at any cutoff, and spread across the region, it is merely uncomfortable to write down.

Calibration, and the number that measures it#

Chapter 2's table showed model A is close to calibrated but not there — the 0.80–0.95 band averages 0.885 confidence and 92.1% accuracy, so the score understates by three and a half points in that region. Expected calibration error summarises that gap as a single weighted average, and it is worth computing because it is the number that tells you whether a threshold can be read as a probability or only as a rank:

def ece(rows):
    """rows: (share_of_volume, mean_confidence, measured_accuracy) per bin."""
    return sum(w * abs(conf - acc) for w, conf, acc in rows)

model_a = [(0.610, 0.988, 0.987),
           (0.190, 0.885, 0.921),
           (0.130, 0.655, 0.738),
           (0.070, 0.375, 0.407)]

print(f"{ece(model_a):.3f}")   # 0.020

Each row of model_a is one confidence band from chapter 2's table, written as three numbers: that band's share of all cases, the mean confidence the score reported inside it, and the accuracy a human adjudicator measured there. The function treats every row the same way. It takes the distance between what the score claimed and what the measurement found — an absolute distance, so that overstating and understating both count as error instead of cancelling — multiplies it by the band's share of volume, and adds the four results together.

The weighting is what makes the total an operational number rather than an average of four unrelated gaps. The top band holds 61% of the traffic and claims almost exactly what it delivers, so it contributes nearly nothing. The band from 0.50 to 0.80 claims 0.655 and measures 0.738, a gap of more than eight points, but it holds about a tenth of the traffic, so only a small share of that gap reaches the total. Weighting by volume stops a badly behaved band that almost never fires from dominating the summary of a score that mostly fires elsewhere.

Model A comes in at 0.020, meaning its score is on average two points away from the truth — good enough that "0.80 confidence" is a roughly honest statement. Model B on the same bins comes in at 0.11, so its 0.95 is eleven points optimistic on average and reading it as a probability would be a mistake in every decision that depends on it.

The concept is older than language models and arrived in modern deep learning through Guo and colleagues' 2017 paper On Calibration of Modern Neural Networks, which documented an uncomfortable pattern: as image classifiers got deeper and more accurate through the mid-2010s, they got steadily worse calibrated, with confidence outrunning accuracy by wider margins. The fix in that paper is temperature scaling — divide the logits by a single scalar fitted on a held-out labelled set, which leaves the ranking untouched and pulls the confidences back toward the accuracies. The same trick works on a single-token classifier's five-way distribution, with one scalar per model version. Note the name collision: this scalar has nothing to do with the sampling temperature from topic 1.6, is fitted rather than chosen, and is applied after generation rather than during it.

Why that particular repair fits this particular problem follows from what dividing by a positive constant does and does not do. Dividing every logit — the raw score a model produces for each candidate before those scores are normalised into probabilities — by the same positive number changes the gaps between them, and the gaps are what decide how peaked the resulting shares are. It cannot change which logit is largest, or which is second, because dividing a set of numbers by one positive constant leaves their order untouched. Ranking held fixed and values moved is exactly the pair of properties the repair has to have, on a model whose ranking was fine and whose values were not.

The scalar is fitted rather than chosen because there is nothing in the model to read it off. What it corrects is the distance between the confidences one model reports and the frequencies one labelled set measured, which is an empirical fact about that model, that prompt and that traffic rather than a property of the method. Fitting it on held-out cases keeps the value from memorising the sample it came from.

Whether it is worth fitting depends on who consumes the number. If the only consumer is a threshold in your routing code, calibration buys nothing, because the threshold is derived from measurement anyway and a monotonic rescaling moves the threshold without changing which cases fall on which side. If the number is shown to a person — an agent seeing "94% confident" next to a suggested action, a reviewer prioritising a queue by score, a product owner reading a dashboard — then calibration is the difference between a number that can be reasoned about and a number that systematically misleads. Model B's uncalibrated 0.95 shown to a clinician means "about nineteen times in twenty" to the clinician and "about eighteen times in twenty at best" in reality, and that gap compounds across thousands of decisions. The rule of thumb is to fit the scalar whenever a human reads the score, and skip it whenever only code does.

Deriving a threshold you can defend#

Thresholds get treated as tuning parameters, and it is obvious why: a threshold is a number in a config file with an unambiguous direction, and moving it visibly changes review volume the next day. What it actually is, is the output of a measurement, and it carries every condition of that measurement with it — the model version the labelled set was scored under, the prompt version in force at the time, and the mix of traffic the set was drawn from. A measurement can only be moved by measuring again, and treating it as a dial is how a cutoff outlives what it was derived from.

Calibration is optional; deriving the threshold from measurement is not. The procedure has five parts and each one is somewhere a team usually cuts a corner.

Label a set drawn from production traffic. Synthetic or curated evaluation sets have the wrong difficulty distribution and the wrong length distribution — chapter 3's abstention rule failed for exactly that reason — and a threshold derived on them will not survive contact with real inputs. Stratify so that rare-but-important cases are represented at a rate you can measure rather than at their natural 0.4%.

Sweep the threshold and read off the operating point you actually want, the operating point being the particular combination of review volume and auto-stream precision you intend to run the system at:

def sweep(scored, thresholds):
    """scored: list of (confidence, is_correct)."""
    n = len(scored)
    print(f"{'thr':>6} {'review %':>9} {'auto prec':>10} {'errors caught':>14}")
    total_errors = sum(1 for _, ok in scored if not ok)
    for t in thresholds:
        review = [(c, ok) for c, ok in scored if c < t]
        auto   = [(c, ok) for c, ok in scored if c >= t]
        caught = sum(1 for _, ok in review if not ok)
        prec   = sum(1 for _, ok in auto if ok) / max(len(auto), 1)
        print(f"{t:6.2f} {100*len(review)/n:8.1f}% {100*prec:9.1f}% "
              f"{100*caught/total_errors:13.1f}%")

sweep(scored_model_a, [0.50, 0.60, 0.70, 0.80, 0.83, 0.90, 0.95])

Pick the threshold against a constraint someone owns — a review-capacity number from the staffing plan, or a precision target from a risk register — rather than against a round number that looks reassuring. The output of the sweep is a menu; choosing from it is a business decision and should be recorded as one.

Size the labelled set for the estimate you need, not for the total volume. To pin down a 95% precision figure to within two percentage points at 95% confidence takes about 1.96² × 0.95 × 0.05 / 0.02² ≈ 456 labelled cases above the threshold. That is the easy part. Placing the threshold well also requires enough cases in the sparse low-confidence region, and if only 7% of traffic falls below 0.50 then 456 cases down there means labelling roughly 6,500 overall. Teams routinely label 200, land a threshold with a confidence interval eight points wide, and treat the point estimate as a fact.

Each part of that expression is doing a job, and the shape of it is why labelling budgets come out wrong by a factor rather than by a margin. The two middle terms are the proportion being estimated and what is left of it, and their product is why a precision figure near the top of the scale is cheap to pin down: a proportion close to one varies less from sample to sample than one near a half. The divisor is the tolerance squared, and the squaring is the term that hurts, because halving the width of the interval you will accept costs four times the cases and tightening it tenfold costs a hundred times.

Record the threshold with its provenance: the model identifier, the prompt version, the date, the size of the labelled set, and the measured precision and review volume at that cutoff. A threshold in a config file with no attached measurement is indistinguishable from a guess six months later, and it will be treated as load-bearing by whoever inherits it.

What shifts underneath a threshold#

Topic 1.6 made the point that a provider can revise a model behind a stable identifier, and that the same seeded request can return different logits on different execution paths. A threshold is exactly the kind of artefact that quietly breaks when that happens, because nothing errors — the numbers keep coming back, they just mean something slightly different, and review volume drifts.

Versions moving underneath a threshold produce one particularly convincing wrong conclusion. A fall in review volume reads as an improvement, and it reads that way for a sound reason: a more accurate model is right more often, and a model that is right more often should need less review. A compressed confidence distribution produces the identical observable, and nothing on a dashboard separates the two, because the difference between them lives in the accuracy of the cases that stopped being reviewed. Re-scoring the frozen labelled set is what tells them apart.

The cheap defence is a canary. Freeze 500 labelled cases, re-score them on a schedule, and store the confidence histogram alongside the accuracy. Two alerts are worth having: one on the share of the canary set that lands above the production threshold, firing on a move of more than three percentage points week over week, and one on the two-sample Kolmogorov–Smirnov statistic between this week's confidence distribution and the frozen baseline, which is the largest gap between the two distributions' cumulative curves and therefore registers a change in the shape of the distribution rather than only in its average. The first catches the Wednesday migration within a day rather than within three weeks. The second catches the subtler case where the mass shifts without crossing the threshold, which is the early warning that the next revision will cross it.

The two alerts are watching different moments of the same failure. A share-above-threshold alert can only fire once mass has crossed the line, so what it fires on is a change that has already altered routing: cases that used to reach a reviewer are being acted on automatically. The distribution statistic fires on movement that has crossed nothing yet, because mass sliding upward inside the band below the cutoff moves the cumulative curve without changing how many cases sit above the line. One is a detection and the other is a warning. Storing a histogram rather than a mean is what makes the second of those possible, because an average holds steady while mass moves out of the middle into both tails, and holds steady again while a distribution compresses around it — the exact shape of model B's problem.

Pinning to a dated model snapshot where the provider offers one removes most of the surprise, at the cost of eventually being forced onto a new snapshot on the provider's schedule rather than yours. Either way the threshold is a per-version artefact, and a model migration is not complete until it has been re-derived on the new version.

Traffic shifts do the same damage from the other direction, and they are less visible. The measurement is valid for inputs that resemble the labelled set, so threshold-based routing quietly stops working when a new product launches, a new customer segment arrives, or an outage changes what people write in. Those are the days when the review queue should be filling up and the days when the score is least trustworthy. The canary catches the shift after the fact; nothing catches it within the request that suffers from it.

Everything in this chapter makes a number mean what it says on the distribution you measured, which is a narrower guarantee than it sounds. It does not repair anything from chapter 4: a well-calibrated score on a five-way classifier is a well-calibrated probability over those five options, and a ticket that belongs to a sixth category still gets confidently misrouted. Calibration is a statement about aggregate frequencies on a sample, not about any individual case, and a threshold is a statement about one model version on one prompt at one point in time.

What all of this gives you is a per-signal, per-model number with a measurement attached. That is the unit that makes different confidence methods comparable to each other, which is the last thing to settle: given a fixed evaluation set and a fixed budget, which signal do you actually buy.