Single-token classifier confidence

A label decision is one where the model's whole visible output is a single choice from a fixed, small set — a queue name, a severity, a yes or a no — and the code downstream branches on that choice. Chapter 1 showed that the logprob array of an ordinary completion is mostly grammar: articles, punctuation and inflectional endings sitting within a hair of certainty, with a handful of content positions where the distribution genuinely spreads. A label decision has no grammar in it. One position is generated, that position is the answer, and the distribution over that position is the distribution over the answers.

The claim in that last sentence is the foundation of everything below it, and it is worth slowing down on, because it is false of almost every other output a model produces. At every position the model emits one share for each entry in its vocabulary, and those shares total one. When the answer is a sentence, the shares at the first position are shares over ways of starting a sentence, and the answer itself is spread across all the positions that follow, so no single distribution in the array is a distribution over answers. When the answer is one token, and the tokens the prompt admits are exactly the answers on offer, the two objects coincide: the share sitting on a token is the share sitting on the answer that token stands for. This is the only place in the topic where they coincide by construction rather than by approximation.

That makes this the narrowest and best-founded use of a logprob in the topic, and it is worth building once end to end, because two details decide whether the number means anything: whether each label really is a single token, and what the probability is renormalised over. Both are cheap to get right and silent when they are wrong. With those settled, the score can be read against a labelled sample, which is the only thing that establishes what it orders.

Making the answer one token#

The design only works if each label really is a single token, and the labels a product manager writes down usually are not. Before anything else, check:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")
LABELS = ["billing", "technical", "account", "sales", "other"]

for label in LABELS:
    ids = enc.encode(label)
    print(f"{label:<10} {len(ids)} token(s)  {[enc.decode([i]) for i in ids]}")

billing    1 token(s)  ['billing']
technical  1 token(s)  ['technical']
account    1 token(s)  ['account']
sales      1 token(s)  ['sales']
other      1 token(s)  ['other']

Five for five here, which is luck rather than design — those are all common English words. Swap in a real taxonomy and it falls apart immediately: refund_request is four tokens, Billing with a leading capital is two, and any label with an underscore or a hyphen shatters the way topic 1.1 showed. The robust move is to stop routing on the label text and route on a single-character key instead, mapping A through E to the queues in code. Single uppercase letters are one token in every vocabulary you are likely to meet, the mapping lives in your repository where it can be reviewed, and renaming a queue no longer changes the model's output space.

What a multi-token label costs is worth being precise about, because the failure is not that the method becomes noisier. It stops being the method. If refund_request occupies four positions, the first of those carries a distribution over opening fragments rather than over labels, and two labels beginning with the same fragment are not separated by it at all. Reading that first probability answers a question about spelling. Reading the product of all four answers a question about the whole string, and a product of four numbers below one is smaller than a product of two, so a longer label scores lower for being longer and the comparison between labels is contaminated by how each of them happened to be tokenised. A single-character key removes both problems together, because one position holds the entire answer and every answer costs the same number of positions.

With that settled, the request is short. Cap the output at one token so nothing else can be generated, ask for the top five alternatives so you can see what the model was weighing, and pin the temperature at zero so the returned label is the argmax — the single highest-scoring candidate — rather than a draw:

import math

PROMPT = """Classify the support ticket into exactly one category.
A = billing   B = technical   C = account   D = sales   E = other
Answer with a single letter and nothing else.

Ticket:
{ticket}"""

KEYS = {"A": "billing", "B": "technical", "C": "account", "D": "sales", "E": "other"}

def classify(ticket: str) -> tuple[str, float]:
    resp = client.chat.completions.create(
        model=MODEL,
        temperature=0,
        max_tokens=1,
        logprobs=True,
        top_logprobs=5,
        messages=[{"role": "user", "content": PROMPT.format(ticket=ticket)}],
    )
    top = resp.choices[0].logprobs.content[0].top_logprobs

    # Keep only candidates that are valid keys, then renormalise over them.
    mass = {}
    for alt in top:
        key = alt.token.strip().upper()
        if key in KEYS:
            mass[key] = mass.get(key, 0.0) + math.exp(alt.logprob)

    total = sum(mass.values())
    key = max(mass, key=mass.get)
    return KEYS[key], mass[key] / total

Two of the arguments in that request are doing more work than they look. Capping the output at one token is what makes the position under examination the only position, so there is nothing after it that could revise the answer. Pinning the temperature at zero matters for a subtler reason: the function returns two things that have to describe the same candidate — the label the router acts on and the number the threshold compares — and it takes both from the highest-scoring key it can see. With sampling switched on, the token the endpoint actually emitted could be some other key, and the system would then route on one candidate while reporting a number about another.

The loop and the three lines beneath it do three things in sequence, and each is worth naming. They discard every returned candidate that is not one of the five keys, which leaves a set of shares that no longer totals one, because dropping members of a set that summed to one leaves a set summing to less. They add the survivors together to get total. Then they divide the winning key's share by total. Dividing a group of numbers by their own sum always produces a group that adds to one, since the sum of the quotients is the original sum divided by itself, and having the shares total one is the property that makes them readable as probabilities at all.

The renormalisation in those last lines matters more than it looks. The raw probability of the chosen token is diluted by mass sitting on things that are not answers at all — a leading space, a newline, a lowercase variant, the opening of a sentence like Based. On a well-behaved prompt that leakage is a percent or two, but on an empty or garbled ticket it can be a third of the distribution, and a score that quietly counts "the model was about to start explaining itself" as uncertainty about billing-versus-technical is measuring the wrong thing. Dividing by the mass that landed on the five valid keys gives a probability conditioned on the model answering the question at all, which is the quantity you actually want to threshold.

Working that garbled-ticket case through makes the difference concrete. Suppose two thirds of the mass lands on the five keys and the remaining third sits on a leading space, a newline and the opening of an explanatory sentence. If the winning key holds half of the total mass, the raw share reports one half and the renormalised share reports one half divided by two thirds, which is three quarters. A threshold reads those two numbers very differently, and only one of them is about the ticket: the raw one has folded in the model's hesitation about whether to answer with a letter at all, which is a fact about the prompt and the input's shape rather than about whether the ticket is billing or technical.

Naming what the renormalised number is a probability of keeps that distinction from slipping back. It is the chance the model picks that key given that it picks one of the keys — a conditional, in the ordinary sense that it has been computed inside a restricted set of outcomes rather than across everything the model might have said. That is the right quantity for a router, which has already decided that anything which is not a key is a malformed response handled on a different path, and which wants a number about the classification rather than about the formatting.

Five labels is a comfortable size for this. Taxonomies with thirty categories are not, and the reason is the top_logprobs ceiling. Providers return the top few candidates per position — up to twenty on OpenAI-style endpoints — so with thirty labels you see at most twenty of them and the mass sitting on the other ten is invisible. Renormalising over what came back then divides by a total that is systematically too small, inflating every score by an unknown amount that varies case by case. If your taxonomy is larger than the alternatives ceiling, either collapse it into a two-stage decision — a coarse classifier over six groups, then a second call within the chosen group — or accept that you are reading the top candidate's raw probability rather than a renormalised one, and that the leakage is unmeasured.

The direction of that error is worth deriving rather than memorising, because it decides whether the top of the range can be trusted and the top of the range is where the routing decisions are. With more labels than the ceiling, some keys are never returned at all. Whatever mass sat on those keys is therefore missing from the total the code divides by. Dividing by a total that is too small makes every quotient too large, so the scores come out inflated rather than deflated, and the size of the inflation is however much mass happened to be hidden on that particular case. A hard case spreads its mass over more keys than an easy one, so more of it falls outside the returned list precisely where the score most needed to be low, and the bias is not a constant anyone can measure once and subtract.

That is the argument for splitting the decision in two, rather than a preference for smaller prompts. A coarse classifier over a handful of groups, all of them inside the alternatives ceiling, gets back every candidate it could have chosen, so the total it renormalises over is complete and the quotient means what it says. The second call, made within the chosen group, is a small closed set for the same reason and inherits the same guarantee. What the split costs is two calls per ticket and one weakness that no amount of confidence in the second stage repairs: a ticket the coarse stage sends to the wrong group is then scored against labels that do not contain its answer, and it will be scored confidently.

What the score orders, measured on adjudicated tickets#

A score is only useful if it orders cases the way you hope, and the only way to find out is to label some. Two thousand tickets from a support router, sampled across a full week so the Monday billing surge and the Friday afternoon noise are both represented, scored by the function above and adjudicated by human reviewers:

The column that decides whether to ship this is accuracy, and it falls monotonically as confidence falls, from 98.7% at the top to 40.7% at the bottom. That is the whole claim: the score sorts the cases. It is not a promise that 0.80 means eight times in ten — the 0.80–0.95 bucket is right 92% of the time, so the score is miscalibrated in the middle, meaning the number does not match the frequency it appears to promise, and pessimistic at the bottom — and chapter 7 deals with what to do about that. For routing, sorting is enough.

The difference between sorting and promising decides what a score may be used for, and the cleanest way to hold it is by asking what consumes the number. A threshold consumes an ordering. It asks which side of a line a case falls on, and it is indifferent to what the values are called as long as the cases below the line are worse than the cases above it. A number displayed to a person consumes the value instead, because anyone reading a figure on a probability scale is entitled to treat it as a frequency. The table supports the first reading and visibly fails the second: the band from 0.80 to 0.95 is right 92.1% of the time, which is better than the numbers inside it suggest, and the band below 0.50 is right 40.7% of the time, which is also better than its numbers suggest. Both discrepancies point the same way and neither is the same size, which is a pattern a threshold absorbs without noticing and a displayed number cannot.

Two figures in the table deserve to be checked rather than accepted, because both are estimates from a sample. The bottom bucket contains 140 tickets, so its 40.7% accuracy carries a 95% interval of roughly plus or minus eight percentage points — wide enough that the true value could be anywhere from 33% to 49%, though not wide enough to threaten the ordering. The top bucket's 98.7% rests on 1,220 tickets and is tight to within a point. When someone asks whether the difference between the two middle buckets is real, the honest answer is that with 380 and 260 tickets respectively the intervals overlap slightly, and separating them properly would need a few hundred more labels. Reporting the interval alongside the point estimate costs one line and prevents a lot of arguing about noise.

Where an eight-point interval comes from is worth carrying, because the same shape of calculation applies to every bucket anyone ever builds. The uncertainty in an estimated proportion narrows with the square root of the number of cases behind it, so it takes four times the sample to halve the interval. The top bucket rests on 1,220 tickets and the bottom on 140, a ratio of roughly nine to one in the counts and therefore about three to one in the widths — one tight to within a point, the other spanning some sixteen points end to end. The operational reading is the one to keep. The gaps between the buckets' accuracies are much wider than the intervals, so the ordering survives; the individual values do not survive at all, and quoting one of them as a measured accuracy is a claim the sample cannot support.

The score is a probability over the options you offered#

The number is conditioned on the five keys and on nothing else. If a ticket belongs to a sixth category that the taxonomy does not contain — a security-disclosure report, say — the model still spreads its mass across A through E, and it can spread it confidently. In the labelled sample above, seven tickets reporting a suspected data leak were classified as technical at 0.97, and every one of them should have gone to a queue that does not exist. A high score is a statement about the model's choice among the alternatives you offered, and it cannot signal that the alternatives were wrong.

The belief that a high score is a statement about the ticket is tempting for a respectable reason: on the traffic the taxonomy was built for it is very nearly true, and the table is the evidence. Cases above 0.95 are right 98.7% of the time. What the score has no way to do is notice that the question was wrong. It is computed from a distribution over five keys, and a distribution over five keys cannot express none of these unless one of the keys says so — the mass of a data-leak report had nowhere to go but A through E, and it concentrated on technical because a security report reads more like a technical ticket than like a billing one. Confidence that a ticket is technical rather than billing is not confidence that the ticket is one of the five things on the list, and no amount of calibration converts the first claim into the second.

The score also moves with the prompt in ways that have nothing to do with the ticket. Reordering the five options changes the numbers, because position in the list carries weight independent of content; in this sample, moving other from last to first raised its share of predictions from 6% to 11% without a single ticket changing. Rewording a category description shifts the boundary between two queues, and every threshold derived from an older prompt goes stale the moment the prompt is edited. Version the prompt alongside the threshold, and treat a prompt change as invalidating the measurement.

That sensitivity is not a defect in the implementation, and reading it as one sends people looking for a fix that does not exist. The model is scoring continuations of a piece of text, the options are part of that text, and their order is part of how they appear in it, so the prompt with other listed first and the prompt with other listed last are two different inputs. A quantity computed from an input has no obligation to be invariant to a change in that input, and nothing in the way the model was trained ever asked for such an invariance. The versioning rule above is a consequence of that rather than a precaution: the threshold was measured against one input, and editing the prompt produces a different one.

Asking the model to reason before answering does not break the method, which surprises people. If you let it write two sentences of analysis and then emit the label, the label is still one token and its logprob is still readable — it is now conditioned on the reasoning, which is a different and often better-separated quantity. What breaks is the cost model, since you are no longer capping output at one token, and the logprobs of the reasoning tokens themselves are irrelevant to the decision. Read the last token, ignore the rest.

Compared with the self-report from chapter 1, where the model is asked to rate its own confidence and produces text, this costs nothing. The flag adds no tokens to the request and no latency to the response; the entire mechanism is reading a field that the provider was going to compute anyway. That asymmetry — a free signal that ranks cases against a paid signal that does not — is the strongest argument for reaching for logprobs first whenever the output shape permits it, and the peer-reviewed work collected in Logprobs Know Uncertainty: Fighting LLM Hallucinations is the place to go for how far the signal extends beyond single-label outputs.

One decision, one token, one number is the whole of the case for this method, and each of the three has to be true for the score to mean anything. The decision has to be a choice among a closed set that the code downstream branches on; the token has to be verified against the tokeniser rather than assumed; the number has to be renormalised over the valid keys and then checked against adjudicated data, which is what turns it from a plausible field in a response body into a statement about how often the model is right in each band. Everything after that is a staffing question, and the table is what makes it answerable.

The catch is that "whenever the output shape permits it" is a real restriction. The moment the answer is a sentence rather than a label, there is no single decisive token, and the obvious repair is to average the array from chapter 1 across the whole response. That average is the most common confidence metric in production and it measures something other than what its users think.