Token logprobs

There are two numbers a system can attach to a model's output as a measure of doubt. One is generated: ask the model to rate its own confidence and it produces text, drawn from a distribution over the vocabulary, conditioned on everything before it, with nothing in that process consulting the answer it is commenting on. The other is measured during generation: at every position the model produced, it also produced a score for the token it emitted relative to the rest of the vocabulary, and most providers will hand those scores back if you set a flag.

This chapter is about the second one. A logprob is the natural log of the probability assigned to the token that was actually emitted; providers return one per generated token, in an array running alongside the text, and most will also return the top logprobs, meaning the handful of alternative tokens that were competing at the same position, each with a value of its own. Every entry comes back as zero or a negative number, and entries close to zero mark the tokens the model most heavily favoured. The value costs nothing to obtain, because it was computed as part of emitting the token whether or not anyone asked to see it. What the chapter builds is the ability to read one of these arrays: how to request it, why it is on a log scale, what it costs to keep, and what its distribution over a real completion looks like. Downstream, the same numbers turn up wherever a system treats some outputs as more doubtful than others — ranking a review queue so a human sees the shakiest cases first, an abstention rule that withholds an answer below some cutoff, or an interface that shades each word by its probability. Each of those is defensible in some circumstances and a quiet source of error in others, and sorting out which is which is the work of this topic.

The flag is one field on the request#

The object these numbers come from is the next-token distribution. At each position the model emits one score per vocabulary entry, a softmax normalises those scores into a distribution summing to one, and the sampling settings reshape that distribution before a token is drawn — the material of the previous topic in this book. What this topic adds is that the value assigned to the token actually drawn is retrievable after the fact, along with the values of the next few runners-up.

On an OpenAI-style chat completions endpoint, that is logprobs: true, plus top_logprobs: n if you also want the alternatives at each position:

from openai import OpenAI

MODEL = "gpt-4o-mini"
client = OpenAI()

resp = client.chat.completions.create(
    model=MODEL,
    temperature=0,
    logprobs=True,
    top_logprobs=5,
    messages=[
        {"role": "system", "content": "Answer in one short sentence."},
        {"role": "user", "content": "How long is the standard refund window?"},
    ],
)

for item in resp.choices[0].logprobs.content:
    alts = ", ".join(f"{a.token!r}:{a.logprob:.2f}" for a in item.top_logprobs[1:4])
    print(f"{item.token!r:>10} {item.logprob:7.4f}   alts: {alts}")

The response body carries a parallel array to the generated text, one entry per token:

{
  "choices": [{
    "message": {"content": "The standard refund window is 30 days."},
    "logprobs": {
      "content": [
        {"token": "The",      "logprob": -0.0912,
         "top_logprobs": [{"token": "The", "logprob": -0.0912},
                          {"token": "Our", "logprob": -2.6134},
                          {"token": "Ref", "logprob": -4.8021}]},
        {"token": " standard","logprob": -0.4471, "top_logprobs": [...]},
        {"token": " refund",  "logprob": -0.0021, "top_logprobs": [...]},
        {"token": " window",  "logprob": -0.0009, "top_logprobs": [...]},
        {"token": " is",      "logprob": -0.0002, "top_logprobs": [...]},
        {"token": " ",        "logprob": -0.0104, "top_logprobs": [...]},
        {"token": "30",       "logprob": -0.4103,
         "top_logprobs": [{"token": "30",  "logprob": -0.4103},
                          {"token": "14",  "logprob": -1.5990},
                          {"token": "60",  "logprob": -2.9077},
                          {"token": "90",  "logprob": -4.1210}]},
        {"token": " days",    "logprob": -0.0006, "top_logprobs": [...]},
        {"token": ".",        "logprob": -0.0031, "top_logprobs": [...]}
      ]
    }
  }]
}

Support for this is uneven and worth checking before you design around it. OpenAI-style chat completions expose logprobs and top_logprobs with a documented ceiling of 20 alternatives per position; Google's Gemini API has responseLogprobs and logprobs in the generation config; open-weight servers such as vLLM and Text Generation Inference expose them for any model you host yourself, which is the only way to get at the full vocabulary vector rather than a truncated top-k. Anthropic's Messages API does not return them at all. Some providers also drop the array on particular paths — structured-output modes and tool-call arguments are the usual gaps — so confirm on the exact endpoint your production code uses rather than on the plain chat endpoint you tested with.

Why the numbers come back as logarithms#

Every value in that array is negative or zero, because it is the natural log of a probability and probabilities are at most one. Converting back is exp: a logprob of 0.0 is certainty, -0.69 is a probability of 0.5, -2.30 is 0.1, and -4.61 is 0.01.

That sign is worth deriving rather than accepting, because every rule built later in this topic reads it as a direction. The natural logarithm of one is zero, and the logarithm is an increasing function, so any input smaller than one comes back as an output smaller than zero. A probability cannot exceed one, which is what puts a ceiling of zero on the whole array, and the only way to reach that ceiling exactly is for the model to have given the token the entire available mass. The size of the number then reads straight off: a value a hair below zero is a token the model was all but certain of, and a value far below zero is one it thought unlikely and emitted anyway. In this array "close to zero" and "confident" are the same statement, which is the reverse of the usual habit of reading a bigger number as more of something.

One row is worth working in full, because the rest of the table is then reconstructible rather than something to look up. The natural logarithm of two is about 0.69, so -0.69 is the logarithm of one half, and taking the exponential of it hands back a probability of 0.500 — the row the table calls a coin flip. The same fact gives the table its spacing. The natural logarithm of ten is about 2.30, so a drop of that much anywhere on the log scale divides the probability by ten, and the table shows two of those drops in a row: from 0.0 down to -2.30 takes the probability from certainty to 0.100, and from -2.30 down to -4.61 takes it from 0.100 to 0.010. Two steps of equal size, two divisions by ten, which is what makes this a scale of ratios rather than a scale of amounts.

The log scale is not a formatting preference. A sequence's probability is the product of its per-token probabilities, and a 400-token answer whose tokens each sit around 0.9 has a joint probability near 10^-19, which underflows a float32 and becomes indistinguishable from zero. Sums of logs stay in a comfortable range, so the sequence score is the sum of the array — a number like -38.4 rather than a denormal. Everything downstream in this topic works in log space for that reason.

Each link in that argument is checkable, and the chain is worth walking once, because the log scale is the convention a reader has to accept before any of the rest of this topic parses. The model produces each token conditioned on the tokens before it, so the probability it assigns to a whole answer is the probability of the first token, times the probability of the second given the first, times the third given the first two, and onward to the end. A chain of conditional probabilities multiplies rather than adds, which is why a sequence score is a product in the first place and not a total. Multiplying four hundred numbers each around nine tenths pushes the result down by a factor of nine tenths four hundred times over, and the joint probability of 10^-19 quoted above is where that lands.

What fails at that point is the storage rather than the model. A float32 has a smallest positive value it can represent, and any product below that is stored as zero, so two answers of very different quality both come back as zero and can no longer be told apart. Taking a logarithm removes the problem without changing what is being measured, because the logarithm of a product is the sum of the logarithms of its factors, which is the one identity the whole topic runs on. Four hundred moderate negatives added together is an ordinary number in the tens, which is what -38.4 is, and the ordering between two sequences survives the change because the logarithm is increasing — a sequence with the larger probability still has the larger log-probability.

The scale also traps people. Read as though they were linear, -0.4 and -3.2 look like a modest difference in confidence; in probability they are 0.67 and 0.04, a factor of sixteen. Anything that averages, thresholds, or plots these values is working in a space where the distance between -0.1 and -0.2 is small and the distance between -3.0 and -6.0 is enormous, and a threshold picked by eye on the log scale rarely lands where the person picking it thought it would.

That factor of sixteen is the same identity read backwards. Subtracting one logprob from another gives the logarithm of the ratio between the two probabilities, because the logarithm of a quotient is the difference of the logarithms. The gap between -0.4 and -3.2 is nearly three, the exponential of nearly three is about sixteen, and that is the distance between 0.67 and 0.04. The second pair in that paragraph works the same way: -3.0 and -6.0 are three apart, so they stand in a ratio of about twenty to one, even though both look like small numbers sitting close together. Equal steps on this scale are equal multiplications rather than equal differences, which is why the visual distance between two points on a plot of these values carries no information about how much more often one token would be produced than the other.

One further detail decides whether the numbers mean what you assume. Providers differ on whether the returned logprobs describe the model's raw distribution or the distribution after your temperature and truncation settings have been applied, and the documentation is not always explicit. This is worth two requests to settle: send the same prompt at temperature=0 and at temperature=2 and compare the returned values at a position where the model was undecided. If they move, you are reading a post-sampling distribution and every number you derive is a function of your sampling config as much as of the model.

There is a practical cost to turning the flag on, and it is in bytes rather than in tokens or latency. The request costs nothing extra — no additional prompt tokens, no additional generated tokens, no measurable change in time to first byte, because the values were computed during generation whether or not you asked for them. What changes is the size of the response body. A 260-token answer serialises to roughly 1.5 KB of JSON normally; with logprobs: true and top_logprobs: 5 the same answer comes back at around 40 KB, since each position now carries six token strings, six floats and six byte arrays. At a few thousand requests a day that is a gigabyte a week if you log the responses verbatim, which teams discover about a month in. The usual fix is to log a derived summary — the sequence sum, the minimum, the five lowest positions with their tokens — and keep the full array only for a sampled fraction of traffic, which is enough to debug with and small enough to keep for a year.

The shape of that growth matters more than the figure, because it is what decides how the cost moves when the request changes. Asking for alternatives does not add a fixed amount to the response; it multiplies what every position carries, and the multiplier is the number of alternatives requested plus the chosen token itself — which is where the six token strings, six floats and six byte arrays above come from at a setting of five. Doubling top_logprobs therefore roughly doubles the body again, and requesting the documented ceiling of 20 alternatives is a different order of expense from requesting five. None of that touches the computation, since the model produced a score for every vocabulary entry at every position in either case, so the flag is a decision about what gets serialised out of work that had already been done.

The array is mostly boring, and that is the point#

Read the trace above from left to right and the structure is obvious. " is" at -0.0002 and " window" at -0.0009 carry no information about whether the answer is right; after The standard refund there is essentially nothing else English permits. Of the nine tokens, six sit above 0.99 probability. The model was not making a decision at those positions in any sense that matters to a reviewer.

One position was a decision. At "30" the model assigned 0.66 to thirty, 0.20 to fourteen, 0.05 to sixty and 0.016 to ninety — a genuine spread over the numbers a refund policy might plausibly contain. That single value is the only part of the array that carries information about the claim the sentence is making, and it is diluted by eight neighbours that were never in doubt.

This shape repeats at every scale. Run the same exercise on a paragraph and the great majority of positions are punctuation, articles, prepositions, inflectional endings and conventional sentence openings, all sitting within a hair of zero, with a handful of content positions where the distribution actually spreads. Over 200 answers from a policy assistant, averaging 260 tokens each, 71% of positions came back above 0.95 probability, another 16% between 0.8 and 0.95, and only 4% below 0.5. Ten of roughly 260 positions per answer sit anywhere near the region where the word "uncertain" would be reasonable, and the other 250 are the model doing English. Any summary statistic computed over all 260 is therefore a statistic about the 250, unless it is deliberately built to ignore them.

That breakdown is forced by the language rather than being a fact about this particular assistant. Most positions in fluent English are settled by the positions before them: a determiner has to be followed by something that can follow a determiner, an inflectional ending is fixed once the stem and the tense are, and a conventional collocation admits one continuation in practice even where a dictionary would permit several. Where only one continuation is admissible, the training text contained essentially only that continuation after this kind of prefix, so essentially all of the mass lands there — a probability within a hair of one, which is a logprob within a hair of zero. The exact proportions would move on a different model or in a different register, and the shape would not, because what produces it is the redundancy of the language rather than anything the model decided.

The consequence runs through the rest of this topic. A mean taken over the array is a mean taken over grammar, a threshold set on that mean is a threshold on grammar, and any method hoping to read correctness out of these numbers has to find some way of listening to the few positions where a choice was being made at all.

A useful first exercise on your own traffic is to pull one real completion, sort its tokens by logprob, and look at the ten lowest. Those ten are where the model was choosing; whether they are also where it was wrong is a separate question, and one this topic spends most of its length on.

Two further structural facts are easy to miss on a first read. Each value is conditioned on the tokens before it, including the model's own earlier output: if the model commits to a wrong entity at position 4, positions 5 through 40 will report high probability, because they are the natural continuation of that mistake. High probability late in an answer is partly evidence that the answer is internally consistent, which is not the same property as being correct. And the top_logprobs list adds a second dimension worth looking at rather than only the chosen token's value. At "30" in the trace above, the runner-up "14" at 0.20 tells you something the chosen token's 0.66 does not: the alternative under consideration was a different plausible policy window, so the position was a real fork in the answer. Contrast a position where the chosen token sits at 0.66 and the runner-up is a synonym or a different capitalisation of the same word — same number, nothing at stake. Reading the alternatives is what separates "the model was choosing between two answers" from "the model was choosing between two ways of saying one answer", and no arithmetic on the chosen token alone recovers that difference.

The first of those two facts carries more weight than its two sentences suggest, and the chain under it is short enough to check link by link. The model is always scoring a continuation of the text in front of it, and the text in front of it includes everything it has already emitted in this answer. Once a wrong entity has been written at position 4, the tokens that continue it fluently are exactly the tokens the model rates highly, so the array reports near-certainty across a passage that is confidently elaborating a mistake. High probability late in an answer therefore measures agreement between the answer and its own opening, which is a property of the text alone, while correctness is a property of how that text stands to something outside it.

The limits of a per-token number#

It is not the model's assessment of the answer. A self-reported 95 is an assessment, in the sense that the model was asked for one and produced text that looks like one; the logprob is a byproduct of generation that no part of the model chose to emit. The two are different objects and only one of them is cheap and mechanical.

Confusing them is tempting for a structural reason rather than a careless one. The self-reported number arrives inside the same JSON object as the extracted fields, in the same shape, from the same call, and everything about how it is presented says it was measured alongside them. What produced it was a draw from a distribution over the vocabulary, conditioned on a prefix in which the extraction already looked complete, and no step in that draw consulted the extracted value or the page it came from. A number that is generated cannot report on the generation that produced it, and a field taking four distinct values across a whole backfill is what that inability looks like from outside.

It is not a probability that the statement is true. It is a probability over vocabulary entries at one position, given the prefix, under the distribution the model learned. Nothing in that definition mentions the world, and chapter 4 works through a case where a completely fabricated answer carries higher per-token probabilities than a correct one.

That reading is hard to dislodge because ordinary speech runs the two claims together. "The model was confident" and "the model was right" travel as one idea when the subject is a person, and a number presented as confidence inherits the whole of that association. The definition is the defence: the quantity is a share of the mass at one position among the entries of a vocabulary, and a vocabulary holds word pieces rather than facts. Whether the token that share names is true of the world is a question no arrangement of shares over a vocabulary reaches.

It is not free of your configuration. The value depends on the prompt, on the sampling settings, on the model version, and on whatever the provider does between the logits and the response body. Two numbers produced under different configurations are not comparable, which chapter 7 turns into a concrete rule about thresholds.

And it is not available at every position you might want it. A truncated top_logprobs list tells you about the top few candidates and says nothing about the shape of the remaining hundred thousand entries, so questions like "how flat was this distribution overall" cannot be answered from a hosted API at all.

And it is not, on its own, a statement about what was at stake. Reading only the chosen token's value is tempting because it is the one field every response carries and the one number every tutorial prints, and it collapses two situations with nothing in common — two thirds sitting above a rival answer, and two thirds sitting above a different capitalisation of the same word. The alternatives are what separate those, so a system that stores the chosen value and discards the rest has thrown away the more informative half of the array.

What a completion gives you, then, is an array of per-token measurements taken during generation, obtainable for the cost of a boolean and some response bytes, and mostly composed of positions where nothing was being decided. The value of reading one early is that it sets an accurate expectation for everything built on top: any statistic over the whole array is dominated by grammar, the interesting positions are a handful per answer, and the alternatives list is where the distinction between competing answers and competing phrasings lives.

The next question is when a single number extracted from that array actually deserves to be called a confidence score, and the answer starts with the narrowest case there is: an answer that is exactly one token long.