Mean logprob and length normalisation

The mean logprob of a completion is the sum of the per-token log probabilities from chapter 1 divided by the number of generated tokens. It is the default confidence metric in a great many production systems, because it is one line of code, it is free, and it produces a single number in a familiar range that looks like it summarises how sure the model was. Thresholds get set on it, abstention rules get built on it, and review queues get sorted by it.

What the number actually summarises is a different question, and it is answerable by arithmetic rather than by argument. Chapter 1 established the shape of the array: on a typical answer the great majority of positions are punctuation, articles, prepositions and conventional sentence openings, all within a hair of certainty, with a handful of content positions where the distribution spreads. Averaging over that array gives every one of those near-certain positions an equal vote with the few that carry the claim.

Most of the terms are grammar#

Take a typical answer from a policy-answering assistant — three sentences, 47 tokens after tokenisation. Splitting the array by whether the token carries content gives the following:

import math

FUNCTION_TOKENS = {" the", " a", " an", " is", " are", " of", " to", " and",
                   " in", " for", " that", " you", " your", " it", ".", ",",
                   " this", " will", " be", " can", " have", " with", "The"}

def decompose(items):
    filler = [i.logprob for i in items if i.token in FUNCTION_TOKENS]
    content = [i.logprob for i in items if i.token not in FUNCTION_TOKENS]
    mean = lambda xs: sum(xs) / len(xs)
    print(f"filler  n={len(filler):3d}  mean={mean(filler):+.3f}")
    print(f"content n={len(content):3d}  mean={mean(content):+.3f}")
    print(f"overall n={len(items):3d}  mean={mean([i.logprob for i in items]):+.3f}")

filler  n= 38  mean=-0.080
content n=  9  mean=-1.897
overall n= 47  mean=-0.428

Thirty-eight of the forty-seven positions are grammatical scaffolding sitting within a hair of zero. They contribute -3.04 to a total of -20.1, so they pull the mean sharply toward zero while carrying no information about whether the answer is right. The nine content tokens — the policy name, the number of days, the two conditions, the exception — carry -17.1 between them and are the entire factual payload of the sentence.

Every number in that output is one multiplication and one division away from the array itself, and reconstructing them is what makes the rest of the chapter checkable. A mean is a total divided by a count, so a mean multiplied back by its count returns the total it came from: thirty-eight filler positions averaging -0.080 contribute -3.04, and nine content positions averaging -1.897 contribute -17.1. Every position belongs to exactly one of those two groups and none belongs to both, so the two contributions add to the array's total of -20.1. Dividing that total by the 47 positions in the array gives the overall mean of -0.428, which is the number a production system stores.

Reading the two contributions against each other is where the finding is. Four fifths of the positions supplied a little more than a seventh of the total, and the remaining fifth supplied the rest. The average weights every position identically, so the group carrying the claim gets nine votes out of forty-seven while the group carrying no claim gets thirty-eight.

Now consider what happens when those nine tokens are wrong in a fluent way. Suppose the model, instead of naming the correct policy and the correct window, produces a plausible-sounding alternative that the surrounding text makes very likely: a well-known policy name, a round number, a standard-sounding condition. Content tokens like that score better than the correct ones, because the correct answer here happens to contain an unusual figure. At a content mean of -0.6, the overall mean becomes (38 × -0.080 + 9 × -0.600) / 47 = -0.180. The fabrication scores -0.18 against the correct answer's -0.43, and an abstention rule set at -0.7 fires on neither.

That formula is the same skeleton with one term replaced, and it is worth walking rather than reading. The filler term does not move, because a fabrication is written in the same articles and prepositions as a correct answer and those positions score the same either way. Only the content mean changes, from -1.897 to -0.600, an improvement of nearly one and a third on the log scale. The overall mean moves by nine forty-sevenths of that improvement, because nine of the forty-seven positions received it and the other thirty-eight received none, and nine forty-sevenths of one and a third is about a quarter — which is exactly the distance from -0.428 to -0.180.

The proportions in that step are the part to carry forward. A large move in the only quantity that was ever about the answer produced a small move in the number a threshold reads, and the shrinking factor is nothing more than the share of positions that carry content. The sum of the array rose from -20.1 to a little short of -8 along the way, which is a substantial change to the sequence and a modest one to its average.

That is the mechanism in one arithmetic step. The average is dominated by terms that are near-constant across every answer the model produces, so what varies between answers is mostly how ordinary the wording is. A correct answer that has to say something unusual gets a low score; a fabrication assembled from high-frequency material gets a high one.

The domination is worth stating as a property rather than as an observation about one answer. A term that takes nearly the same value in every answer contributes nearly the same amount to every answer's average, and a quantity that is nearly the same everywhere cannot separate one case from another. Grammar is that kind of term. An article, a preposition or a sentence-final full stop sits within a hair of zero in any fluent answer, right or wrong, because its probability is settled by the words around it rather than by any fact about the world. What is left free to vary is the content mean, and the average divides that quantity's influence by the ratio of content positions to total positions before letting it move anything.

It is worth being precise about why the mean rather than the sum is the thing people reach for, since the alternative fails even more obviously. The sum of the array is the log probability of the whole sequence, which is the mathematically meaningful quantity — it is what the model actually assigns to that string. It is also monotonically decreasing in length, because every additional token adds a negative number. The 47-token answer above sums to -20.1; a 200-token answer of identical quality sums to somewhere near -85. Any threshold on the sum is a threshold on length wearing a disguise, and dividing by the token count is the obvious repair. The repair removes the crude length dependence and leaves a subtler one, and it introduces the filler-domination problem as the price of the fix. Both the sum and the mean are legitimate summaries of the sequence; neither is a summary of the sequence's correctness.

Each half of that repair is worth checking rather than accepting. Every entry in the array is the log of a probability, every probability is at most one, and the log of a number at most one is at most zero — so each additional token can only push the sum further down and never back up. A threshold on the sum therefore fires on long answers whatever their quality, which makes it a length rule wearing a confidence name. Dividing by the token count converts a total into a per-position quantity, and per-position quantities are at least comparable between answers of different lengths, which is why the division is the right first move. What it cannot fix is that the mix of positions changes with length, so the quantity being averaged is not the same quantity in a short answer and a long one.

The residual length dependence, measured#

Because the filler ratio rises with answer length — longer answers spend proportionally more of their positions on connectives, subordinate clauses and transitions — the mean climbs toward zero as answers get longer, regardless of correctness. Measured across 600 answers from the same assistant, binned by length and split by whether a human judged the answer correct:

Read the columns rather than the rows. Within any single length band, correct and incorrect answers are separated by 0.03 to 0.08 — a real effect, but a small one. Across bands, the swing from short to long is 0.39, five to ten times larger than the correctness signal. A threshold at -0.7 therefore fires almost exclusively as a function of length: on this data it abstains on 31% of answers under 25 tokens and 2% of answers over 60, and the correct-versus-incorrect split inside each of those groups is nearly even.

The climb down both mean-logprob columns has three links and each can be checked on its own. A longer answer needs more connectives, more subordinate clauses and more transitions to hold itself together, so the share of its positions given over to grammatical scaffolding rises with its length. Those positions sit within a hair of zero, which the decomposition above measured directly. An average is pulled toward the values that make up most of its terms, so a larger share of near-zero terms drags the average closer to zero. That is the movement from -0.63 in the shortest band to -0.24 in the longest, and it happens in the correct column and the incorrect column alike, which is what tells you it is not about correctness.

The gap column also shows the correctness signal shrinking as answers get longer, from 0.08 in the shortest band to 0.03 in the longest. That follows from the same mechanism: a 200-token answer has proportionally more filler than a 20-token one, so whatever the content tokens are doing gets averaged away more thoroughly. The score is weakest on exactly the long, discursive answers where a human reviewer would most like a hint about where to look.

Pooled across all lengths, mean logprob separates correct from incorrect answers with an AUROC of 0.71. That statistic is the probability that a randomly chosen correct answer scores higher than a randomly chosen incorrect one, so 0.5 is a coin flip and 1.0 is perfect separation, and 0.71 looks like a working detector. Computed within each length band it drops to 0.58, which is barely above chance. The pooled figure is mostly re-measuring the length distribution.

AUROC is the right statistic for this comparison for a reason worth naming. It depends only on the order the score puts the answers in and not on the values themselves, so multiplying every score by a constant or shifting all of them leaves it unchanged. That makes it the one number on which a mean logprob, a minimum logprob and an agreement fraction can be compared without first being put on a common scale, and it is why this topic reports it rather than an accuracy.

Why the pooled figure sits so far above the within-band one is worth working out, because the same trap is available to any score. Pooling puts short and long answers into a single ranking. Length moves the score by 0.39 across the bands while correctness moves it by between 0.03 and 0.08 inside one, so a ranking by this score is very nearly a ranking by length. In this sample the short answers are also the harder ones, so a ranking by length is accidentally close to a ranking by difficulty, and a statistic that looks only at ordering rewards the coincidence.

That is a fact about this sample rather than a subtlety of the statistic. On a set where length and difficulty happened to be unrelated, the pooled and within-band figures would sit close together and the pooled 0.71 would have been honest. Nothing in the pooled number says which of those two worlds produced it, which is why computing it within bands is not an optional refinement.

Holding length constant#

The single experiment that settles whether a score like this carries signal is to hold length constant. Take a labelled set of a few hundred answers, bin them by output token count into three or four bands with at least fifty answers each, and compute the correct-versus-incorrect separation of mean logprob within each band. Then compute the same separation using output length alone as the score, ignoring logprobs entirely.

Three outcomes are possible and each dictates a different action. If within-band separation collapses to chance while length alone reproduces the pooled result, the rule is a length filter wearing a confidence costume, and it should either be replaced or rewritten honestly as "abstain on short answers". If within-band separation holds up, the score is carrying real signal and the threshold should be derived per band rather than globally. If both carry signal, you have two features and should say so rather than pretending you have one. The experiment costs a few hundred labels and an afternoon, and it is the only version of the question that a larger evaluation set cannot answer for you.

Three beliefs make that experiment feel unnecessary, and each is tempting for a nameable reason. The first is that an abstention rule with 63% precision is working, which is tempting because the precision is real, measured, and far better than a rule picking at random would deliver — and wrong because a rule that abstains on short answers inherits whatever precision the short-answer bucket happens to have, without having measured any property of the answer it suppressed. The second is that more labelled data will settle a threshold that behaves unpredictably, which is tempting because more data usually does narrow an interval — and wrong here because the width of the interval was never what made the threshold unsound. The third is that a low mean logprob means the model was unsure about the facts, which is tempting because the number is assembled out of probabilities and probability is the vocabulary of uncertainty — and wrong because the probabilities it is mostly assembled out of belong to commas.

A partial repair exists and is worth knowing about, because someone will propose it. Since the length dependence is smooth and roughly monotonic, you can fit it and subtract it: regress mean logprob on log token count over a labelled sample, and score each answer by its residual — how much more or less ordinary its wording is than a typical answer of that length. On the 600 answers above, the residual score reaches an AUROC of 0.62 pooled, which is worse than the 0.71 the raw pooled score claimed and better than the 0.58 the raw score really had. That is honest progress, because the number it produces is no longer a re-description of length. What it leaves untouched is the filler domination from the first section: the residual is still mostly a statement about phrasing, so the fluent fabrication still scores well. Length normalisation makes verbose and concise responses comparable to each other, which is a real improvement, and it does nothing about the fact that the few tokens determining correctness are outvoted by the many that do not.

Subtracting a fitted curve is legitimate here for a narrow reason worth stating. The regression estimates the part of the score that is a function of length alone, and removing a component that depends only on length cannot remove any part of the score that depends on correctness, because the two are functions of different things. What is left is the answer's departure from the typical wording of answers its size, and a departure is at least a quantity about this answer rather than about how long it is. The 0.62 is what an honest version of the 0.71 looks like.

What the residual inherits whole is the first section's problem, and nothing done to length touches it. The quantity being regressed is the overall mean, which is dominated by grammatical positions, so the residual is a statement about how ordinarily an answer is phrased for its size. A fluent fabrication is phrased ordinarily by construction — that is what makes it fluent — so it scores well on the residual for the same reason it scored well on the raw mean.

Where the mean earns its place#

None of this makes mean logprob a useless number. It fails specifically as a correctness proxy, and it succeeds at something narrower that is genuinely worth monitoring: detecting output that has gone off the distribution entirely. Repetition loops, truncated garbage, responses that come back in the wrong language, outputs where a prompt-injection attempt has dragged the model into an unrelated register — all of these produce mean logprobs far below anything normal traffic generates, often -1.5 or lower where healthy answers cluster between -0.2 and -0.6. As a canary on the generation itself, with a threshold set several standard deviations below the observed mean rather than at the edge of the normal range, it earns its place in a dashboard.

The reason that narrower use works, where the correctness use does not, is that it asks a question the number can answer. A mean logprob is the model's own rating of the text it produced, averaged over its positions, and output that has left the distribution the model was fitted to is output the model rates poorly at position after position rather than at the handful of positions carrying a claim. The dilution that ruins the correctness reading is not a problem here: when the whole array is depressed, an average over the whole array is the right instrument.

That reading holds where the departure is spread across the output — a wrong language, a truncated fragment, a register dragged somewhere unrelated — and it is weakest where a fault is confined to a few positions, which is the same limitation in different clothes. It also settles where the alarm's threshold belongs. Set at the edge of the normal range it fires on short answers, for the reasons the length table gives; set several standard deviations below the observed mean it fires only on output that no length band produces. A quantity thresholded that far out costs almost nothing in false alarms, and gives up any pretence of catching an individual wrong answer.

There is also a better-motivated variant than the mean, and it follows directly from the decomposition at the top of this chapter. If the decisive positions are the few low-probability ones, score the answer by its minimum token logprob, or by the mean of its lowest five, and stop letting the filler vote:

def scores(items):
    lps = sorted(i.logprob for i in items)
    return {
        "sum":     sum(lps),                    # falls with length; unusable alone
        "mean":    sum(lps) / len(lps),          # dominated by filler
        "min":     lps[0],                       # the single riskiest position
        "mean_k5": sum(lps[:5]) / min(5, len(lps)),
    }

On the same 600 answers, mean_k5 separates correct from incorrect within a length band at an AUROC of 0.66 against the mean's 0.58 — better, still short of the pooled illusion, and noisier because it depends on a handful of tokens that a rephrasing can move. It is the right shape of idea: look where the model was actually choosing.

That those variants do better follows from the decomposition rather than from the measurement, which is what makes it a prediction and not a finding. Sorting the array puts the most negative entries first, the most negative entries are the least probable ones, and the first section showed that those are the content positions. Taking the lowest entry, or the mean of the lowest five, therefore selects the group carrying the claim and gives the thirty-eight grammatical positions no vote at all. The noisiness is the direct cost of the same choice: a statistic built from five positions moves whenever any one of those five is rephrased, where a statistic built from forty-seven barely notices.

The mean of an array whose terms are mostly grammar is a measurement of grammar, and that is the whole of the finding. Dividing by token count is the right repair for the sum's crude length dependence and leaves a residual one large enough to swamp the correctness signal, which is why the pooled AUROC of 0.71 collapses to 0.58 inside a length band. The number remains useful as an off-distribution alarm and as the raw material for the minimum-token variants, and it is not a correctness estimate at any threshold.

Both of those variants rest on the same assumption, which is that a wrong answer contains at least one position where the model hesitated. Chapter 4 takes an answer that is entirely fabricated and shows that every position in it can sit above 0.9, because the fabrication is exactly what the surrounding text made likely.