Hallucination and token probability

A great deal of production tooling is built on the assumption that fabrications look uncertain from the inside — that a model inventing a fact hesitates in a way the logprob array records, and that a threshold somewhere will therefore catch it. The assumption is testable, because a fabrication and its correct counterpart can both be traced token by token and compared on the same statistics.

The assumption is tempting for a reason worth naming, because holding it is not carelessness. A person who invents a fact does hesitate, and the array arrives looking like a record of the model's own deliberation: one number per position, produced at the moment that position was decided, rather than composed afterwards by something with an interest in the answer. Everything about where it comes from suggests introspection. What it actually records is how ordinary each token was in its slot, and ordinariness is a property of the surrounding text rather than of the model's grip on the subject, so the array can be a faithful measurement of one thing and silent about the other.

The comparison below runs on a fabricated legal citation, which is a convenient specimen because a citation has an unusually rigid format and a small number of arbitrary values inside it, so the two kinds of position separate cleanly.

The trace of a fabrication#

In May 2023 a federal judge in the Southern District of New York asked a law firm to produce copies of six cases cited in a brief filed on behalf of a passenger suing an airline. One of the six was Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) — a case name, a reporter volume, a page number, a circuit and a year, all correctly formatted, all invented. Run a citation-shaped prompt against a model and print the array for that string, produced under a prefix that had already established a legal-brief register:

def trace(resp):
    for item in resp.choices[0].logprobs.content:
        p = math.exp(item.logprob)
        bar = "#" * int(p * 20)
        print(f"{item.token!r:>8} {item.logprob:7.3f}  p={p:.3f} {bar}")

 'Var'  -2.104  p=0.122 ##
'ghese'  -0.087  p=0.917 ##################
    ' v'  -0.006  p=0.994 ###################
     '.'  -0.002  p=0.998 ###################
 ' China'  -1.883  p=0.152 ###
'Southern'-0.041  p=0.960 ###################
' Airlines'-0.019 p=0.981 ###################
   ' Co'  -0.114  p=0.892 #################
     ','  -0.031  p=0.969 ###################
    ' 9'  -1.402  p=0.246 ####
    '25'  -0.938  p=0.391 #######
     ' F'  -0.024  p=0.976 ###################
    '.3'  -0.011  p=0.989 ###################
     'd'  -0.004  p=0.996 ###################
  ' 133'  -1.147  p=0.318 ######
     '9'  -0.902  p=0.406 ########
    ' ('  -0.048  p=0.953 ###################
    '11'  -0.617  p=0.540 ##########
    'th'  -0.003  p=0.997 ###################
  ' Cir'  -0.017  p=0.983 ###################
   '. 2'  -0.202  p=0.817 ################
    '019'  -0.688  p=0.503 ##########
     ')'  -0.009  p=0.991 ###################

Twenty-three tokens, mean -0.454. Sixteen of them sit above 0.9, and every one of those sixteen is a piece of citation format — the v., the F.3d, the Cir., the parenthesis, the continuation of a name whose first fragment has already been committed to. The prefix made a citation-shaped object overwhelmingly likely, and the model produced a citation-shaped object with great confidence. The seven positions that carry real spread are the ones choosing digits and the ones choosing which name to start.

Reading down the column position by position shows where the high numbers come from, and each one has a reason a reader can check without taking the model's word for it. 'ghese' sits at 0.917 because 'Var' above it has already committed the text to a surname, and once the opening fragment of a surname is on the page the remaining fragments are close to forced. ' F' at 0.976 and '.3' at 0.989 are the reporter designation, which follows a volume number in this format and is drawn from a short fixed list. ' Cir' at 0.983 follows 'th' for the same kind of reason, and the closing ')' at 0.991 follows the year because a parenthesis opened before the circuit has to close after the date. Every one of those is a rule about how a citation is written down.

The positions that behave differently are the ones where an arbitrary value has to be chosen, and it is worth saying why arbitrariness is the thing that produces breadth. ' 9' at 0.246 and '25' at 0.391 are the reporter volume; ' 133' at 0.318 and '9' at 0.406 are the page. A volume number is one integer out of a large range of integers that would all read as perfectly ordinary in that slot, and the model's whole job at that position is to say how ordinary each candidate is. When the candidates are all about equally ordinary the mass has to spread across all of them, and no single one of them can hold much of it. '11' at 0.540 and '019' at 0.503 are the same situation over a narrower range, which is why they spread less and still spread.

Now put a real citation from the same area of law through the same trace and compare the summary statistics:

The genuine citation scores worse on both metrics. This is not a fluke of these two examples; it is what the mechanism predicts. A real reporter volume is one arbitrary number among thousands of equally plausible arbitrary numbers, so the model's distribution over it is broad whether or not it happens to produce the right one. Digits carry uncertainty in correct answers for exactly the same reason they carry it in fabricated ones, and the two are not separable from the numbers alone.

The step that makes this a prediction rather than a coincidence is that breadth at such a position is a property of the slot and not of the value that ends up filling it. Nothing about landing on the volume a case really has would narrow the distribution, because the model's estimate was formed before the value was chosen and from the same evidence either way. So the breadth is present in identical measure in a citation that resolves and in one that does not, and a statistic computed over the array meets the same broad position in both cases with nothing to key on.

The minimum-token column moves for a different reason, and it moves against the genuine citation. Both minima fall on the onset of a party name, where the model is choosing which of a great many possible parties to put in the slot, and that is the widest choice anywhere in either string. The table records Z as the less expected of the two openings, so the correct citation's single worst position is worse than the fabrication's. Neither number was ever measuring whether a case exists; both are measuring how common a name fragment is, and a scoring rule that reads the minimum will rank the two exactly the way the table does.

Measured over 400 labelled answers from an internal question-answering evaluation, the minimum-token score from chapter 3 separates fabricated from correct answers with an AUROC of 0.68 — real signal, mostly coming from cases where the model produced something garbled or hedged. Restricted to the subset a human annotator marked as "fluent and confidently stated but wrong", it falls to 0.51. On the failures that matter most, the score is a coin flip.

The gap between those two figures is where the second stumble lives, and it is easy to walk past. The 0.68 is real signal, which is why a team that measures the pooled number comes away believing the detector works. What it is mostly detecting is output that came back garbled or hedged, and output like that is visible to a reviewer with no array at all. The subset where the model states something fluently, confidently and wrongly is the subset a detector was wanted for, and there the score is 0.51. A detector that fires on the errors you would have caught anyway and stays quiet on the ones you would not is worse than no detector, because it also hands over a number saying the case was examined.

The objective has no term for truth#

The reason is in the training procedure rather than in any particular model. A language model is fitted by minimising cross-entropy against the next token actually observed in a training corpus, which is a penalty that grows whenever the model assigned low probability to the token that in fact came next. Every gradient the model ever received was a signal about which continuation appeared, in text, after some prefix. There is no term in that loss for whether the sentence corresponds to anything, and at inference there is no step that consults a source. The distribution the model produces is a description of what continuations are typical given the prefix, under the statistics of the text it was fitted to. Typicality and truth coincide often enough for the models to be useful, and they come apart precisely where the training text contained a strong pattern but no specific fact.

Unpacking what "minimising cross-entropy against the next observed token" does is worth the two lines it takes, because the whole argument rests on it. Training shows the model a prefix taken from a corpus, lets it produce its distribution over what comes next, and then looks up the token that in fact came next in that corpus. The penalty charged is large when the model gave that observed token a small share and small when it gave it a large one. So the only pressure ever applied is toward putting mass where the corpus put text, and the corpus is the sole authority in the arrangement.

Inference adds nothing the training procedure left out. A forward pass reads the prefix and produces one score per vocabulary entry, and no step in it queries a database, a search index or a document; there was no source in the loop during training and there is none at inference either. Typicality is the only quantity the whole arrangement ever optimised, and truth reaches the output only through the correlation between the two in the corpus. Where that correlation is strong, because a fact is stated the same way across a great deal of text, the typical continuation is usually the true one. Where the corpus carried a strong pattern and no specific fact, the correlation is simply absent, and the model still returns its most typical continuation — which is what a fabrication is.

That framing also explains why the sixteen high-probability tokens above are uninformative. A distribution can be sharp for two quite different reasons. It can be sharp because the training text consistently contained one continuation after this context, which is the case that makes a high probability meaningful — the model has seen the answer many times and it does not vary. Or it can be sharp because the local structure admits only one shape, which is what is happening at .3, th, Cir and the closing parenthesis: nothing about legal citations permits an alternative there, whatever the case being cited. Both come back as 0.99. Nothing in the returned array distinguishes them, and hosted APIs do not return enough of the vocabulary vector to let you reconstruct the difference.

Only the first of those is worth acting on, and there is a clean way to see that the second carries nothing. Sharpness of the second kind would be identical for a case decided last year, a case decided a century ago and a case invented five seconds ago, because none of those change what can legally follow an open parenthesis. What would tell the two apart is how much of the vocabulary was ever in contention at that position, and that is the one quantity a truncated response body does not carry.

The obvious next idea is to stop reading the chosen token and read the shape of the distribution instead — compute the entropy over the returned alternatives at each position, on the theory that a broad distribution signals genuine uncertainty regardless of which token was picked. Entropy here is a single number summarising how spread out those alternatives are, near zero when one candidate holds almost all the mass and rising as the mass is shared between several. It is a better-motivated statistic than the mean, and it runs into two walls. The first is the truncation from chapter 1: entropy computed over five or twenty returned candidates is not the entropy of the distribution, and how much mass sits outside the returned list is exactly what you cannot see. On a peaked position the top five hold 0.999 and the truncation is harmless; on a flat position they might hold 0.3, and the truncated entropy is then a severe underestimate of the real one, with the error largest precisely where the signal would matter. The second wall is the one this chapter is about: entropy over the citation's format tokens is near zero for the same reason their probabilities are near one, so the statistic inherits the same dilution problem in a more expensive form.

The first of those two walls deserves its arithmetic rather than its name. Entropy is a sum with one term per candidate in the distribution, and every term in it is non-negative, so dropping candidates can only remove terms and can only make the total smaller. A returned list of a few alternatives is a prefix of the sorted distribution, which makes the number computed from it a lower bound on the real entropy rather than an estimate of it, and nothing in the response body says how far below. That is also why the shortfall runs the wrong way round: it is negligible on a position where the top few candidates hold nearly everything, and it is most of the distribution on a position where they do not.

Underneath the second wall is a property of the number itself, and it explains why every statistic tried so far inherits the same defect. A probability at a position is a share among the tokens that could occupy that one position, so the only competition it is capable of expressing is a competition that shows up as two different tokens in the same slot. Two claims about the world produce that shape only when they differ at a single token and agree on everything before it. That is an arrangement somebody can build deliberately, and it is not one that free prose falls into.

Chapter 2's single-token classifier escapes this because the answer space was constrained by construction. The five keys were mutually exclusive, exhaustive within the design, and semantically distinct, so a distribution spread across them meant the model was genuinely weighing alternative answers. In free text there is no such guarantee: the alternatives at any given position are usually alternative phrasings, and the model weighing 925 against 926 is not weighing two competing claims about the world in any sense that a probability can express.

Finding a confidently wrong example in your own domain#

The exercise is worth doing on your own material rather than reading about someone else's, because the specific shape of confident fabrication differs by field and seeing it in your own traces is what makes the limitation stick.

Take a query where you know the correct answer and where the answer contains a specific value the model is unlikely to have memorised — an internal configuration key, a part number, a version-specific default, a threshold from a document not on the public web. Ask without providing the source. Capture the full logprob array. Then ask the same question with the source pasted into the prompt, so the model produces the correct answer, and capture that array too.

Two controls make the comparison interpretable. Ask each question at the same temperature in both conditions, since chapter 1's caveat about post-sampling distributions means a temperature difference alone can move every number. And keep the prompts the same length where you can, or at least record the lengths, because chapter 3 established that answer length shifts the mean by more than correctness does, and a grounded answer that quotes the source is usually longer than a fabricated one.

Both controls exist to stop the same thing happening, which is a separation appearing in the numbers for a reason that has nothing to do with correctness. Left uncontrolled, the temperature difference would move every value in both arrays before a single fact was involved, and the lengths would move the two means apart on their own. Either would produce a visible gap between the fabricated and grounded conditions, and a gap is exactly what the experiment is looking for — so without the controls the run cannot fail, and a result that cannot fail is not a measurement.

What an internal run of this on twelve API-documentation questions produced is that the two arrays are statistically indistinguishable. Mean logprob for the fabricated answers came in at -0.31, for the grounded answers at -0.34. Nine of the twelve fabricated answers scored higher than their grounded counterparts. The one visible difference was in the token that named the config key itself: fabrications named plausible keys built from common fragments — retry_backoff_ms, max_connection_pool_size — at around -1.1, while the grounded answers copied an unusual real key from context at around -0.15. The signal, such as it is, points the wrong way: copying from context is easy and scores high, inventing is harder and scores lower, and neither says anything about whether the key exists.

Two kinds of uncertainty#

It would be an overcorrection to conclude that per-token probabilities carry nothing. Where the model's uncertainty is genuinely about which of several alternatives to state — the five labels in chapter 2, a digit it half-remembers, a name it has seen in two conflicting forms — the distribution really does spread, and the spread really does correlate with error. The 0.68 AUROC above is not zero. That uncertainty is visible because the alternatives are competing tokens at the same position.

Other uncertainty is about whether the whole frame is right — whether the case exists, whether the config key was ever real, whether the question presupposes something false. That uncertainty has no competing token to appear at, because the model is not choosing between "cite a case" and "say no such case exists" at position 11 of a citation; it settled that several tokens earlier, if it considered the question at all. Per-token probabilities are a window onto the first kind and blind to the second, and the second is where hallucination lives.

It is also worth being clear about what retrieval does and does not change here. Putting the source document in the prompt makes the answer far more likely to be right, which is the point of doing it. It does not give the probabilities a channel to the world; it changes what is typical, so a grounded answer is now typical and true, and an ungrounded fabrication in the same pipeline still scores the same as before. Grounding fixes the answer, not the signal, and a retrieval-augmented system with a logprob threshold on top is still thresholding typicality.

A closely related move fails for a closely related reason, and it is the one the brief above rested on. Asking the model whether its own citations are real is tempting because the answer comes back as a definite yes or no in exactly the register of a check, and because the alternative is looking six cases up by hand. It is not a check. The second question is answered by a second generation, conditioned on a prefix in which the citations already sit as ordinary well-formed text, and the ordinary continuation after a list like that and a question about it is a confirmation. Nothing in the second pass consulted anything the first pass had not.

High probability is a statement about how ordinary a continuation is, and the fabricated citation is ordinary in every respect that the model can measure: the format is standard, the name fragments are common, the volume number is one arbitrary integer where an arbitrary integer belongs. That is why it scores better than a real citation, and why the class of failure that reaches production undetected is exactly the class the numbers cannot reach. A fabrication that looks uncertain is one your reviewers were going to catch anyway.

The consequence for design is that no measurement of a single generation can tell you what you want to know, because every position in a single generation is conditioned on the same prefix and the same decisions. If the model commits to a fabrication early, everything after it is confidently consistent with the fabrication. Getting a second opinion requires the model to commit more than once, and the standard way to arrange that is to ask it the same question several times and see whether it says the same thing.