Hallucination and next-token prediction

The word is used loosely enough to be worth pinning down before anything leans on it. A model that answers a question about a company's internal systems with a retry count, a backoff schedule and a document citation has produced four or five separable factual claims; a hallucination is any one of those claims that nothing supports. The defining property is not that the claim is wrong — plenty of software returns wrong values — but that it is wrong in a form no ordinary engineering control can catch. A service returning 5 where it should return 3 is caught by an assertion, a schema, or a type. A sentence saying five where the truth is three is exactly as valid, character for character, as the sentence saying three, and no assertion can be written against prose whose correct wording nobody enumerated in advance.

That property is what makes the category worth naming: naming it is what allows it to be counted, given a rate and a threshold and an owner, in the way an error budget is. This chapter is about where the behaviour comes from, which turns out to be the training objective itself rather than any defect in a particular model or prompt. It sets out what the objective optimises, shows the probability distribution at the exact token where a model commits to a fabricated citation, covers a second pressure from how models are evaluated, and then rules out the fixes that follow from misdiagnosing it — lower temperature, better decoding, better calibration.

Nothing in the loop holds a truth value#

Getting at the cause needs the generation process stated plainly first, because everything below is a consequence of its shape. Generation does not produce sentences. A model reads a sequence of tokens — the word fragments a tokenizer cuts text into — and produces one number for every entry in its vocabulary, describing how well that entry fits as the next fragment. Turning those numbers into a probability distribution makes each of them non-negative and makes them sum to one, at which point each is the model's estimated likelihood of that entry coming next. One token is then chosen from the distribution, appended to the sequence, and the whole procedure runs again over the sequence that now includes it. A four-hundred-word answer is that loop several hundred times over, and nothing larger than a token is ever decided at any point in it.

Training shapes those distributions and shapes nothing else. A stretch of human-written text is fed in, and at each position the model is asked what comes next before being shown what actually did. Cross-entropy is the score used: it is small when the model gave high probability to the token that turned out to be there and large when it gave that token low probability, so minimising it means being unsurprised as often as possible. The weights are then nudged in whichever direction would have made the observed token likelier. Repeat that across a very large body of text and the result is a model whose distributions match the statistics of the text it was fitted to.

What matters for the failure this chapter is about is what that objective contains, and therefore what it optimises for: a probability distribution over sequences that matches the distribution of the training text as closely as possible. There is no second term in the loss for correspondence with the world, no lookup against a knowledge base, and no representation of "this claim is checkable and I have not checked it".

That absence is not an oversight in the training setup, it is a consequence of what the data can supply. A body of text records what people wrote, and people write true things, false things, fiction, drafts, satire, outdated documentation and speculation, all in the same registers. A model fitted to that material learns the form of a confident technical assertion very well, because confident technical assertions are abundant and highly patterned. It learns the specific content of any given assertion far less reliably, because that content appears once or twice while the form appears millions of times. When the two come apart at generation time, the form wins, because the form is what the distribution is sharpest about.

That is the chain answering the question this chapter is named for, and it is worth laying out link by link, because "fluent output is not evidence of true output" reads as a slogan until the mechanism behind it is visible. Training adjusts weights in proportion to how much a pattern reduces surprise across the corpus. The grammatical and rhetorical shape of a confident technical sentence recurs in millions of places, so it reduces surprise millions of times, and the weights come to encode it very sharply. A particular fact — one team's retry budget — appears in a handful of places, so it reduces surprise a handful of times, and the weights encode it faintly if at all. At generation time both are consulted through the same mechanism and expressed through the same output, a single distribution over next tokens. The sharply encoded thing therefore drives the choice, and the faintly encoded thing barely participates in it.

The consequence is the one that matters operationally. Fluency and truth are produced by one process out of evidence of wildly different strength, so a sentence being well-formed tells you that the strongly encoded part of the model was engaged — which it always is. It carries no information about whether the weakly encoded part had anything to contribute. That is not a correlation that happens to be weak. It is two quantities with no common cause, which is why no amount of reading the output separates them: reading is precisely the thing the strong signal is good at satisfying.

One way to see why no truth term could be added is to ask what it would be computed against. Supervised training needs a target, and a target for truth would require, for every span of text in a multi-trillion-token training set, a label saying whether the proposition it expresses holds. Nobody has that, nobody can construct it at that scale, and for a large share of the material the question is malformed — dialogue in a novel, a hypothetical in a design document, a comment describing behaviour the code no longer has. The objective that can be computed is the one that is computed: how surprised was the model by the token that actually appeared. Everything the model knows about facts is a side effect of getting good at that.

That side effect is uneven in a way that maps directly onto which questions get answered wrongly. A fact stated ten thousand times in consistent phrasing — the boiling point of water, the syntax of a for loop — is absorbed as a sharp, reliable regularity, because predicting the next token in those contexts requires it. A fact stated twice, in one repository's internal documentation, is barely a regularity at all; the gradient signal from two occurrences is negligible against the signal from the millions of occurrences of the surrounding form. Internal systems ask almost exclusively about the second kind, which is why an assistant that is impressively accurate on general programming questions is unreliable on your own retry budget, and why that unreliability is invisible from a demo.

The claim here is narrower than the loose version that circulates, and the narrow version is the one that holds. Parameters absorb an enormous amount of stable, high-frequency fact, and on well-attested material a large model is right far more often than chance. What is true is that at each generation step, the quantity being maximised is the fit of a continuation to learned patterns, and a false continuation that fits the patterns well outranks a true continuation that fits them poorly.

What the distribution looks like at the moment of invention#

The abstraction becomes concrete at the token where a model commits. Anthropic's API does not expose per-token probabilities, so the trace below comes from a locally hosted open-weights model asked how many times a payments webhook consumer retries. The model has just emitted PAY-114 section 3. and is choosing the next token.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

prompt = (
    "Q: How many times does the payments webhook consumer retry?\n"
    "A: The consumer retries up to five times with exponential backoff, "
    "as documented in runbook PAY-114 section 3."
)
ids = tok(prompt, return_tensors="pt").input_ids
logits = model(ids).logits[0, -1]
probs = torch.softmax(logits.float(), dim=-1)

top = torch.topk(probs, 5)
for p, t in zip(top.values, top.indices):
    print(f"{tok.decode(t)!r:>8}  p={p:.3f}")

entropy = -(probs * probs.clamp_min(1e-12).log()).sum()
print(f"entropy = {entropy:.2f} nats over {len(probs)} tokens")

     '2'  p=0.312
     '1'  p=0.244
     '3'  p=0.191
     '4'  p=0.108
     '5'  p=0.061
entropy = 2.41 nats over 128256 tokens

One term there needs defining before the output can be read at all. Entropy measures how spread out a probability distribution is: it is largest when every option is equally likely and smallest when one option holds all the mass. A nat is the unit it comes out in when natural logarithms are used, and the only property needed here is a comparison. A distribution spread evenly across a vocabulary has an entropy equal to the natural logarithm of the vocabulary size, which for these 128,256 entries is the 11.8 nats mentioned below. A measured 2.41 sits far under that, so the model has ruled out almost the whole vocabulary and is choosing among a handful of survivors.

Three things are visible in that output. The distribution is concentrated: five tokens hold 91.6% of the mass, and every one of them is a plausible subsection number. The entropy of 2.41 nats over a 128,256-token vocabulary is low — a uniform distribution over the vocabulary would sit near 11.8 nats — which means the model is, in the only sense it can be, confident. And the confidence is entirely about format. The model has correctly inferred that what follows section 3. in a citation is a single digit, and it has ranked the digits by how often each appears in that position across everything it has read. It has not consulted the document, because there is no step at which it could.

The 91.6% is those five listed probabilities added together, and performing the addition is worth the moment it takes, because it is what turns the rest of the vocabulary into a rounding error. Whatever mass is left over is spread across more than a hundred and twenty-eight thousand other entries, which is why every plausible sample at this step is a digit and none of them is a word, a punctuation mark or a line break. The model has settled the kind of thing that comes next with something close to certainty, and left the identity of it to a near-even draw among five candidates.

Sampling from that distribution gives 3.2 a bit under a fifth of the time, 3.1 roughly a quarter, and so on. Every branch produces a well-formed citation, and the branches are indistinguishable from the outside, because they differ by one character in a string that is grammatically and typographically identical either way. The reader sees a citation. What was generated was a five-way coin flip over subsection numbers, decided by prior frequency. Extend that across a whole response and you have the general case: a four-sentence answer containing eleven separable factual claims runs the same lottery eleven times, at eleven different levels of concentration, and returns one flat block of prose with no per-claim annotation of which draws were sharp and which were near-ties.

Guessing scores better than admitting ignorance#

A second pressure operates on top of the pretraining objective rather than instead of it. Most evaluation of language models scores a response as right or wrong. Under that rule, saying "I don't know" scores zero, and guessing scores whatever the model's chance of being right happens to be. Any positive chance beats certain zero, so a model tuned against such benchmarks — through fine-tuning on curated examples, preference optimisation against human rankings, or plain selection of which checkpoint to ship — is pushed towards producing an answer whenever it has any belief at all. The paper Why Language Models Hallucinate makes this argument at length, and it is the sharpest available account of why the behaviour persists in systems whose builders would very much like it not to.

Take a question where the model's best candidate answer has probability p of being correct, and compare the expected score of answering with the expected score of abstaining, under three grading schemes.

Each cell in the first three columns is an expected value, which means the score attached to each outcome multiplied by the chance of that outcome, with the products added up. Under binary grading a right answer scores one and a wrong answer scores nothing, so answering is worth the chance of being right and the row is p copied across. Under penalised grading a wrong answer scores minus one, so the expected value is the chance of being right minus the chance of being wrong, and where p is low the second term dominates and drags the total below zero. Abstaining scores zero in both schemes, so it wins whenever the answering row is negative — which under penalised grading is exactly when the chance of being right falls below the chance of being wrong. Partial credit arrives at the same threshold from the other direction, by paying something for an abstention instead of by punishing an error.

Under the first row — which describes the overwhelming majority of public leaderboards and a good share of internal evaluation harnesses — a model that abstains whenever it is unsure is strictly dominated by a model that always guesses. Under the second and third rows the calculus changes and abstention becomes rational below a threshold, but you only get that behaviour if you actually grade that way, and almost nobody does by default.

None of this requires the model to have an intention, and the vocabulary is worth handling carefully. Calling the output a lie imports a mental state — knowing the truth and asserting otherwise — that has no counterpart in the machinery. The model did not know a retry budget was three and say five anyway. It had no separately stored proposition about the retry budget at all, only a distribution over continuations in which five outranked three given the surrounding words.

Why turning the temperature down does not help#

The most common first fix is to reduce randomness: drop the temperature to zero, or decode greedily, on the theory that the invention comes from sampling. It does not. Greedy decoding at the trace above emits 2, producing the citation PAY-114 section 3.2 — a section that does not exist — with complete determinism. Lowering temperature narrows the set of outputs you can get; it does nothing about whether the highest-probability output is true, because probability and truth are different orderings over the same candidates. What temperature buys is reproducibility of the error, which is genuinely useful for debugging and useless as a fix.

The same reasoning disposes of the decoding strategies one rung up. Beam search keeps several partial sequences alive and returns the one with the best total log-probability, which is a more thorough search of the same landscape and therefore finds a more probable citation, not a truer one. Nucleus and top-k sampling truncate the tail before sampling, which removes low-probability tokens; the fabricated 2 was the highest-probability token, so no truncation reaches it. Self-consistency — sample the answer five times and take the majority — does slightly better, because branches that come from genuinely near-uniform distributions disagree with each other and the disagreement is detectable. It also costs five generations, and it is silent exactly where the model is confidently wrong, since five samples from a peaked distribution agree every time.

The neighbouring idea this gets confused with is calibration. A calibrated model is one whose stated or implied confidence matches its empirical accuracy: when it says 70%, it is right about 70% of the time. Calibration is a real and measurable property, it can be improved, and a well-calibrated model is more useful than a poorly calibrated one. It is still not a solution to the problem in this chapter, for two reasons. Token-level probabilities measure confidence in the next token, not in a claim, and the two diverge — the model above was tightly concentrated on the format of a section number while having nothing to say about which section was right. And a calibrated 40% belief expressed as a fluent declarative sentence still reaches the reader as a fact, because the confidence lives in a number the reader never sees while the assertion lives in prose they do.

A related move deserves naming here, because it is the first thing most people try. Asking the model whether it is sure, or asking it to rate its own confidence, produces a number or a sentence generated by the identical process that produced the claim: one more distribution over next tokens, conditioned on a prompt that now contains the words "are you sure". It is not a read-out of an internal state, because there is no internal state about the proposition to read out. What the question does influence reliably is register, since in human writing a challenge is usually followed by hedging or by revision — which is why a model asked whether it is sure will frequently add a qualifier or change its answer, and will do so at about the same rate whether the first answer was right or wrong.

The rule that follows is about evidence. Fluency, specificity, correct formatting, plausible-looking identifiers and an appropriate technical register are all products of pattern-fitting, and pattern-fitting is what the model is best at. They therefore carry no information about whether the content is true, and a reader who treats them as reassurance has the relationship backwards: the most dangerous outputs are the most fluent ones, because fluency is what stops anybody from checking. That rules out an entire family of proposed fixes — anything that treats invented facts as a wording problem, to be solved by asking the model to be careful, to be honest, or to only say things it is sure of. Those interventions operate on the register of the output, and the register was never the failure.

What is left is measurement, and measurement needs categories. The retry answer contained two different failures wearing the same face: a number that contradicted material the system holds, and a citation pointing at a document section that does not exist. Those have different causes, respond to different interventions, and need to be counted separately before any of it can be managed. Separating them is the next chapter's work.