Stop reasons and truncation

The stop reason is the field on a chat response that says why generation ended: the model finished, it hit a stop sequence you supplied, it ran into the output cap, it wants to call a tool, or a content filter intervened. It is present on every response from both major providers, it costs nothing to log, and it is the single most useful field for distinguishing a problem in your plumbing from a difference in the model's behaviour.

That distinction is what this chapter is for. Shorter answers after a provider migration, with a few percent ending mid-clause, is one of the most common post-migration complaints and one of the most commonly misdiagnosed, because the symptom is indistinguishable by eye from a model that follows a length instruction less faithfully. It is not indistinguishable by measurement. The chapter covers where the field lives on each provider and where the vocabularies fail to line up, how to read it over a population of responses rather than one at a time, and the three mechanisms that put a cap in a request other than the one in your configuration file.

The vocabulary, and where the field lives#

The stop reason sits in a different place and speaks a different vocabulary on each provider, which is one reason it gets skipped.

The second column collapses two distinct endings into stop, so a response that hit your stop sequence and one that finished naturally are indistinguishable without also inspecting the text — a good example of a difference that looks like a rename and is not one. The row that matters for truncation is the third: a cap-truncated response is labelled, unambiguously, on both sides.

Reading the field before forming a theory#

The field is worth reading over a population rather than on a single response, because what identifies a cap is the distribution rather than any one value.

from collections import Counter

reasons = Counter(r["stop_reason"] for r in sampled)          # 1,200 responses
print(reasons)
# Counter({'end_turn': 1151, 'max_tokens': 49})

caps = Counter(r["usage"]["output_tokens"] for r in sampled if r["stop_reason"] == "max_tokens")
print(caps.most_common(3))
# [(1024, 2), (512, 47)]

Two facts fall out of those six lines. Forty-nine of twelve hundred responses were cut off rather than finished, which is a plumbing question and not a prompting one. And forty-seven of those forty-nine stopped at exactly 512 output tokens, when the configuration said 1024. The two counts in the first line add up to the sample, so nothing else happened: every response either finished or was cut, and the cut ones are a shade over four in a hundred — close enough to the proportion a human reviewer noticed by eye that the two observations are almost certainly the same population seen twice.

A distribution with a spike at a single value is the signature of a limit rather than of a model choice. A model that ends early because it has nothing more to say produces a smooth spread of lengths; a cap produces a wall, with a pile of responses at one number and nothing above it. Two truncations at 1024 in the sample above are the genuinely long-answer case, and they were present before anything changed.

That signature is forced by the mechanism rather than being a rule of thumb, which is what makes it safe to reason from. A cap is a counter held by the loop that drives generation: the model produces a token, the loop appends it and increments, and when the count reaches the ceiling the loop stops calling the model. The stopping condition has nothing to do with the text, so every generation that would have run long stops at the same number, and nothing can land above it. A natural ending is decided by the model instead — it emits the token that closes its turn when the answer is finished — and where that falls depends on the question, the ticket, the phrasing. Different content, different lengths, no reason for any two to coincide.

So the two hypotheses make different predictions, and the histogram distinguishes them without anyone having to judge the writing. If lengths cluster hard at one value with a cliff above it, a counter did that. If they spread, the model did. This is also why the two responses at 1024 are informative rather than noise: they are the shape a cap makes at the value you actually configured, sitting in the same sample as the shape a different cap made at a value you did not.

The three mechanisms that produce this shape#

Once you know a cap was hit and the cap was not the number in your configuration, there are three places the number can have come from, and it is worth checking all three rather than stopping at the first.

The first is a default filling in for a value you stopped sending. max_tokens is required on one provider's API and optional on the other, so a migration that removed the field to satisfy a type checker, or an adapter that only forwards parameters both sides accept, replaces your explicit ceiling with the provider's implicit one. The implicit ones are not generous and are not stable across model families. The variant worth knowing about is reasoning models, where the cap covers the model's internal reasoning tokens as well as the visible answer: a request with a 512-token budget can spend 512 tokens reasoning and return an empty string with a length-based stop reason, which reads in production as the model refusing to answer.

That outcome follows from the previous paragraph's counter and needs no extra rule. The loop counts every token the model emits, and a reasoning model emits its reasoning first and its answer afterwards. The counter cannot distinguish them, because to the loop they are both tokens produced by the same call. So a budget consumed during the reasoning phase leaves nothing for the phase that would have produced text, and the reply comes back empty — not blank because anything failed, but blank because the loop stopped before the visible part began. Anyone sizing a cap for one of these models is sizing a budget for two consumers, only one of which they can read.

The second is the tokenizer, the component that splits text into the units the model actually consumes and that your bill and your caps are counted in. Each provider ships its own vocabulary of those units, so two providers count the same English text differently — a 10% difference on ordinary prose is unremarkable, and on text with code, JSON, or non-Latin script the gap widens considerably. A summary that runs to 3,000 characters is around 640 tokens under one vocabulary and around 710 under another, so a cap of 700 is comfortable on one provider and truncating on the other for the exact same output. Nothing about this is visible in your configuration; both sides say 700.

Where that divergence comes from is worth knowing, because it tells you which of your traffic will diverge most. A vocabulary is built by scanning a body of text and repeatedly merging the character sequences that occur together most often, until a fixed number of entries has been created. Two providers scan different bodies of text and stop at different sizes, so they end up with different merges — one may hold _configuration as a single entry where the other spells it out of three pieces. Divide the two counts above into the character length and both come out between four and five characters per token, which is the entire effect: a small difference in average coverage, applied across every token in the output.

Small differences in an average are exactly the kind that survive testing and fail in production, because the average is taken over content. Prose is what both vocabularies were mostly built from, so prose is where they agree best. Anything under-represented in that material — identifiers, punctuation-dense JSON, a script written in characters the vocabulary covers thinly — is assembled from more, smaller pieces, and by a different amount on each side. A cap validated on English tickets is therefore not a cap that has been validated, and the traffic that breaks it is predictable in advance rather than a surprise.

The third is your own code. Wrappers accumulate defensive clamping, usually written the day someone sent a max_tokens larger than a model allowed and got a 400.

MODEL_MAX_OUTPUT = {
    "claude-3-5-sonnet-20240620": 8192,
    "gpt-4o": 16384,
}

def clamp(model: str, requested: int) -> int:
    return min(requested, MODEL_MAX_OUTPUT.get(model, 512))   # the bug is the default

The registry is a hardcoded dictionary, a migration introduces a model identifier that is not in it, and the fallback quietly becomes the cap for every request on the new path. A clamp with a conservative fallback is a reasonable idea implemented in a way that fails silently, and the fix is to raise rather than to substitute when the model is unknown, so that an unrecognised identifier is a deploy-time error instead of a two-week quality regression.

Comparing the tokenizer mechanism fairly needs a little care, because you cannot count one provider's tokens with the other's vocabulary and learn anything. Each provider counts with its own, so the fair comparison is to send the same fixed text to each and read input_tokens from the response — a one-token generation is enough, and some providers expose a dedicated counting endpoint that does it without generating at all. Doing this on a representative sample of your traffic rather than on one paragraph matters, because the ratio is not constant: prose diverges by a few percent, and text heavy in code, JSON or non-Latin script can diverge by a third, which means a cap that is safe for English tickets is not safe for the same tickets in Japanese.

What the stop reason does not tell you#

It tells you that the model stopped and how it was stopped; it says nothing about whether the answer was any good. A response with a natural stop reason can still be wrong, shallow or off-format, and no amount of stop-reason analysis substitutes for an evaluation of output quality. It also does not distinguish a stop sequence from a natural ending on the provider whose vocabulary collapses those two cases, which occasionally matters when a stop sequence appears inside legitimate content.

It is also worth deflating the premise that usually starts these arguments. "Same tier by the vendor's own marketing" is a claim about price and positioning, not a claim about behaviour on your task, and two models in the same tier will differ in verbosity, formatting habits and how literally they take a length instruction. That is a real and expected difference, it is measurable, and it is addressed by prompt work rather than by rollback. What makes it a distraction is when it is offered as an explanation before anyone has established that the model was permitted to answer.

The ordering is the whole discipline, and it is what makes raising the cap the wrong first move. Raising it is tempting because it often makes the symptom disappear, and a symptom that disappears without a cause identified will return: if the true cause was a clamp's fallback, raising the requested value changes nothing, and if it was the tokenizer, the same cap will truncate again the day someone routes German or Japanese traffic through the endpoint.

So the evidence that ends the debate has three parts, and none of it requires an opinion about either model. The stop reason distribution on the truncated population tells you whether the model was interrupted or finished. The output token histogram tells you whether a limit or behaviour produced the lengths: a spike at one value is a limit, a spread is behaviour. And the effective request body, captured from the wire rather than read out of your configuration, tells you what cap was actually sent. If all three point at plumbing, fix the plumbing and re-run the length comparison, because until then you have no measurement of the model at all. If the stop reasons come back as natural endings and the lengths are simply shorter, you are looking at a genuine behavioural difference, which is addressed by re-prompting, by an explicit length target, or by a different model choice.

Following that order in the case above left one question open. Comparing token counts fairly meant asking each provider to count the same text with its own vocabulary, and the endpoint that does that on one of the two providers was, at the client version pinned in the repository, not exposed by the client at all. It existed on the HTTP API. It was reachable with a header. The choice that follows from that situation — and it is a choice, with three plausible answers and a week to ship — is the subject of the next chapter.