Stop sequences, max tokens, truncation, silent output cutoff

Read these first

• Autoregressive generation

What you will know

• Set a stop sequence and explain what happens to the text that matched it

  The delimiter is consumed rather than returned, so parsers that expect to see it must add it back or use another boundary marker.

• Detect a response truncated by `max_tokens` rather than finished naturally

  A length-limited response can look coherent right up to its final token, making the provider’s finish reason the reliable signal.

• Reproduce a cutoff response that appears complete to a caller and make the caller reject it

  Partial structured output may contain plausible fields or a valid-looking prefix while still omitting required data and downstream work.

• Add a check that distinguishes "finished" from "ran out of room"

  Parsing success is insufficient because a response can be syntactically acceptable yet stopped for a limit or another non-natural termination reason.

• Recount a bug you have caused or seen caused by an unchecked finish reason

  A concrete incident should identify the missing check, the misleading symptom, and the failure that reached a user or another system.

Chapters

1. 1

   Stop reasons and the four endings

   Generation ends when the model emits its end-of-sequence token, when a stop sequence you supplied matches, when the token cap is consumed, or when the provider terminates the request. Only the first is the model deciding it is done, and the provider reports which one happened in a field beside the text rather than inside it. This chapter defines that field, works through all four endings and what each looks like on the wire, and shows why the two that are most often forgotten are the two whose output reads like a finished answer.

   Read chapter
2. 2

   Stop sequences and the consumed delimiter

   A stop sequence is a string you nominate at request time, and when the model produces it the serving layer cuts immediately before it, so the delimiter itself never reaches you. Parsers written against the full grammar then fail on well-formed output. This chapter covers where the cut happens, how the matcher behaves during streaming, why consuming the delimiter is the sensible default given what the parameter was built for, and the two repairs that work against the one that only looks like it does.

   Read chapter
3. 3

   max_tokens and length instructions

   max_tokens is a counter in the serving layer, not an instruction: it never enters the prompt, never reaches the model, and halts the decode loop mid-word if that is where the count runs out. A length instruction in the prompt is the only one of the two mechanisms the model can read. This chapter separates their properties, gives a procedure for sizing a cap from logged output lengths, and covers what thinking tokens do to a cap that used to be comfortable.

   Read chapter
4. 4

   Detecting truncation from the finish reason

   A length-limited response stays fluent and grammatical right up to its final token, so reading it — or eyeballing samples in review — will not reliably separate finished from ran out of room. The provider reports why generation stopped on every response, and that field is the only place the statement is made. This chapter covers why fluency is no signal, how to write the check as an allowlist rather than a denylist, where the field hides in a streaming response, and what the check does not tell you.

   Read chapter
5. 5

   Truncated JSON and lenient parsers

   Cut a generation mid-stream and what you normally hold is unbalanced JSON that a strict parser rejects outright, which is the loud and easy case. The quiet case requires something that closes the object for you: partial-JSON parsing, a repair pass, or constrained decoding that force-closes. This chapter explains why strict JSON catches truncation with probability approaching one, catalogues the four setups that dismantle that property, and gives an experiment that establishes which case your own code is in.

   Read chapter
6. 6

   Schema validation and truncated output

   An extraction returning {\"id\": \"A-1\", \"total\": 4200, \"notes\": \"\"} passes structural validation, carries every required field, and is wrong, because generation was cut before notes had content. A schema describes a document, and whether generation was interrupted is a property of the process that produced it. This chapter covers what a validator does and does not assert, why optional free-text fields are where truncation concentrates, and why field-level rules written to catch it decay within two quarters.

   Read chapter
7. 7

   The completion check at the response boundary

   A response can be syntactically perfect and still have stopped for a non-natural reason, and every layer between the API call and the durable write answers its own question correctly while nobody asks whether generation completed. The check belongs at the boundary where a provider response becomes a domain object, because that is the last place the finish reason exists. This chapter covers where to put it, how to test it, the metric that belongs beside it, and why the layers above and below cannot host it.

   Read chapter
8. 8

   Newline stop sequences and answer length

   Using a blank line as a stop sequence to keep answers short severs every answer that legitimately needs two paragraphs, and swapping it for a tight token cap only moves the cut from paragraph one to token 150. Length is controlled by the instruction and by the task, with stops and caps acting as bounds sized above a measured tail. This chapter works through both configurations against the same traffic, gives the one to ship, and covers where a newline terminator is genuinely the right choice.

   Read chapter