Chat templates and the generation prompt

Every instruct checkpoint ships with a chat template, and most people never look at it because the layer above them applies it invisibly. It is a string in the Jinja templating language, stored in the tokenizer configuration, that defines how a list of message objects becomes one flat sequence of tokens. It is not documentation of a convention; it is the executable definition of the convention, and it is the same code path the model's own post-training data was generated with.

This chapter renders one and reads what comes out, in three forms: the string, the integer token IDs, and the Jinja source itself. Four things in that output do real work — a beginning-of-sequence marker, a pair of tokens that bracket each role name, an end-of-turn marker, and the trailing generation prompt — and each one has a specific failure that follows from omitting it. All four are easier to understand as integers than as text, because what makes a role label authoritative is that it is not spelled out of the ordinary vocabulary at all.

What the template emits#

Rendering takes three lines and no model.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

messages = [
    {"role": "system",    "content": "You are a support agent. Be brief."},
    {"role": "user",      "content": "My order hasn't arrived."},
    {"role": "assistant", "content": "Can you give me the order number?"},
    {"role": "user",      "content": "GB-4471."},
]

text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(repr(text))

The string that comes out, with the escapes expanded so it can be read:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 08 Aug 2026

You are a support agent. Be brief.<|eot_id|><|start_header_id|>user<|end_header_id|>

My order hasn't arrived.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Can you give me the order number?<|eot_id|><|start_header_id|>user<|end_header_id|>

GB-4471.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Two things there are worth pausing on. The first is the date preamble, which nobody asked for: Llama 3.1's template injects a knowledge-cutoff line and today's date into the system block unconditionally, so a prompt you believed was byte-identical between two deployments will differ if they render on different days, and a golden-file test over the rendered string will fail every midnight. The second is the last line. The string ends after an assistant header with no content, which is the mechanism by which the model is told to speak.

Tokenising the same messages shows what the model actually receives.

ids = tok.apply_chat_template(messages, add_generation_prompt=True)
print(len(ids), ids[:8], ids[-4:])
# 96 [128000, 128006, 9125, 128007, 271, 38766, 1303, 33025] [128006, 78191, 128007, 271]

The interesting integers are the ones above 128,000. Llama 3's vocabulary has 128,000 ordinary entries and then a block of reserved control IDs on top, and those are the markers. 128000 is <|begin_of_text|>, the beginning-of-sequence or BOS marker that says a document starts here. 128006 and 128007 bracket a role name — 9125 is system, 882 is user, 78191 is assistant — and 271 is the double newline that follows the header. 128009 is <|eot_id|>, end of turn, which closes each message. The final four IDs are the generation prompt: start-header, assistant, end-header, newline, and then nothing.

The belief this displaces is that the model receives a conversation, with the roles as separate fields, and reads your question out of the user field. It never does. It receives one flat sequence of integers in which a marker happened to appear, and it extends that sequence. Nothing inside the model checks who wrote which part, because there is nothing to check with — a role is not metadata attached to a span of text, it is a token sitting in the same list as the words of the sentence. Everything the template buys is bought by making that list resemble the lists the checkpoint was trained on.

The second row is the one that explains why a hand-rolled serialisation stays fluent while going off-task. Writing Assistant: into a string produces three ordinary tokens — the same tokens you would get in a novel, or in a Reddit thread, or in the middle of a sentence about assistants. Ordinary tokens carry no authority. The model has been trained on hundreds of thousands of examples where an answer follows the control sequence 128006, 78191, 128007, 271 and on essentially none where an answer follows the literal characters Assistant: in a document that also contains User: labels. Given the wrong sequence it falls back on what pretraining gave it, which is to continue plausibly, and plausible continuation of an interview transcript is more interview.

"Unforgeable by ordinary text" is the strongest claim in that table, and it is worth seeing where the strength comes from. A tokenizer converts text to integers by repeatedly merging adjacent pieces according to a table of merges fixed when it was built, and every integer reachable that way comes from the vocabulary those merges were learned over. Llama 3's runs to a hundred and twenty-eight thousand entries and stops. The marker IDs live above that line, in a block bolted on afterwards, and no sequence of merges over ordinary characters arrives there. That is the whole of the guarantee, and it is a lexical fact rather than a policy anyone enforces.

The claim also has a condition, and stating it here stops the reassurance being carried further than it goes. A second matcher runs over the raw string before the merges do, looking for registered marker strings anywhere in it, and it is switched on by default — because that matcher is precisely how the template's own <|eot_id|> becomes 128009 instead of seven ordinary characters. So the guarantee holds against any text that reaches the merge path and does not hold against text that passes the matcher first. Which of the two a message body meets is decided by your own code, and chapter 4 is about deciding it deliberately.

The generation prompt is a request, not a formality#

add_generation_prompt=True is the flag most easily got wrong, because leaving it off produces something that still looks correct. Render the same four messages with add_generation_prompt=False and the string ends at GB-4471.<|eot_id|> — a complete, well-formed conversation with no open turn. Feed that to the model and the highest-probability continuation is a new <|start_header_id|>, and the role it puts inside is not reliably assistant. On the four-message example above, sampling at temperature 0.7 produced a fresh user turn on 6 of 20 attempts, each one inventing a customer complaint that had never been made.

Why four tokens do that much work follows from the training data, one step at a time. Post-training examples are rendered conversations, and in every one of them the tokens immediately following 128006, 78191, 128007, 271 are the assistant's reply. In every one of them, the tokens immediately following an <|eot_id|> are a fresh header. The model scores the next token given everything before it, so those two regularities become two different predictions about two different sequence states. A string ending on an end-of-turn marker leaves the sequence in the state whose observed continuation was a header, and the role it fills that header with is whichever role tended to speak next in the data. A string ending on an assistant header leaves it in the state whose observed continuation was an answer. The generation prompt is not an instruction to speak. It is the position from which speaking is the ordinary thing to do next.

The flag has a second, less obvious consequence during training. When you build supervised fine-tuning data — the collection of example conversations used to train a checkpoint further on your own task — you render without the generation prompt and with the assistant's real reply, then mask the loss over everything up to the assistant header so the model is only scored on its own tokens. Render training data with the generation prompt included and you have inserted a duplicate header into every example; render inference prompts without it and the model has no turn to take. The same function, two call sites, opposite arguments, and both mistakes produce a model that trains without error and behaves strangely afterwards.

The masking half of that is where the reason becomes visible. Training nudges the weights at every position where the model's top-scoring token differs from the real one, and a rendered conversation holds the user's words alongside the assistant's. Score the model at the user's positions and you are training it to write the user's turns as well, which is exactly how a fine-tuned checkpoint picks up the habit of composing the customer's next message before answering it. The mask confines the nudging to the spans the assistant produced, and the assistant header is how the code that builds the mask locates where each of those spans begins. Anything that moves a header moves a boundary, and a mask has no way of being wrong out loud.

The template is code, and it is readable#

Because the template is Jinja stored in tokenizer_config.json, it can be printed and read, and reading it answers questions that no amount of experimentation will.

print(tok.chat_template[:400])

{{- bos_token }}
{%- if custom_tools is defined %}{%- set tools = custom_tools %}{%- endif %}
{%- if not date_string is defined %}
    {%- set date_string = "26 Jul 2024" %}
{%- endif %}
{%- if messages[0]['role'] == 'system' %}
    {%- set system_message = messages[0]['content']|trim %}
    {%- set messages = messages[1:] %}
{%- else %}
    {%- set system_message = "" %}
{%- endif %}

Three behaviours are visible in those twelve lines. The date string has a hard-coded default that is used when the caller does not pass one, so a rendered prompt can silently claim a date eighteen months in the past. A system message is recognised only in position zero, so a system message placed second in the list is rendered as an ordinary turn with the role name system inside a header — legal, well-formed, and not what the model was trained on. And custom_tools is a template variable rather than a message role, which is how tool definitions — descriptions of functions the model may ask the application to run on its behalf — are injected for models that support them; a stack that serialises tools by hand into the system prompt is producing a different string from the one the tool-calling behaviour was trained against.

The stopping side of the contract lives in a different file. generation_config.json for Llama 3.1 Instruct lists "eos_token_id": [128001, 128008, 128009] — the end-of-sequence or EOS IDs at which a serving stack is meant to stop generating, here end-of-text, end-of-message and end-of-turn — while the base checkpoint lists only 128001. A serving stack that reads the wrong one, or a client that overrides stop with its own guess, will generate straight through the end-of-turn marker into the next header, producing a reply followed by a fabricated user question followed by another reply. That symptom is often misread as a template problem when it is a stop-token problem, and the two are distinguishable by where the extra text begins: after a well-formed <|eot_id|> it is stopping, and before any marker at all it is framing.

That the two ends of a turn live in two different files is itself worth reading as a statement about where responsibility sits. The template decides what goes into the sequence; the generation config decides what takes the sequence out of the loop. No single component owns both. A checkpoint that has been trained to make <|eot_id|> the likeliest token at the end of an answer will still run on forever if the server was never told that ID terminates generation, because the model's half of the contract is to make the token probable and the server's half is to notice it. Two halves, two files, and one symptom that looks the same from outside.

Three families, three alphabets#

None of these markers are standard. The concept is shared and the bytes are not, so a serialiser written against one family is silently wrong against another.

# ChatML — Qwen, and many fine-tunes of other bases
<|im_start|>system
You are a support agent. Be brief.<|im_end|>
<|im_start|>user
My order hasn't arrived.<|im_end|>
<|im_start|>assistant

# Mistral / Mixtral instruct — no role names at all
<s>[INST] You are a support agent. Be brief.

My order hasn't arrived. [/INST]

# Gemma — a different word for the same idea
<start_of_turn>user
My order hasn't arrived.<end_of_turn>
<start_of_turn>model

ChatML, the first of those, is the format OpenAI published for its own chat models, and enough community fine-tunes copied it that a serving stack with no template of its own tends to fall back on it. The Mistral format is instructive because it has no system role at all. Its template folds system content into the first user turn, so code that assumes a system message is preserved verbatim will find it merged, and code that sends two consecutive user messages will find that some versions of the template raise an exception rather than render them. Gemma calls the assistant model, which means a message dictionary carrying "role": "assistant" fails template validation outright. These are separate formats that happen to encode a similar idea, and the only reliable way to produce the right one is to ask the tokenizer that shipped with the weights.

None of this is standardised, for a reason worth understanding rather than resenting. A marker means something only because one checkpoint's post-training was carried out with it, and each team ran that post-training independently, before any of it had a shared name. There is also nothing to standardise onto after the fact. The behaviour was learned against those specific IDs, so adopting another family's spelling would mean training the checkpoint again, not editing a config. The idea travels between families and the bytes cannot, which is why "it uses ChatML" is a claim about a particular checkpoint rather than about an architecture, and why the architecture name in a repository tells you nothing about which alphabet is in force.

Pinning the template so it cannot drift#

Family is not inherited reliably, which is what makes guessing dangerous even when you know the architecture. A fine-tune built on a Llama base by a team that preferred ChatML will ship ChatML markers added to the vocabulary as new tokens, so the checkpoint is a Llama by architecture and a ChatML model by serialisation. The only source that is right by construction is the tokenizer directory that came with the weights, which is an argument for pinning the model and its tokenizer to the same revision and for keeping a test that fails loudly when the rendering changes:

def test_prompt_shape():
    text = tok.apply_chat_template(FIXED_MESSAGES, tokenize=False, add_generation_prompt=True)
    assert text.endswith("<|start_header_id|>assistant<|end_header_id|>\n\n")
    assert text.count("<|begin_of_text|>") == 1

Asserting on the ending and on the marker counts rather than on the whole string keeps the test stable across the date line while still catching a family change, a missing generation prompt, and the duplication that a second rendering would introduce.

Correct serialisation is a floor rather than a ceiling. It gets your text into the shape the model was trained on, and after that the answer can still be wrong, badly reasoned or hallucinated, because framing has nothing to do with knowledge. The template is also not the system prompt: the template decides how a system message is wrapped, and what you put inside it is a separate decision that the template will faithfully render whether it is good or terrible. It is not a security boundary either. The rendered string is one flat sequence of tokens, and the control markers are only special because the tokenizer maps particular byte sequences to reserved IDs. Nothing at the model level distinguishes an <|eot_id|> that the template emitted from one that arrived inside a message body.

The template is the executable definition of what a turn is for a given checkpoint, and the four markers it emits are the difference between a document the model continues and a conversation it answers in. Reading its output as integers rather than as text is what makes that concrete: a role label backed by reserved IDs cannot be imitated by ordinary vocabulary, and an assistant header with nothing after it is a request for the next token rather than a formatting flourish.

Rendering the template yourself is not always necessary and is sometimes actively harmful, because you are frequently not the only layer doing it. The three-line apply_chat_template call above is exactly what a chat endpoint runs on your behalf, server-side, from the messages you post. Render it yourself and then post the result as a message, and somebody renders it again. Working out which layer owns the rendering on each endpoint you call is the difference between a prompt that arrives templated once and one that arrives wrapped twice.