Diffing token IDs across code paths

Two paths into one model is the normal state of an application rather than an unusual one. A notebook calls /v1/completions with a prompt string somebody has been maintaining for months; the service calls /v1/chat/completions with a list of messages, because that is what its client SDK exposes. Both reach the same process, serving one copy of the same checkpoint with no adapters loaded — an adapter being a small set of extra weights layered onto a checkpoint at serve time to change its behaviour without retraining it. When their outputs diverge, the difference is a string that can be printed.

This chapter is about printing it, in the one form that cannot be argued with. The prompt in your code is what you meant to send, the request body in your logs is what your client constructed, and neither is what the engine tokenised. The integers are, and most serving stacks will hand them over. What follows is the diff, the fallbacks for stacks that will not, a reading of what each shape of divergence means, and the kind of test that confirms a fix on output which is fluent whether it is right or wrong.

"Cannot be argued with" is a claim about where each artifact sits in the pipeline rather than about anybody's honesty. The model is a function of one thing, a list of integers. Your prompt string sits upstream of the template, which may rewrite it. The request body in your logs sits upstream of the server's rendering, which may wrap it. The rendered string sits upstream of the tokenizer, which decides which of its byte sequences become reserved IDs. Each of those is a transformation that can change what the model sees without changing the artifact you were looking at, and the integer list is the only artifact with no transformation left after it. That is what makes a diff of the integers decisive and a diff of anything earlier merely suggestive.

What a chat endpoint actually adds#

The first theory in any channel where this comes up is that the chat endpoint must be injecting a hidden system prompt that degrades quality. It is a reasonable-sounding theory, it is almost never the explanation, and it is expensive because it sends people looking for something invisible. A chat endpoint's job is serialisation rather than content: it takes your messages, runs the tokenizer's chat template over them, and sends the result.

The theory is tempting for a good reason, and naming it is what allows it to be put down rather than re-argued every time. If you assume your string arrived at the model as you wrote it, then something the server added is the only remaining explanation for a behaviour difference, and a hidden instruction is the most vivid thing a server could plausibly add. The assumption is the faulty part. The server does add something, and what it adds is not hidden — it is in the rendered prompt and in the token count, both of which you can print in one call.

Being precise about the exceptions is worth the paragraph, because "nothing" is not quite right. It adds the template's own markers. It adds whatever the template hard-codes, which for Llama 3.1 includes a knowledge-cutoff line and today's date, and for a few community templates includes a default system message that appears when you send none. If you pass tools, it serialises them into the prompt in the format that checkpoint's tool-calling behaviour was trained on, which can be several hundred tokens. Some providers prepend a safety preamble, and they document it. All of those are visible in the rendered prompt or in the token count, which is the point: they are additions you can see, and the way to see them is to look rather than to speculate.

Diffing the two ID lists#

Most serving stacks expose the tokenizer directly. vLLM has a /tokenize endpoint that accepts either form of request and returns exactly what the corresponding generate call would have been given.

import requests

BASE, MODEL = "http://vllm-8b:8000", "meta-llama/Llama-3.1-8B-Instruct"

def ids_for_completion(prompt: str) -> list[int]:
    r = requests.post(f"{BASE}/tokenize", json={"model": MODEL, "prompt": prompt})
    return r.json()["tokens"]

def ids_for_chat(messages: list[dict]) -> list[int]:
    r = requests.post(f"{BASE}/tokenize", json={
        "model": MODEL, "messages": messages, "add_generation_prompt": True})
    return r.json()["tokens"]

a = ids_for_completion(notebook_prompt)     # the path that works
b = ids_for_chat(app_messages)              # the path that does not

i = next((k for k in range(min(len(a), len(b))) if a[k] != b[k]), min(len(a), len(b)))
print(f"lengths {len(a)} vs {len(b)}, first divergence at index {i}")
print("works:  ", a[max(0, i-2):i+6])
print("broken: ", b[max(0, i-2):i+6])
for name, tid in [("BOS", 128000), ("header", 128006), ("eot", 128009)]:
    print(f"{name}: {a.count(tid)} vs {b.count(tid)}")

A doubly-templated request prints this:

lengths 96 vs 106, first divergence at index 2
works:   [128000, 128006, 9125, 128007, 271, 38766, 1303, 33025]
broken:  [128000, 128006, 882, 128007, 271, 128000, 128006, 9125]
BOS: 1 vs 2
header: 4 vs 6
eot: 3 vs 4

Index 2 is the role name inside the first header: 9125 is system on the working path and 882 is user on the broken one, and three tokens later a second 128000 appears where no beginning-of-sequence marker can legitimately be. That is the nesting signature — an already-rendered prompt string passed as the content of a single user message, so the server templated it again. It is the same doubled state chapter 3 described from the outside, now visible as integers rather than inferred from behaviour, and reading it costs one HTTP round trip with no model invoked.

When the stack will not show you integers#

Not every deployment exposes a tokenize route, and a hosted API never will. The fallbacks are ordered by how much they tell you.

If you control the server, request logging is the most direct substitute. vLLM's --log-requests prints the prompt it received for every call, which is the string after templating and before tokenisation, and that is enough to count markers. Text Generation Inference returns the prefill tokens with IDs when you ask for "details": true. An OpenAI-compatible completion endpoint that supports echo will hand your prompt back verbatim, which is less useful than it sounds — it echoes the string you sent, not the one the server rendered — but on a raw completion path those are the same thing, so echo confirms nothing was added in transit.

Why a hosted API withholds a tokenize route is worth knowing before you go looking for one. The rendered prompt is the provider's own serialisation of your messages, and handing it back would publish both the template they use and any preamble they attach, which is the sort of thing providers treat as theirs. What they do publish is a token count. The rest of this section is about how much can be recovered from a single integer when it is the only integer you are given.

If you control nothing, usage.prompt_tokens is still a measurement and a surprisingly sharp one. Render the conversation locally with the tokenizer that matches the model, count the tokens, and compare against what the API reports.

local = len(tok.apply_chat_template(messages, add_generation_prompt=True))
resp  = client.chat.completions.create(model=MODEL, messages=messages, max_tokens=1)
print(local, resp.usage.prompt_tokens, resp.usage.prompt_tokens - local)
# 96 106 10

A difference of a few tokens is the provider's own framing and is expected. A difference equal to the framing cost of an extra full rendering — around ten tokens for a short Llama conversation, more for a long one, and growing with the number of messages rather than staying constant — points at a second application of the template. A difference that scales with the length of your content rather than with the number of turns points somewhere else entirely, usually at a system prompt or a tool schema being injected. The absolute numbers will not match exactly across implementations, so what you are reading is the shape of the discrepancy rather than its value.

A token count carries evidence for the same reason the integers do. Tokenising is deterministic: one byte sequence produces one list of integers and therefore one length, every time it is run. So a gap between the count you computed locally and the count the provider reports is a gap between two byte sequences, and there is no third explanation available. What a count cannot tell you is where in the sequence the difference sits, and that is the whole of what it gives up by being one number instead of a list.

What the shape of the divergence means#

Four shapes cover almost everything, and reading them takes longer to describe than to do.

Four shapes cover it because the space of serialisation outcomes is genuinely small. One operation is in question, running a template over a message list, and only three things can happen to it: it runs zero times, it runs once, or it runs more than once. Running once then splits on which template ran, since a stack without one of its own will supply somebody else's. That gives four states and no fifth. Which is why a diff matching none of these branches is itself a result — it says the problem is not serialisation — and why the last branch is a finding rather than a dead end.

The first branch is the mirror image of the doubling case and is what happens when the working path is the chat endpoint and the broken one is a completion call with a raw string: no control tokens anywhere, and output that continues rather than answers. The third branch is subtler and usually means the two paths loaded different tokenizer revisions, or that a serving flag supplied a --chat-template that does not match the checkpoint. Comparing tok.chat_template between the two environments settles it, as does checking whether the rendered strings differ only in the date line, which is a difference you can ignore.

The second branch has a variant worth recognising on sight, because it looks like nesting and is not. If the two lists are identical except that the broken one carries one extra 128000 at position zero and nothing else differs, the template was applied once and a beginning-of-sequence marker was added on top by a tokenizer call with its defaults left alone. That costs one token and a couple of points of accuracy, it produces no dramatic behaviour, and it is the single most common serialisation defect in code that renders prompts client-side and then posts them to a completion endpoint.

There is an ordering to the checks that saves time. Compare lengths first, since a large discrepancy narrows the field immediately. Compare the first eight IDs next, because framing errors show up at the start and content errors do not. Count the markers third. Only then look at the middle of the sequence, where a divergence usually means the two paths are sending different content — a stale prompt version, a retrieval step that ran in one path and not the other, or a message list assembled in a different order — which is a real bug but a different one, and not a template problem at all.

Why this class of bug needs a diff at all, rather than a careful read of a dozen outputs, follows from what the defect touches. A serialisation error changes which question the model is answering. It leaves untouched the model's command of language, its formatting habits and its willingness to sound certain, because none of those were ever a function of the markers. So every signal a person uses to judge output at a glance — grammar, tone, structure, confidence — is exactly as good after the bug as it was before, and the one signal that moved is the one that requires already knowing the right answer. That is why the evidence here is integers and labelled items rather than samples, and why "it looks fine" is a report about the failure mode rather than about the system.

When the IDs are identical#

Sometimes the diff comes back clean, and that result is worth as much as the other three because it removes an entire category from the search. If both paths send the same integers and the outputs still differ, the cause is downstream of serialisation, and the candidates are short and checkable: a different temperature or top_p between the SDK's defaults and yours, a seed set in one place and not the other, max_tokens truncating one path, a stop sequence configured in the application that cuts the output at a token the notebook happily kept, or a different quantisation or LoRA adapter — LoRA being the usual method for training one — behind a model alias that resolves differently in two environments.

That list is also the limit of what this technique covers. Diffing token IDs tells you what the model was asked. It says nothing about whether the prompt is any good, whether retrieved context was relevant, or whether the model is capable of the task, and a team that has just fixed a template bug should be careful not to conclude that all quality problems have this shape. Most do not.

The reason to check it first is arithmetic rather than prevalence: it takes five minutes, it needs no model call, and it eliminates the most common mechanical explanation for two paths disagreeing. A diff that comes back with doubled markers has told you the cause outright; a diff that comes back clean has told you where not to look, which on a two-day argument about GPUs and batch sizes is worth as much. Either way the next step is a fixed labelled set rather than a reading of twenty outputs, because everything this class of bug produces is fluent.

Exact match is the right metric on that labelled set, and the reason it is right does not generalise to most tasks. A model given the wrong frame is not producing a slightly worse version of the correct answer. It is producing a competent answer to a question about your prompt. There is no partial credit to award because there is no partial overlap: the category is a different category and the object is a different object. Where a metric has to separate "nearly right" from "wrong", exact match is crude and misleading. Where the two states are "answered the task" and "answered something else", it is precisely as sharp as the distinction being measured.

There is a related migration that produces the same symptom — good text, bad scores — with no bug anywhere in the stack. Moving a prompt from an instruct model to a base model to save money changes serialisation and prompting style at once, and it fails in a way that looks identical to a missing template until you understand what the cheaper checkpoint is actually for.