Per-model instruction authority

Every element you put in a prompt — a delimiter, a heading, a worked example, a sentence about output format, a system turn — arrives at the model as tokens with no intrinsic force. Whatever force it has comes from having seen, during instruction tuning and preference optimisation, many examples where text of that shape in that position was followed by a particular kind of continuation. Those are the two post-training stages named in the previous chapter, the first teaching from demonstrations and the second from comparisons between candidate responses, and each lab runs them on its own data with its own conventions about how prompts are laid out.

The consequence is that the same sentence carries a different amount of weight depending on which family reads it. That is a normal, expected property of a port, and the reason it needs a name is that the alternative description — obedience — implies a single scalar on which models can be ranked. There is no such scalar. A model can assign less authority to an output-format sentence and more to a tool schema than the model you came from, which is a difference in where the weight sits rather than a difference in how well it follows instructions.

What a low-authority element looks like when it fails#

Take the sentence most extraction prompts end on:

Return only the JSON object, no other text.

On a family that treats that as a hard constraint on the bytes it emits, every response begins with {. On a family that treats it as a description of what the important part of the answer is, some fraction of responses open with a sentence of preamble:

Here is the extracted invoice data:

followed by a blank line and a fenced json block holding the answer:

{"invoice_number": "INV-2024-8831", "total": 1842.50, "currency": "EUR"}

The JSON inside the fence is correct. Every field is right, the types are right, and a parser that stripped the fence and the preamble would score the second model identically to the first on extraction accuracy. What differs is entirely the authority the sentence carries, and the failure mode is one shape repeated rather than a scattering of malformed output — which is the distinction that decides what the fix has to be.

Authority is assigned in post-training, not in the sentence#

This is not a subtle effect at the margins. Some families were tuned heavily on XML-ish tags and treat <instructions>...</instructions> as a strong structural boundary; others were tuned on Markdown and treat ## Instructions the same way, while reading a stray XML tag as ordinary text. A worked example carries different force again: on a model where few-shot examples were a dominant part of the tuning mix, a single demonstration of the exact output format outranks three sentences describing it, and on a model tuned to prioritise the most recent explicit instruction, the sentence wins and the example is treated as illustrative.

Underneath all that variation is one fact about how a model reads: there is no instruction channel. Your rule, the document, the worked example and the user's text arrive as a single stream of tokens, and nothing in the architecture marks one span as binding and another as data. What makes a span behave as though it were binding is the statistics of what followed spans like it during tuning. Authority, then, is not a permission level an element holds. It is a conditional probability — given text of this shape, in this position, how often did a compliant continuation follow — and the value of that probability was set by somebody else's training mix.

Two consequences follow, and both decide what the rest of this chapter does. The first is that authority cannot be raised from inside the prompt past what the tuning put there, which is why the fix further down is a mechanism rather than a firmer sentence. The second is that authority belongs to an element in a position, not to a model, so a measurement tells you about that element in that slot and stops there. The same words moved from the end of the prompt into the system turn are a different element as far as this probability is concerned, and would have to be measured again.

The last row of that table is the one that becomes an incident rather than a formatting complaint, and the difference is worth converting. A prohibition holding at 96% is being violated four times in a hundred, which is one time in twenty-five. The other four rows describe output you can inspect: a bleed across a boundary, a wrong format, a preamble. Each of those announces itself to a parser. A violated prohibition produces output that is well formed, plausible and against the rule, so nothing downstream flags it, and the rate is the only thing that tells you it is happening.

Much of this is documented, which surprises people who expect model behaviour to be folklore passed between teams. Each lab publishes prompting guidance describing the conventions its own tuning used, and the guidance is specific enough to act on: which markup to prefer, whether an example or a sentence wins when the two conflict, what the system turn is meant to carry. Anthropic's Prompting best practices: model-specific guidance and migration is written for the case where a prompt is moving onto a new generation, and Model-Specific Prompting: How Claude, GPT, and Gemini Differ sets the three families' formatting conventions beside each other. Reading the target family's guidance before running the port narrows the search considerably, because a divergence that matches something the vendor already describes is a known convention rather than a discovery, and the fix for it is usually in the same document.

Recording the divergence before explaining it#

The discipline that keeps a port from turning into an argument about which model is better is to write the divergence down as a fact, in a form specific enough that someone else can reproduce it, before anyone offers a theory about why it happens. One divergence per entry, with the two measured numbers and the exact prompt fragment involved.

That ordering is not politeness, and the reason is mechanical. A theory offered before the record decides what gets recorded. If the working explanation is that model B is weaker, the entry somebody writes holds the failure rate and nothing else, because the tool-call rate and the abstention count from the same run are not evidence for that theory and so do not look like data worth keeping. Written in the other order, the entry holds a fragment, a set hash, two rates and the accuracy on both sides, and it remains usable by a reader who arrives six months later with a different theory. That is the only property that makes a register worth the trouble of keeping one.

# ports/invoice-extraction/divergences.yaml
- id: DIV-001
  element: "trailing output-format sentence"
  fragment: "Return only the JSON object, no other text."
  set: evalsets/invoices-500.json@a91f3c7d02b1
  model_a: {id: "family-a-v3", first_char_is_brace: 1.000, n: 500}
  model_b: {id: "family-b-v2", first_char_is_brace: 0.940, n: 500}
  failure_mode: "fenced code block preceded by one sentence of preamble"
  extraction_accuracy: {model_a: 0.962, model_b: 0.964}
  status: open

Two things about that record earn their place. The extraction_accuracy line sits next to the failure so that nobody reading the entry in three months mistakes it for evidence that model B is weaker overall — the accuracy is a fraction of a point higher. And failure_mode names the shape rather than the count, because the shape is what determines the fix. Six per cent of responses arriving as invalid JSON in six different ways would be a different problem from six per cent arriving as the same fenced block every time.

The measurement setup from the previous chapter is what makes the entry trustworthy: the same frozen set on both sides, the same decoding intent, and enough repeats that 0.940 is a number with a spread behind it rather than a single sample.

Patching where the authority is, rather than raising your voice#

Once the divergence is recorded, the fix is chosen for the model that has the problem, and the first thing worth trying is almost never a stronger sentence. Wording is the control with the lowest authority available on this axis, which is precisely what the measurement demonstrated. What tends to work is moving the constraint from the instruction channel onto a mechanism the model cannot route around.

The pull toward a firmer sentence is worth naming, because it is close to universal and because experience reinforces it rather than correcting it. It is tempting for two reasons. It is what works with a person. And it does sometimes work here — capitalising the sentence, or repeating it at the end, shifts the probability enough to clear the marginal cases, and the responses that flip get read as the model having finally understood. What happened is that a weak channel was pushed slightly harder. The measurement has already established that this element carries low authority on this family, so escalating inside it is escalating in the one place the evidence says is weak. The few points it buys are held by nothing, and the next version of the model can take them back without any notice that it has.

For the fenced-JSON case, a provider that supports prefilling the assistant turn — starting the model's response for it and letting it continue — removes the failure by construction:

resp = client.messages.create(
    model="family-b-v2",
    max_tokens=512,
    messages=[
        {"role": "user", "content": render_extraction_prompt(document)},
        {"role": "assistant", "content": "{"},   # prefill: the reply already begins
    ],
    stop_sequences=["\n\n"],
)
payload = "{" + resp.content[0].text

The preamble is no longer expressible, because the response has already started with an opening brace and there is nowhere to put "Here is the extracted invoice data:".

That holds because of how generation is defined, not because of anything the model was told. The model is handed a token sequence and asked for the next token, over and over. Prefilling places { inside the assistant turn before generation begins, so the first token the model chooses is already conditioned on a reply that has started with a brace. The preamble is not being declined; the positions it would have occupied are spent, and there is no operation available to the model that reaches backwards and inserts them. That is exactly the line between a constraint the model can route around and one it cannot — the first competes with everything else in the token stream, the second changes what the token stream is.

Two details in that snippet are easy to read past and both are load-bearing. payload = "{" + resp.content[0].text puts the brace back on because the API returns only the tokens the model generated, and the prefill was input rather than output, so the response arrives one character short. And stop_sequences=["\n\n"] is doing a separate job. Prefill guarantees the response opens with a brace and guarantees nothing about how it closes, so the claim to make for it is narrow: it removes this failure shape, it does not make the output valid JSON, and the parse still has to be attempted and can still fail.

Prefill is one mechanism among several, and which ones exist is a property of the provider rather than of your prompt. Some offer schema-constrained decoding, where the sampler is restricted to tokens that keep the output a valid instance of a schema you supply. Some offer a JSON mode that guarantees parseable output with no control over its shape. Some offer stop sequences that let you cut a response at a known boundary. The next chapter but one works through what these guarantees actually differ in, because treating a constrained decoder and a firm sentence as interchangeable is a distinct mistake with worse consequences than the one here.

Where no mechanism exists, the patch is per-model wording, and it should be small and it should be attributed to the divergence that motivated it. Adding "Begin your response with the character {" to model B's variant and nothing to model A's is a legitimate fix. Adding it to a single shared prompt that both models see is how a prompt accumulates instructions nobody can explain, and keeping those variants from drifting into three unmaintainable copies is its own chapter later in this topic.

The read that does not follow from the data#

None of this says the two models are equivalent, and it would be a strange overcorrection to conclude that every divergence is cosmetic. Some are not: a model that violates a prohibition 4% of the time is genuinely worse for a product that depends on that prohibition, whatever it does elsewhere, and no amount of "authority is per-model" makes that acceptable. The claim is narrower — that a difference measured on one axis is evidence about that axis and nothing else, and that "worse at following instructions" is a claim about a general capability which a single formatting divergence does not support.

It also does not say the divergence is always cheap to fix. The prefill cost one line of code because the provider offered prefill. A provider that offers neither prefill nor constrained decoding leaves you with wording, retries and a repair step, which is real engineering work with a real latency cost, and that cost belongs in the migration's arithmetic rather than being waved away as a formatting detail.

The register is what keeps both of those honest over time. An entry with two measured numbers, a frozen set hash and a named failure shape can be re-run when the provider ships a new version, and dropped when the divergence stops reproducing. A verdict about a model cannot be re-run at all, which is why it outlives the evidence that produced it and turns into the thing everyone on the team believes about a provider they last measured a year ago.

The invoice case was tidy because the divergence was in an element whose authority you can substitute for. The harder version is a divergence in an element you cannot: a rule that sits in the system message, obeyed on one family and violated on another, where the fix people reach for is to say it louder. That is the next case, and the reason it behaves differently from its neighbours has to do with a piece of machinery between your message list and the model that most people never look at.