Instruction distance and recency

A request to a model arrives as one flat sequence of tokens: the instructions, the retrieved material and the question, concatenated in whatever order the assembling code chose. Nothing in that format marks one span as more important than another — there is no priority field, no way to declare that a rule outranks a document — so the only lever an author has over the relative force of two pieces of text is the order they are written in. Instruction placement is the deliberate use of that lever: deciding where each rule sits relative to the material it governs and to the request it has to survive to. It is a control worth having because a model's attention over a long input is not spread evenly. Text near the beginning of the sequence, and more strongly text near the end, influences what gets generated more than text buried in between; the pull of the opening is called primacy, the pull of the end is called recency, and the weak zone between them is the lost-in-the-middle behaviour from topic 1.12.

Engineers meet this wherever a prompt has grown long enough to have sections. A retrieval-augmented generation pipeline — RAG, meaning the code searches a corpus for material related to the request and pastes what it finds into the prompt before sending it — puts thousands of tokens nobody wrote by hand into the middle of every request, while the instructions stay exactly where the author left them. Agent loops meet it when tool definitions and a growing transcript push the system block further from the current step on every iteration, and chat products meet it once a conversation runs to fifty turns. The rules are identical in all three cases; what changes is how much text stands between them and the moment the model starts writing.

Distance is a variable in the prompt, not a property of the text#

Topic 1.12 established that a long context degrades performance on the surrounding task and that material in the middle of a long window is attended to more weakly than material at either end. That result was about evidence — a passage that answers a question, moved through a document — and the same geometry applies to instructions, with one difference that matters in production. An instruction has no correct answer to score, so the failure is invisible to any evaluation that only checks whether the response is good. A response that is helpful, accurate, well-formatted and contains a full account number scores as a success on every axis such a team is measuring.

The mechanism is worth stating once, plainly, because the rest of this topic depends on it. The model produces the next token conditioned on everything before it, and the influence of any earlier span on that prediction falls off with distance and with the amount of competing material in between. Thirty-eight thousand tokens of ticket threads are not neutral filler; they are text about the same subject, in the same domain, written in the same register, and they compete directly with an instruction for whatever attention the model allocates while writing the reply. By the time the model reaches the question, the rules are 38,000 tokens of similar-looking material away, and the last thing it read was a customer complaining about a billing error and quoting their own account number in full.

That fall-off is worth taking apart into its links, because each one is checkable and each one is skippable. The model produces one token at a time, and it produces each one by weighing every token already in the sequence and deciding how much each should count towards what comes next. That weighing is a division of a fixed share rather than an independent verdict on each span: the weights assigned across all earlier positions are normalised to sum to one, so every token in the prompt takes some of the available influence away from every other token. Adding thirty-eight thousand tokens of retrieved material therefore does not leave an instruction where it was and put new text beside it. It leaves the instruction competing with thirty-eight thousand more claimants for a quantity that did not grow.

The second link is what those claimants look like, because the share a span receives is not drawn at random. It depends on how relevant the span looks to the token about to be produced, which means filler resembling nothing in the request would cost an instruction very little. Retrieved ticket threads are the opposite of filler. They are text about the same subject, in the same domain, in the same register, and several of them are worked examples of the very task the model is about to perform — a support agent answering a customer. They are the most plausible material in the window to continue from, which is exactly what makes them expensive to have there.

The third link is order, and it is the one the opening of this chapter already supplied. The instruction was read once, tens of thousands of tokens ago; the customer's message quoting an account number in full was read last. Nothing in the format marks the first as a rule and the second as data, because there is no priority field to mark it with. So at the moment of generation the only differences between those two spans are where they sit and how much each looks like the thing to say next. Position is not standing in for importance in this machinery. It is one of the two inputs that produce importance.

A useful way to hold this is that a prompt has a position axis as well as a content axis, and only one of them is usually under review. Teams review prompts the way they review configuration files, where every line has equal force regardless of where it sits, and a rule twelve lines down binds exactly as hard as the first one. That intuition is what makes the failure surprising: the prompt reads correctly, and it is correct, in the sense that a human reading it top to bottom would apply all five rules. The model is not reading it top to bottom.

The belief underneath that habit is worth stating in the form a reader would write it down, because it is the one this chapter has to dislodge: a rule in a prompt is a setting the model applies. It is tempting for good reasons. The prompt is a file, it is versioned, it is reviewed line by line, and every other file in the system that reads like a list of rules is read by something that applies rules in the order they are written. What makes it false is that nothing in a request is a rule to the model. A prompt is evidence, weighed against the other evidence in the same window, inside a prediction about what text comes next. The sentence to be able to write afterwards is not "the model ignored the rule" but "the rule lost".

Which rules survive the distance, and which do not#

The split between rules that survive and rules that fail is not random, and understanding it prevents a wasted round of work. Here are five rules from one support triage assistant, with compliance measured over an audit of 600 production responses — per-rule compliance meaning the share of the responses a rule applied to in which that rule actually held.

The three that survive are all continuously reinforced by the output itself. Once the model has written two sentences in Spanish, the next token is overwhelmingly likely to be Spanish, whether or not it remembers being told; the instruction only has to win once, and then the response's own prefix carries it. The ticket reference sits at the very start of the generation, where the instruction's distance is at its shortest for the whole response. The signature is a strongly conventional block that the model would plausibly emit anyway given the surrounding examples in the retrieved threads.

Reinforcement is the same mechanism from the previous section running in the author's favour, and it is worth walking rather than accepting. Every token the model generates joins the sequence, and the next token is conditioned on it along with everything else. Generated tokens are therefore always the closest material in the window — they sit at the very end, ahead of every document and every instruction. A rule whose effect is visible in the text already written is consequently re-argued at each step by the best-placed evidence available, whatever became of the original instruction. Spanish is the clearest case. After two sentences of Spanish, the question "what language is this reply in" is answered by the reply itself, and the instruction has been made redundant by its own success.

The two that fail are conditional and negative. Nothing in the growing response reinforces them, they bind only on the subset of requests where the condition holds, and one of them asks the model to not do something that the immediately preceding text — the customer's own message, quoting their account number — makes locally probable. Position and conditionality compound: a conditional rule gets no help from the output's momentum, so it depends entirely on the instruction still carrying weight at the moment the condition arises, which may be 200 tokens into a response that began 38,000 tokens after the rule was stated.

Reading the reinforcement mechanism backwards is what turns that observation into a test you can apply to a rule you have not measured yet. A rule leaves no trace in the prefix when it forbids something, because an account number that was never written looks exactly like an account number that was never relevant. It leaves no trace while its condition has not fired, because a rule that has not yet applied has produced no text to be conditioned on. Both shapes strand a rule in the same place: at the moment it finally binds, it is competing on distance alone, against material that has in one of these cases just quoted the very digits the rule is about.

That also explains why rewording produces small, unstable movements. Rewording changes how strongly a rule is expressed at its position; it does not change the position. On a prompt where distance is the dominant term, the achievable gain from wording is a few points, which is inside the noise of a 600-response audit.

That last clause is worth deriving rather than taking on trust, because it is what makes three rounds of rewriting a predictable waste. Compliance for a conditional rule is measured only over the responses the rule applied to, and the abstention rule applied to the cases where retrieval failed — roughly ninety of the six hundred. On a denominator that size, one case flipping moves the rate by more than a point, so a wording change worth three or four points and a wording change worth nothing produce numbers that look the same on the same audit. The account-number rule has a larger denominator and fares a little better, and neither is large enough to resolve the effect a rewrite could plausibly have. Position is the only term in this prompt big enough to see at this sample size, which is a fact about the measurement as much as about the mechanism.

Restate late, keep the full set early#

The standard remedy is to state the critical rules twice: once in the full instruction block at the top, where a human reviewer reads them, and once as a short block placed after the retrieved documents and immediately before the question. The tail copy is not a rewrite of the contract but a reminder of the two or three lines that must survive to generation time, in the position where the least material stands between them and the first output token.

Two questions arrive at this point, and both are answered by the mechanism rather than by practice. The first is why sixty tokens placed after thirty-eight thousand should outweigh the same sixty tokens placed before them. They do not outweigh anything; they are not competing with the same field. Everything standing between the tail copy and the first generated token is the question, so those sixty tokens contend against a few dozen others rather than against tens of thousands, and the fixed share of influence they divide is being split far fewer ways. The second is whether stating a rule twice makes the prompt ambiguous. It does not, because both copies say the same thing; repetition is not the situation where two requirements cannot both hold and one has to lose, which is chapter 3's subject and a different problem entirely.

CRITICAL = """Before answering, apply these:
- Never write a full account number. Use the last four digits only.
- If the documents above do not answer the question, say so and escalate.
  Do not assemble an answer from adjacent material."""

def build_messages(rules: str, docs: list[str], question: str) -> list[dict]:
    """Full rule set stays in `system`; the two critical ones are repeated last."""
    body = "\n\n".join(
        [
            "<documents>",
            *(f"<doc id='{i}'>{d}</doc>" for i, d in enumerate(docs)),
            "</documents>",
            CRITICAL,                      # ~60 tokens, after 38k of documents
            f"<question>{question}</question>",
        ]
    )
    return [{"role": "user", "content": body}]

resp = client.messages.create(
    model=MODEL,
    max_tokens=1200,
    system=[{"type": "text", "text": rules}],   # all five, in review order
    messages=build_messages(rules, docs, question),
)

On the same 600-case audit, re-run with this layout and nothing else changed, the account-number rule went from 41% to 88% and the abstention rule from 33% to 79%. The three rules that were already working stayed within a point of where they were, which is the result you want and is worth checking explicitly rather than assuming — a change that lifts two rules by pushing a third off a cliff is not an improvement, and chapter 3 is entirely about that failure.

The duplication costs about sixty tokens per request and buys two things that are easy to conflate. The tail copy buys compliance. The head copy buys reviewability: it is where a compliance team reads the contract, where a new engineer learns what the system promises, and where the rules stay adjacent to their own comments and their history in version control. A prompt whose only statement of a rule is a terse fragment wedged between a document blob and a question is a prompt nobody can audit, and the pressure to delete the "redundant" top copy in a future cleanup is real. Leave a comment saying the duplication is deliberate.