Splitting a reformat from a rule change in a prompt diff

A prompt kept in version control and published as an immutable revision gives an investigation two things to work with: a diff a reviewer can read before the change ships, and an ordered list of revisions with production traffic labelled by which one served it. Both of those degrade in the same way, for the same reason, and the reason is not carelessness. It is that prompts invite bulk editing in a way source code does not.

Somebody sits down to improve a prompt they find hard to read, and improving it means moving things around; their editor rewraps on save because that is how it is configured for every other file in the repository; the placeholder renames are a tidiness they noticed halfway through; and while they are in there, a rule that has been annoying them for a month gets adjusted. Every one of those is a reasonable thing to do in an afternoon. The pull request that results is a single artefact whose parts are visible only to the person who made them, and exhorting people to be more careful does not fix it, because nobody was being careless.

Four changes wearing one diff#

Here is what a 62-line diff on a booking agent's system prompt contained. Two sections swapped order, so the tool list now comes before the tone guidance. Every paragraph was rewrapped from long lines to 80 columns. Two placeholders were renamed, {{customer_name}} to {{customer.name}} and {{res_id}} to {{reservation_id}}. And one instruction changed:

-- If the request is ambiguous about dates, party size or venue, ask one
-  clarifying question before using any booking tool.
+- If the request is ambiguous, make the most reasonable assumption, state the
+  assumption in your reply, and proceed.

Two lines out of sixty-two, and they carry the entire behavioural content of the change. The reason a diff like that ships unexamined is that the rule change is rendered indistinguishable from rewrapping: a reviewer reading a full-file diff is being asked to re-derive the semantics of the whole prompt from scratch, and four minutes is roughly the amount of attention a full-file diff receives regardless of how much is hiding in it.

That last observation is worth taking seriously rather than deploring, because it is why the fix has to be structural. Reviewing a diff means reading the changed lines and deciding whether each is right. When few lines changed, that reading is available and the whole of the reviewer's attention goes on judgement. When every line has changed, the reading is not available at all: deciding whether the file is right now means deriving the meaning of the entire prompt from first principles, which is a far larger job than the pull request appeared to ask for and one whose size is invisible from its title. What follows is not a careless reviewer skimming. It is a conscientious reviewer exhausting a budget nobody told them they were spending, and approving on the strength of the parts they did read. The remedy is to make the job smaller rather than to ask for more diligence, because diligence was never the variable.

Two revisions, one decision each#

The reviewer's response is to send it back as two pull requests, each producing its own revision under the publishing rules from earlier in this topic.

The first carries the reorder, the rewrap and the placeholder renames, and it claims to leave behaviour alone. It is large and boring, and both of those are fine, because a reviewer of that pull request has exactly one question to answer: does anything in here alter meaning? The renames need the template code checked in the same change, since {{customer_name}} and {{customer.name}} resolve differently, and the rendering rules from topic 4.9, on interpolating values into a template, decide what happens when a variable is missing.

The second carries the rule and nothing else. Its diff is the two lines above, and the review conversation on it is about the actual decision: whether an agent that guesses and announces its guess produces better outcomes than one that asks, and what happens when the assumption is wrong on a non-refundable booking. That conversation is worth having and cannot happen inside a 62-line diff, because nobody knows the conversation is available.

The value shows up again during a regression hunt weeks later. A pointer history is a sequence of revision ids with timestamps, and request records label every production call with the revision that served it, so the search for the revision where a metric stepped is a bisect — the same binary search over an ordered list that git bisect performs over commits. Replay a few hundred logged conversations against the middle revision, measure the metric, and take the half that contains the step. Over nine revisions, four evaluations locate it, and each costs a couple of minutes and less than a dollar.

Two things make that search work where the one in topic 4.9, on giving a template revision an identity and recording it on every run, did not. The first is arithmetic: halving nine candidates leaves five, then three, then two, then one, so four evaluations are enough, and the count grows with the logarithm of the history rather than with its length — doubling the number of revisions costs one more evaluation. The second is the property a binary search actually requires, which is that the answer at a position is a function of the position. It holds here because each evaluation replays logged conversations against a named revision with every other input held where the request record says it was, so running the same step again tomorrow returns the same verdict. A search whose verdict depends on a live index and a floating model alias has no such property, and converges on an answer that means nothing.

If each revision represents one decision, the answer that search produces is directly actionable, because the revision it lands on is a two-line diff with a review conversation attached. If revisions bundle four changes each, the search converges just as reliably on a revision id and tells you nothing you can use: knowing that the rate fell at pr_a41f0c22 leaves four candidate causes and no way to separate them except by manually reconstructing intermediate versions that never existed. The bisect completes and the investigation restarts from the top.

Software engineering settled this argument for code some time ago, and the tooling records the settlement. Git shipped .git-blame-ignore-revs in version 2.23 precisely because bulk reformatting commits destroy the usefulness of blame, and the convention of keeping a formatting commit separate from a logic commit predates it by decades.

Why blame breaks is worth the step, because the fix only makes sense once the mechanism does. git blame answers one question per line: which commit last touched this line. A reformatting commit touches every line, so once it lands every line answers with the reformat, and the commit that decided what the line actually says sits one hop further back with nothing pointing at it. The information is not gone — it is in the history, and git log will find it — but the tool people reach for first now returns the same useless answer everywhere. .git-blame-ignore-revs deletes nothing. It tells blame to look past the listed commits, so each line answers with the last commit that changed its content rather than its shape. That is the whole of the fix, and it is correct only if the reformat really did leave the content alone, which is the claim the split exists to make checkable.

The same file works on a prompt directory:

# .git-blame-ignore-revs
# Rewrap booking agent prompt to 80 cols (PR #2288, no behaviour change intended)
c7f2a91b3d84e0a1c5f6b2d7e9013a4c8b5d6e2f

With that in place, git blame prompts/booking/system.md on the clarifying-question line points at the two-line revision and its review discussion rather than at a rewrap.

What a machine can classify, and what it cannot#

Reviewer discipline alone is unreliable in the way all unwritten review conventions are unreliable, so it is worth knowing what can be enforced mechanically. A normalisation check can decide whether a diff is plausibly behaviour-neutral by canonicalising both sides and comparing:

# ci/prompt_diff_class.py
import re, sys

RENAMES = {"customer_name": "customer.name", "res_id": "reservation_id"}

def normalise(text: str) -> str:
    for old, new in RENAMES.items():
        text = text.replace("{{" + old + "}}", "{{" + new + "}}")
    text = re.sub(r"\s+", " ", text)               # collapse wrapping and indentation
    blocks = [b.strip() for b in text.split("## ") if b.strip()]
    return "\n".join(sorted(blocks))               # section order made irrelevant

if normalise(before) == normalise(after):
    print("BEHAVIOUR_NEUTRAL: reorder/rewrap/rename only")
else:
    print("BEHAVIOURAL: wording differs after normalisation")
    sys.exit(1)   # requires the `behavioural-change` label and a second approval

Each line of normalise blinds the check to exactly one thing the author claimed was neutral, and reading it that way makes both its power and its limit plain. Applying the rename map means a placeholder rename can no longer register as a difference. Collapsing every run of whitespace to a single space means rewrapping and re-indenting can no longer register either, because a line break was whitespace and is now the same whitespace as a space. Splitting on the heading marker and sorting the pieces means section order can no longer register. What survives all three is wording, which is why a difference after normalisation is a wording difference and nothing else. The limit is that same sentence read backwards: the check is exactly as trustworthy as the claim that those three transformations leave behaviour alone, and the last section of this chapter is about why that claim is weaker for a prompt than for source code.

Run against the composite diff above this exits non-zero, because after collapsing whitespace, applying the rename map and sorting sections, the clarifying-question sentence still differs. Run against the split reformat pull request it passes, and the check becomes the evidence for the claim in the title. The rename map has to be supplied by the author, which is a feature: declaring "I renamed these two placeholders and changed nothing else" is a falsifiable statement that CI then falsifies or does not.

There is a second thing the split makes possible, which is a pull request template that asks the behavioural revision the questions a reviewer cannot answer from the diff alone. Three fields carry most of the value: what behaviour is expected to change, which metric would show it and in which direction, and which revision to pin if it goes wrong. Filling those in would have forced someone to write "clarifying-question rate should fall" before merging, which is the sentence that turns a three-week mystery into a one-line lookup. Asking the same questions of a reformat is pure friction, which is another reason the two want separating: the ceremony attaches to the change that earns it.

The structured-file row is the one teams underrate. Storing the prompt as an ordered list of named blocks — role, tools, tone, escalation — means a section reorder shows up as a list reordering and a wording change shows up inside one block, so the diff is legible without any tooling. What you give up is the ability to write and read the prompt as continuous prose, which matters more than it sounds when the people editing it are not engineers.

"Behaviour-neutral" is a claim about intent#

The word neutral is doing something dangerous in all of this, and prompts differ from code here in a way that has to be stated. Reformatting source code is behaviour-preserving because a compiler defines the semantics and whitespace is outside them. Nothing plays that role for a prompt. The model reads the literal token sequence, so rewrapping changes token boundaries, reordering sections changes what is nearest the instruction that follows, and renaming a placeholder changes the tokens the model sees around the interpolation point. Measured effects from pure reformatting are usually small and are not reliably zero; one team's rewrap of a classification prompt moved macro-F1 — the score that averages each category's balance of precision and recall, so a rare category counts as much as a common one — by 0.4 points on a 2,000-example set, in the direction nobody predicted.

The tokenisation step is the one to make concrete, because a reader picturing the model reading words will not believe any of that. Text reaches a model as a sequence of tokens — pieces roughly the size of word fragments — and the pieces are chosen by matching a fixed vocabulary against the raw characters, newlines included. A line break is a character. Move it, and the fragments on either side of it can be matched differently, so a paragraph rewrapped to a different width is a different sequence of tokens carrying the same words. Whether that matters is an empirical question with a small answer, which is why the measured effect above was a fraction of a point rather than a collapse. What it is not is zero by construction — and zero by construction is exactly what a reformatting commit is claiming when it says the same thing about source code, because there a compiler defines the semantics and has already thrown the whitespace away.

The placeholder rename carries a sharper version of the same risk. Renaming {{res_id}} to {{reservation_id}} leaves the rendered output identical, since the placeholder is substituted before the model sees anything — provided the template code was updated in lockstep. If it was not, the placeholder either renders as literal text or renders empty, depending on the engine's configuration, and the model now receives a system prompt containing the characters {{res_id}} or a sentence with a hole in it. That is the one case in this chapter where the "behaviour-neutral" revision can break production outright, and it is why the rename belongs with the template change in one pull request rather than being split further.

Both of those outcomes are worth a moment, because they fail in opposite directions and neither of them is loud. An engine configured to leave unknown placeholders alone renders the literal characters, so the model is handed a sentence telling it to use a reservation identifier with a piece of template syntax standing where the identifier should be. An engine configured to treat an unknown name as empty renders a sentence with a gap in it, which reads as a slightly odd instruction and is quieter still. Neither raises an exception, neither moves an error rate, and both arrive inside the revision everybody agreed was the safe half of the split. That inversion — the boring pull request being the one that can take production down — is the reason the rename travels with the template change rather than being tidied into a commit of its own.

The practical consequence is that a behaviour-neutral revision still goes through the evaluation gate and still gets its own pointer move, separately and preferably during quiet traffic. Ordering the two revisions matters as well. The reformat goes first and is left to sit for long enough that its metrics are boring — a day is usually plenty on a service with meaningful volume. The behavioural revision then lands against a baseline that already includes the reformat, so if something moves, the reformat is not a live hypothesis. Landing them the other way round, or an hour apart, gives up most of what the split was for.

There is also a cost to the split, and pretending otherwise gets the advice ignored. Two pull requests, two review cycles, two publishes and two pointer moves is real overhead on a change the author experienced as a single afternoon of tidying, and the author is entitled to notice that it is now Thursday. The proportionate version of the rule is that the split is required when a diff crosses a size threshold — anything over about fifteen changed lines, in practice — and that below it, a reviewer who can read the whole diff in one pass does not need the ceremony. The threshold matters less than having one, because the failure it exists to prevent is the 62-line diff approved in four minutes.

None of this says anything about whether a given rule change is a good idea. Suppose the split lands, the two-line revision gets a proper review, two people agree that stating an assumption beats interrupting the user, and the eval suite scores three points better on task completion. Shipping that revision to all traffic on the next pointer move is still a decision made on evidence from a fixed set of examples about an agent with six tools and a long tail of paths that set never touches, and it is the decision the next chapter takes apart.