Prompt rule provenance

Bisection hands you a colliding pair and no ranking, and inspection cannot supply one. Here are three instructions from the support prompt of a logistics company's chat agent, in three different sections, written by three different people.

2.  Begin every reply to a complaint with an apology. Acknowledge the
    customer's frustration before addressing the issue itself.

...

17. Be warm and concise. Do not over-apologise — one acknowledgement is
    enough, and repeated apologies read as insincere padding.

...

31. Do not apologise on behalf of the company for delays or damage caused
    by a third-party carrier. State the facts, state the remedy, and do
    not characterise the cause as our error.

Roughly a third of inbound messages are complaints about a carrier delay, which is the overlap where all three fire at once. Rule 2 says open with an apology. Rule 31 says do not apologise for this. Rule 17 sits in the middle and is not really in dispute with either, though it will get dragged in. Each of the three is clearly written, each is defensible, and a reasonable person can construct an argument for any of them being the one to delete. Which one reads best, which one sounds most like the brand, which one a customer would prefer — those questions have no answer, because they are questions about taste and the three authors had different taste.

The question that does have an answer#

Change the question from which rule is best to which rule's assumption is still true. That question is empirical, and the record of it is in the file's history.

The step under that swap is worth making explicit, because the rest of the chapter rests on it. A rule in a prompt is not a preference somebody held; it is an action somebody took, and people take actions because they believe something about the world. Rule 2 was written by a person who believed that every delivery complaint was caused by something the company did. That belief is a proposition. A proposition is either true or false, and its truth is a fact about the shipments table, the carrier contracts and the calendar rather than a fact about anyone's taste. Each rule therefore inherits a status that can be settled by going and looking, and once that is on the table the argument about which sentence reads better has nothing left to decide. What the history supplies is the half that is missing from the prompt — the belief itself, which the rule's wording almost never states.

# every commit that ever touched a line containing "apolog", with the diff
git log -L '/[Aa]polog/,+3:prompts/support_system.md' --date=iso

# or, if the file has been reformatted since:
git log --follow -w -p -- prompts/support_system.md | grep -B 20 'apolog'

Both commands repay being able to read, because the first is the sharp instrument and the second is what you fall back to when it fails. git log -L takes a range within a file and returns every commit that touched that range, with its diff. The range here is expressed as a search rather than as line numbers: /[Aa]polog/ locates the first line matching the pattern and +3 extends the range three lines past it, so the command reports the history of whichever region of the prompt talks about apologising, wherever that region has drifted to. The fallback exists because -L follows a region by position, and a reformat moves every position in the file at once. --follow keeps the file's identity across renames, -w makes whitespace-only changes invisible, and searching the full patch text finds the introduction of a line even when a reflow has broken the trail -L was following.

Three commits come back.

commit 8c1f4a2  2023-11-14 10:22 +0000  m.okafor
    port support prompt from retail-bot, adapt product nouns

commit d55b019  2024-06-02 15:41 +0100  j.reyes
    SUP-1189 trim response length; users report padded replies

commit 41ae7c8  2024-09-30 04:12 +0100  s.novak
    hotfix per legal (INC-2291): no liability language on carrier delays

Rule 2 arrived in 8c1f4a2, which is not a commit about apologies at all. It is a bulk port of the entire prompt from a different product — a retail chatbot for a company that shipped from its own warehouse and had no third-party carriers in the flow. In that product, every delivery complaint was genuinely the company's fault, so "always open with an apology" was a correct and complete rule. It came across in the port because everything came across in the port, and nobody at the time was thinking about a case that did not exist yet.

The mechanism by which that rule went wrong is the one this whole chapter turns on, and it has no author in it. Whether a rule is correct is a joint fact about two things: what the rule says, and what the world is like. Rule 2 was correct when it was written, because in the world it was written for the second half of the conjunction held. Nobody then edited the rule, and nobody edited it afterwards either. What changed was the other half — a carrier network launched, and delivery complaints stopped being uniformly the company's fault. The rule therefore became wrong through an event in a different system, recorded in a different repository, on a date nobody connected to a prompt. That is why reading the prompt cannot detect it: everything the prompt contains is exactly what it contained on the day it was right.

Rule 17 arrived in d55b019, attached to ticket SUP-1189, which contains a measurement: median reply length had grown from 62 to 141 words over two quarters and CSAT — the customer satisfaction score collected after a ticket closes — had dropped four points on resolved tickets. The rule is backed by a live metric that is still on a dashboard, and the metric has held since.

Rule 31 arrived in 41ae7c8 at 04:12 on a Tuesday morning, referencing an incident. Reading INC-2291 takes ninety seconds: a customer's chat transcript, in which the agent had apologised for a carrier's mis-delivery, was submitted as evidence in a claim, and legal wanted the phrasing changed the same night. The constraint behind it is not a preference, it is a position the company's counsel holds.

Three ways a contradiction gets into a prompt#

Those three commits are the three common mechanisms by which contradictions arrive in a prompt, and each leaves a recognisable trace in both the history and the text.

The third one is the quietest. A prompt line saying "if the customer asks about their loyalty balance, call loyalty_lookup before answering" survives the removal of the loyalty programme by eighteen months, because the person who deleted the tool searched the codebase for loyalty_lookup and the prompt was a database row that the codebase search did not cover. It costs tokens on every request and it occasionally produces an agent that promises to check a balance it cannot check.

Provenance narrows the question; it does not answer it by itself. A rule that arrived in an emergency hotfix at four in the morning is not automatically correct — plenty of 4 a.m. rules are over-broad patches that a calmer version would have scoped to one input class, and INC-2291 could just as easily have been resolved by a rule about liability language rather than about apologies in general. A rule that arrived in a copied template is not automatically wrong either; the retail template may well have contained the best-written instruction in the file. What history does is convert a question of taste into a question of fact, and questions of fact can be closed by going and checking. Is there still a carrier network? Yes. Is counsel's position unchanged? Ask counsel. Is the loyalty tool still deployed? Check the registry.

Each rule is a claim about the world#

Every one of those commits left behind a claim, and the claim is the thing you check. Writing them out as predicates takes about a minute and converts an argument about tone into three lookups with owners.

The third row is the one that repays the effort, because a predicate whose checkable source is a person is the one most likely to come back qualified. The first two can be resolved with a query. The third takes a message and a wait, and it is the row where the rule as written and the constraint behind it turn out not to be the same thing.

When the history is not there#

Plenty of prompts have no usable history. They live in a database row edited through an admin UI, or they are assembled at request time by concatenating four files and two feature flags, or the repo has one commit called "initial import" from the migration. There are three substitutes, in descending order of reliability.

The evaluation suite is the best of them, because a test case is a fossil of a reason. If some case in the regression set asserts that carrier complaints do not open with an apology, then somebody once cared enough about that behaviour to pin it, and the case usually carries a name or a comment pointing at the reason. Grep the eval suite for the behaviour before you grep anything else.

The ticket tracker is second. Search it for a distinctive phrase from the rule itself rather than for its topic — "do not characterise the cause as our error" will find the ticket where someone pasted the proposed wording, whereas searching for "apology" returns four hundred customer complaints. This works surprisingly often because prompt wording tends to be written in a ticket before it is written in the file. Asking the team is third, and it works better when you paste the exact line and ask who added it and why, rather than asking whether the rule is still needed. People rarely remember whether a rule is needed; they quite often remember the incident that made them write it.

Before any of those, check whether the history exists somewhere that is not git. Version history is a property of whatever system holds the prompt rather than of git in particular. Most admin interfaces that let a human edit a prompt store one row per save, and a surprising number store an author and a free-text note beside it that nobody has ever read back.

select v.created_at, v.author_email, v.change_note,
       substring(v.body from position('apolog' in v.body) - 120 for 260) as context
from prompt_versions v
where v.prompt_key = 'support_system'
  and v.body ilike '%apolog%'
order by v.created_at;

The substring window is what makes that result readable; without it every row returns the whole prompt and you are diffing 4,000-token blobs by eye in a terminal.

Read that window's arithmetic before trusting it on a prompt you did not write. position returns where the match begins, the subtraction moves the window's start back before the match so there is context on the left, and the for clause fixes the window's width so the match sits roughly in the middle of what comes back. The subtraction is the part that breaks first: a rule appearing near the very top of a prompt gives a start index at or below the beginning of the string, and what you then get is a slice shorter than the width you asked for rather than an error. If a row comes back suspiciously short, that is why, and the fix is to floor the start rather than to conclude the rule is not in that version. If the table has a note column that is always empty, the notes were optional and the UI never insisted; if there is no note column at all, adding one and making it required at the API layer is a migration of about twenty lines. That is the only change on this list that improves next year's version of the problem rather than this year's.

One mechanical caveat on the git side: git blame on a prompt file will frequently name whoever last reformatted it, renumbered the rules, or ran it through a line-wrapper. Use -w to ignore whitespace, keep a .git-blame-ignore-revs file listing the reformat commits, and prefer git log -L over blame because it gives you the whole chain of changes to a region rather than only the most recent one. If the answer still looks wrong, the rule was probably moved between sections at some point, and --follow plus a search of the full patch history will find the original introduction.

There is also the case where the history is present, legible and does not decide anything, which is more common than it sounds. Both rules turn out to be backed by constraints that are still live — a metric someone is measured on against a position counsel holds — and the timeline shows two correct decisions taken eleven months apart by people who each had a real reason. Provenance has still earned its afternoon at that point, because it has ruled out the possibility that one of them is a fossil, which is the cheapest explanation and the one worth eliminating first. What it hands you afterwards is a genuinely contested pair, and choosing between two live constraints is a different exercise with different inputs.

A prompt rule is a decision somebody made under conditions they understood at the time, and almost all of the information about those conditions lives outside the sentence. Recovering it is cheap — three commits and one message to counsel, in the case above — and it replaces a two-sprint argument with a deletion whose justification a reviewer can verify without having an opinion about tone.

Knowing which rule to cut and being allowed to cut it are different problems. The pull request that deletes rule 2 has to survive a review in which somebody points out that the safer change is to leave all three rules in place and add a thirty-fifth that says what to do on carrier complaints specifically.