Versioning a prompt

A prompt is the specification of a system's behaviour, and it is edited more often than any other part of that specification. Where it lives decides what you can do when its behaviour degrades: whether you can say which change caused it, whether you can return to a state you have measured, and how much traffic a bad save reaches before anyone notices.

Those three properties are not a matter of taste about tooling. They are the difference between a bad quarter that costs an afternoon to diagnose and a bad quarter that cannot be diagnosed at all, and the second outcome ends in the only move left: rewrite the prompt from scratch and hope — which replaces a degraded prompt of unknown score with a new prompt of unknown score.

Attribution and rollback, specifically#

Why the question of where a prompt lives should carry consequences that large is a fact about the artefact rather than about how any particular team works, and it comes in two parts. The first is that the behaviour a prompt specifies is produced by a sampler rather than by a branch. A change to a conditional in application code either compiles or does not, either type-checks or does not, and either passes the test that covers it or turns it red. A change to a prompt does none of those things. Every edit to it is syntactically valid, because the artefact is prose, and the system's response to the edit is a shift in a distribution over completions that nothing in the toolchain is watching. Nothing about the artefact resists a bad edit.

The second part is the one the earlier chapters measured. The size of a diff carries no information about the size of its effect: swapping a single example for a newer one cost twelve points on a fixed set of cases, while reordering the prompt's sections cost two tenths of one. Those two facts together are what make an unversioned prompt a different risk from an unversioned configuration file. A configuration file resists a bad edit, because a malformed value throws on boot and a wrong value usually breaks something visible within the hour. A prompt absorbs a bad edit and keeps serving, at a quality nobody has measured, until a support queue notices a quarter later.

Because the artefact supplies no resistance of its own, everything that makes a regression recoverable has to come from outside it: a record of what the text used to be, and a number attached to each state of it. Two operations depend on that record, and in a console both fail for the same reason.

The first thing that breaks is attribution. Bisection, as chapter 5 used it, requires a sequence of versions you can re-run; with a string of undocumented edits there is no sequence, so the question "which change cost us this" has no procedure that answers it. Worse, you cannot even establish that the prompt caused the regression, because a quarter contains other changes: a provider rolls its model alias forward, a retrieval index is rebuilt, a support macro that generates half the input tickets gets reworded. Each is a plausible cause. Under a versioned prompt with a score attached to each version, you eliminate the prompt as a suspect in twenty minutes and go look at the other three.

Why a sequence is the requirement, rather than merely a convenience, is worth being exact about, because "keep a history" sounds like hygiene and is closer to a precondition. Attribution is a search. You have one state of the system that scored well, another that scores badly, and a set of edits lying between them; the question is which of those edits is responsible, and answering it means evaluating candidates. A search needs something to search over. Bisection is efficient because each re-scored midpoint discards half of the candidates that remain: eleven edits are cut to five or six by the first run, to three by the second, to one or two by the third, so about four runs settle a quarter's worth of editing.

That efficiency is not what a console is missing. A console holds one value, so the set being searched has exactly one member and there is no midpoint to score. The question is not expensive to answer there. It is unaskable, and no number of runs helps, because there is nothing to run.

The second is rollback. When a regression is confirmed, the operation you want is to return to the last version that scored well, which requires that a scored version exists as an artefact. In a console there is no such artefact — the best available action is reverting to a text someone remembers, which is a new version with a new unknown score, and if the complaint rate then falls you have learned nothing transferable.

Why a remembered text counts as a new version rather than as the old one is the step people skip when they say the prompt can simply be put back. A prompt's identity is byte identity. The model conditions on the exact string it is sent, so two texts that a person would describe as the same prompt — same six slots, same examples, one clause rephrased from memory — are different inputs and produce different distributions over completions. The cleanup in chapter 6, on what shortening a prompt costs, puts a size on that difference: a clause of about twenty-five tokens, whose purpose nobody could remember, was worth eight points. A reconstruction from memory reliably preserves the parts a person would summarise and loses exactly the clauses that read as unremarkable, which is the population the score turned out to depend on.

A third comes free with the other two and bites earlier than people expect: an edit in a console is live for 100% of traffic the moment it is saved. There is no staged rollout, no canary sending the new version to a small share of traffic first while the rest stays on the old one, and no protection against two people editing within the same minute and one silently overwriting the other. On a service taking 400,000 calls a day, a bad save is affecting tens of thousands of users before anyone opens a dashboard.

The speed argument is correct and does not settle it#

The product manager's objection is the serious one: shipping a prompt fix takes 30 seconds today, would take a day with a review and an eval gate, and that speed is part of why the team launched before the competitor did. Dismissing this is how governance proposals die in the meeting after the one where everyone agreed to them.

Two things are worth conceding. The speed was real and it bought something real. And the last row of that table is the one that matters for the objection — an emergency fix must stay fast, because a prompt bug in production is an incident and incidents are not improved by process.

What is not conceded is the framing that the choice is between 30 seconds and a day. A day is a property of a particular gate — one requiring a human approver in a different timezone, a nightly eval, and a release train. An eval that runs 200 cases at eight-way concurrency finishes in about six minutes and costs a couple of dollars, and it runs in continuous integration — CI, the automated checks your repository already runs on every pull request — while the author is still writing the description. The routine change goes from 30 seconds to roughly fifteen minutes, most of which is the author waiting for a robot.

The six minutes is worth deriving rather than quoting, since the objection is about time and a figure nobody can reproduce settles nothing. Two hundred cases run eight at a time is two hundred divided by eight, which is twenty-five sequential rounds. A round is one model call plus the parsing of its reply, and a short classification call takes a handful of seconds, so twenty-five rounds land somewhere between two and six minutes depending on the model and the length of the output. Nothing about that estimate is delicate. Doubling the concurrency halves it, a slower model or a longer reply doubles it back, and what carries the argument is the order of magnitude, which is minutes rather than hours.

What the wait buys is worth setting beside it honestly. The routine change costs an extra quarter of an hour, almost all of it spent waiting rather than working. The emergency change costs nothing at all, because the override below leaves it at thirty seconds. And the quarter described further down this chapter cost the ability to say anything whatever about eleven edits, which is thirteen weeks of a team's work that produced no recoverable knowledge. A fixed, small, predictable cost is being traded against an unbounded and unattributable one.

The emergency path is what keeps the objection answered rather than deferred. Give the system an override — an environment variable or a feature-flag value that supersedes the checked-in prompt — with a mandatory expiry and an alert:

def active_prompt() -> Prompt:
    override = flags.get("prompt_override")          # set by on-call, TTL 24h
    if override:
        log.warning("prompt_override active", expires=override.expires_at)
        return Prompt(text=override.text, version=f"override:{override.id}")
    return Prompt(text=PROMPT_TEXT, version=PROMPT_SHA)

Whoever sets an override gets 24 hours, an automatically opened pull request containing the override text, and a page if it is still active at expiry. The fix ships in 30 seconds, and the thing that makes an unversioned quarter unrecoverable — a change that is permanent, silent and unattributable — is now temporary, loud and attributed.

That sentence compresses the chapter's argument, and it is worth taking apart one property at a time. A console edit is permanent: once saved, the previous text exists nowhere, and the new text is the state of the system until somebody types over it. The expiry inverts that property directly. An override is a state with an end date, so the default outcome of doing nothing is a return to the checked-in text rather than the indefinite survival of whatever was typed at two in the morning during an incident.

A console edit is also silent and unattributable, and the remaining two parts answer those one each. The automatically opened pull request carries the override text into the same place every other change to the prompt is discussed, so the emergency edit gets reviewed on the ordinary schedule instead of never, which is the silence removed. The page at expiry forces a decision from a named person while the reason for the override is still fresh, so the override either becomes a merged change with a score attached or lapses back to a state that already has one, which is the attribution restored. None of the three costs the on-call engineer anything at the moment of the fix, and that is what keeps the thirty seconds intact.

The version field is doing quiet work in the snippet above. Log it with every request alongside the request id, and a support complaint from six weeks ago can be joined to the exact prompt text that produced the answer. That single field is most of what a reconstruction needs, and it costs nothing.

The join it makes possible runs in three steps, and each step needs a different system to be carrying the same identifier. A complaint arrives with a request id, because the support tool recorded the id of the call that produced the answer. The request log carries that id alongside the prompt version, because the snippet above put it there. And the repository carries the exact text at that version, because a commit hash names an immutable state of a file. Follow the id through those three and you arrive at the bytes that were sent, six weeks after they were sent.

What the reconstruction does not give you is the reason for that particular answer. The prompt is recoverable and the sampled draw is not. The picking step that turned the model's scores into one concrete completion was a random choice made once and recorded nowhere, so re-running the same prompt against the same model may return something else. The join establishes what the system was asked. It does not establish why this one reply came out the way it did, and a team expecting the second from it will conclude the logging is broken when it is working exactly as designed.

The smallest gate worth shipping next week#

The common proposal at this point is to move the prompt into the repo and require code review on every change. That is the right destination reached by reasoning that does not survive contact with production, because a reviewer who has not run the eval catches typos and formatting and nothing else. Nobody can tell by reading a diff that swapping one example for a newer one cost twelve points; that was invisible to the author, to three reviewers, and to the model. Review without a score attached adds latency to every change and buys the appearance of control. The score is the reviewer; the human is there to argue about whether the metric was the right one.

Review without a score buys the appearance of control because of what a reviewer can and cannot see, rather than as any comment on reviewers. Reading a diff means evaluating a change against a model of what the text means: whether it says something true, whether it contradicts a line above it, whether it is clear. Every one of those judgements is about the text. What a prompt edit actually does is move a distribution over completions, and that distribution is written nowhere in the diff — it is a fact about the model's response to the new bytes, observable only by sending them. So a reviewer is well calibrated on exactly the class of defect reading can reach: a typo, a contradiction between two visible sentences, an example that breaks a constraint three paragraphs above it if they think to check. The nine-line test from chapter 3, which reads every example against every constraint, exists because that last one is mechanical enough to stop depending on somebody thinking to check.

Everything outside that class is invisible by construction, and the swapped example is the standard demonstration of it. Nothing about the newer example looked wrong. It was a real production transcript, better written than the one it replaced, and it hedged in the way a newly added constraint forbade — a relationship between two parts of one file that no reader was holding in their head at once. For that class of defect the score is the reviewer, and the human is there for the question a score cannot answer, which is whether the metric was measuring the right thing.

The pieces, concretely. The prompt becomes a file under prompts/, loaded at boot, with its git SHA — the commit hash that identifies exactly which version of the file is running — logged on every request. The evaluation set — 150 to 300 cases, tagged by slice — is checked in beside it, along with a single stated metric and a recorded baseline including the run-to-run spread from three seeds of the unchanged prompt. A CI job triggers on any diff touching prompts/** and posts the before-and-after table as a comment. Two hard failures block the merge: schema validity below 100%, and an aggregate more than the noise band below baseline. The example-versus-constraint test from chapter 3 runs as an ordinary unit test in seconds on every commit, because it catches a specific failure and needs no model calls.

What deliberately does not go in the first week: nightly full-set runs, a model-judge metric, a dashboard, a review board, per-slice hard gates, or a prompt-management platform. Each is defensible later and each is a reason for the first version to slip a month. The gate that exists on Friday is worth more than the gate that is still being designed in June.

Week two, and where the prompt lives#

Two failure modes show up almost immediately and are worth pre-empting. The first is a gate that blocks on noise: if the baseline was recorded from one run and the threshold is "any decrease", the job will fail on unchanged prompts and someone will add a bypass label within a fortnight, after which the gate is decorative. Record the spread from three seeds and set the threshold below it. The second is an evaluation set with no owner, which decays as the input distribution moves until it is blocking changes on the basis of last year's traffic; put a name against it and a recurring half-day to add cases from recent production failures. Both are ordinary CI problems and neither is specific to prompts, which is most of the argument for putting prompts where the rest of your CI already lives.

Storage is the part of this that people argue about and the part that matters least. A prompt registry or a managed prompt service is entirely acceptable, provided it versions immutably, exposes a diff, can attach a score to a version, and can be rolled back — those four properties are the requirement, and a git repository happens to have all of them for free. What is not acceptable is any store where the current value is the only value.

The honest limit of all this is that it makes quality claims possible without making quality good. A gate stops a twelve-point regression from merging; it does not tell you that your evaluation set has drifted from what users now ask, that your metric rewards length, or that the 88 you have been defending for two quarters was never very good. Those remain judgement, informed by reading production outputs regularly, which no amount of CI replaces.

Which of this topic's failures a gate would have caught, and which it would have waved through, is worth being specific about. The twelve-point example swap: caught, because it moved a number on a fixed set of cases. The competitor constraint repealed by three well-written demonstrations: caught, and caught earlier and cheaper, by the unit test rather than by the eval. The deleted clause that took a rare slice from eighty-one to twelve: caught only if that slice was named before the cut, which is why chapter 6, on what shortening a prompt costs, argued for defining slices in advance rather than reconstructing them afterwards. The conflict between a length ceiling and a citation rule: not caught by a gate watching one metric, because the change that fixes the metric being watched is the change that wrecks the one that is not, and the gate reads an aggregate that went up. And an evaluation set that no longer resembles production traffic: not caught by anything in this topic at all, because every instrument here reports against that set, so they agree with each other while being wrong together.

Versioning, then, is not a governance layer bolted onto prompt work. It is the thing that makes every technique in this topic executable: a diagnosis needs a folder of failures traceable to the text that produced them, a bisection needs a sequence, a slice table needs a "before" column, and a demonstration-versus-rule test needs somewhere to run. Without a version history each of those degrades into an opinion, which is how a quarter of gradual decline becomes a mystery rather than a bug.

Stated as one property rather than four, every instrument in this topic is a comparison, and a comparison needs two things to compare with a number on each. That is the whole of what versioning supplies, and it is why the storage question decides the technique question rather than sitting alongside it. A store whose current value is the only value holds one thing, which is one short of the minimum any of these methods requires.

You now have the parts: six slots and what each is for, a way to read a bad output back to the one that is missing, the knowledge that a demonstration outranks a rule it contradicts, the difference between an absence and a conflict, an apparatus that makes a comparison mean something, an honest account of what cutting a prompt costs, and a gate that keeps the answers. The next step is not another chapter. Take the prompt you are least confident about — the one whose failures your team describes with a shrug — and spend an afternoon on it: read thirty failures, sort them into buckets, name the slot, write the one sentence, and measure it against a set of cases you fix today and do not look at again.