Template revisions and run manifests

A prompt template edited in place has no name. The file has a name, the module has a name, the variable has a name — but the wording that was in production on Tuesday has none, and once someone changes it, nothing anywhere records which text produced which number. Everything else in an evaluation pipeline is addressable by then: the cases are a file, the scorer is a file, the model has a version string. The template is the piece that silently moves.

This chapter is about giving it an identity and recording that identity where the comparisons happen. The cost is a content hash and about 24 bytes per request, which is why it is worth doing before the week you need it rather than during.

Bisect names a commit; it cannot replay a run#

Start with the situation this is for. A summarisation template turns a customer's last three support messages into a two-sentence brief for the agent picking up the ticket, scored by a suite of 250 cases with human-written reference summaries, run on every merge to main by CI, the continuous-integration service that executes those checks without anyone asking it to. It scored 82.4% last Tuesday and 74.0% on Thursday morning. The suite did not change, the scorer did not change, and the eight points are twenty cases that flipped from pass to fail.

The instinctive response is process: smaller pull requests, one logical change each, better commit messages, and then git bisect over Wednesday's commits to find the one that cost eight points. Everything is already in Git, so the history is there; the problem, on this reading, is that the history is untidy. The argument is reasonable and it does not work, and understanding why is the difference between a fix that helps and a ritual that does not.

git bisect is a search over commits, and each step of the search requires a verdict: at this commit, does the eval pass or fail? Producing that verdict means running the eval at that commit, and the eval at that commit is not determined by that commit. It reads a live retrieval index that has been rebuilt since, and it calls a provider whose alias now resolves somewhere else. Checking out Tuesday's tree and running the suite gives you Tuesday's template against Thursday's index and Thursday's model, which is a fourth configuration nobody has ever measured and cannot be compared to either number.

Stated as a property of the search rather than as a story about one week, the failure is that a binary search needs its answer to be a function of the position it is standing on. git bisect works over code because does this commit pass is decided by the commit: check it out, run the tests, and the verdict is the same tomorrow as it is today. Here the verdict at a commit is decided by the commit and by three other things nobody held still, so the same commit can answer pass in the morning and fail in the afternoon. A binary search over a predicate that shifts underneath it does not fail slowly or visibly. It converges in the usual number of steps, with the usual confidence, on a commit that means nothing — which is worse than not running it, because the output has the shape of a result. What breaks the predicate is that three of the inputs a run consumes are not in the repository at all.

That is the property that matters, and it is worth setting out per input rather than as a slogan.

Three of six rows are outside the repository entirely, so no amount of commit hygiene reaches them. And the first row is the interesting one, because it is inside the repository and still fails the fourth column: the template is a module-level string, so the eval run that produced 82.4% recorded that it used summarise_ticket and recorded nothing about which text that name held at 14:12 on Tuesday. Git knows, but only if someone can tell you which commit the run used, and the run did not say.

The missing thing is not history. It is that runs do not record which artefacts they used, which is a different problem with a different fix and no relationship to pull-request size. A team with impeccable commit discipline and no run records is in exactly this position; a team with messy commits and complete run records can answer the question in four minutes.

Give the template an identity that survives the edit#

The change is small and it is the whole of it: move the templates out of module-level strings into files, and give each file's content an identifier derived from the content itself.

from dataclasses import dataclass
from hashlib import sha256
from pathlib import Path

TEMPLATE_DIR = Path("prompts")

@dataclass(frozen=True)
class Revision:
    name: str
    body: str
    digest: str

    @property
    def id(self) -> str:
        return f"{self.name}@{self.digest}"

def load(name: str) -> Revision:
    body = (TEMPLATE_DIR / f"{name}.md").read_text()
    return Revision(name, body, sha256(body.encode()).hexdigest()[:8])

load("summarise_ticket") returns something whose id is summarise_ticket@a41c9f0e before Wednesday and summarise_ticket@d7104b2c after, and the identifier changes because the bytes changed, with nobody deciding anything. That last part is why this costs almost nothing to adopt. A scheme that asks an engineer to bump a version number will be forgotten on the edit that matters, and a scheme that asks for an approval before a wording change will be routed around by the person who needs to fix a prompt at 02:00 during an incident. Content addressing has no step a human can skip.

That claim is worth unpacking, because it is the whole difference between this scheme and every version-number convention anyone has tried. A manual version is a second artefact: the bytes are one thing, the number naming them is another, and the two agree only for as long as somebody keeps performing an action to make them agree. Any scheme with that shape fails on the edit where the action is skipped, and the edit where the action is skipped is the one made under pressure — which is the edit the investigation is later going to be about. A content hash is not a second artefact. It is a function of the first, computed at load time, so the identifier cannot disagree with the text it names any more than a number can disagree with itself. There is no action to skip, because there is no action.

The renderer from chapter 3 then takes a Revision rather than a str, which makes the identifier impossible to lose between loading and sending. The same shape appears in tooling the reader already trusts: a container tag is a name that moves and a digest is a name that does not, and everyone who has been burned by :latest already knows which one belongs in a deployment record. Database migrations solved the identical problem for schemas two decades ago, and for the same reason — a schema you cannot name is a schema you cannot correlate with a report.

The identifier has to be recorded in two places, and teams routinely do only the first. Every eval run records it, which is what makes two runs comparable. Every production request records it too, which is what connects an eval score to the thing customers actually received. The provenance record from the previous chapter had a field for exactly this and its value was the string unversioned; it now reads:

"template": {"name": "summarise_ticket", "revision": "a41c9f0e", "dirty": false}

That is 24 bytes per request. At 340,000 requests a day it comes to about 8 MB, against the 375 MB the provenance store already writes, so the storage argument against it does not exist.

Both halves of that comparison are worth seeing rather than accepting. Twenty-four bytes multiplied by three hundred and forty thousand requests is a little over eight megabytes a day, and that is the field's entire cost. The three hundred and seventy-five megabytes is the store the previous chapter already argued for and already writes, so the new field adds roughly a fiftieth to something that exists. An objection on storage grounds has to be an objection to some quantity, and there is no quantity here to object to. The objection that is real is a different one — that the identifier buys nothing unless it is written in both places — and it costs the same twenty-four bytes to answer, because the same field goes on the eval run and on the production request.

What a run manifest lets you do that a score cannot#

A score is one number and a manifest is the configuration that produced it. An eval harness can emit one of these per run, alongside the score:

{
  "run_id": "evl_2026_03_19_007",
  "finished_at": "2026-03-19T08:41:22Z",
  "score": 0.740,
  "template": {"id": "summarise_ticket@d7104b2c", "dirty": false},
  "cases": "tickets_v4@6b21ce70",
  "scorer": "summary_rubric@0f9ad3c1",
  "retrieval": {"index": "help_centre", "snapshot": "snap_2026-03-19T02:00Z"},
  "model": {"configured": "claude-sonnet-4-5",
            "served": "claude-sonnet-4-5-20250929"},
  "harness": {"version": "2.3.1", "max_tokens": 1024, "concurrency": 8}
}

The served field is worth pausing on, because it is the row of the earlier table that most teams assume they cannot capture. The provider's response says which model produced it, so the harness reads it off the response rather than off its own configuration, and a run that requested a family alias records the specific version that answered. Two manifests either side of an alias change disagree on that field, which is a fact neither the score nor the repository contains.

Diffing two manifests takes a second and produces the actual suspect list:

Three fields differ, which means three suspects and no others, and the manifest turns attribution into arithmetic. Because the retrieval snapshot is addressable and the model version can be requested explicitly, each suspect can be isolated by re-running Thursday's configuration with one field reverted. A 250-case suite costs roughly $4 in tokens and eleven minutes wall-clock at a concurrency of eight, so three replays cost about $12 and half an hour, and they can be run in parallel.

Three suspects giving three replays is worth noticing as a shape, because it is linear rather than combinatorial. Each replay takes Thursday's configuration and moves exactly one field back to Tuesday's value, leaving the other two where they were, so what it measures is that field's contribution against the Thursday baseline. It does not measure what the field would have done on its own, and reading it that way is the mistake to avoid: a run scoring 79.8% is not the template's score, it is Thursday's score with the template reverted. Isolating each of three fields takes three runs. Isolating every combination of three fields would take eight, and nobody pays for that unless the three contributions fail to add up.

Where this is overhead you should not pay#

Versioning earns its keep where a comparison spans time. Two eval runs a week apart, an incident on Tuesday explained by a prompt shipped on Monday, a rollback that has to restore a specific wording — all of these are comparisons across time, and all of them fail without a name for the artefact. Where nothing is ever compared across time, the machinery buys nothing.

A one-off backfill script is the clearest case. One classified 12,000 archived tickets into product areas so an analytics team could build a dashboard; it ran once over a weekend, its prompt lived in the script, the output was reviewed by hand, and the script was deleted. There is no second run to compare it to and no production request that used it. Putting that prompt through a registry would have produced a revision id that nothing would ever read.

The developer loop is the second case, and getting it wrong is more damaging than skipping versioning entirely. Prompt iteration means editing a wording, running six cases, and editing again, forty times in an afternoon, and a registry that requires a commit before a template can be loaded turns a ten-second cycle into a two-minute one. That tax is paid directly out of prompt quality, because it is the number of iterations that produces a good prompt. The dirty flag in the manifest exists for this: locally the loader hashes whatever is on disk, marks the revision dirty when the working tree differs from HEAD, and CI refuses to accept a dirty manifest as a baseline. Iteration stays fast and no dirty number can ever be quoted as a result.

The third case is a template with no eval and no incident surface. An internal release-notes formatter run by two engineers on demand, where an odd output is edited and re-run, has no regression to detect and no customer to explain anything to, so a revision id on it is filing for its own sake.

What versioning does not do is worth stating as plainly, because the manifest looks more powerful than it is. It does not tell you which change caused a regression; it tells you which changes are candidates, and the replays do the rest. It does not make a run reproducible on its own — a manifest that names a retrieval snapshot nobody kept, or a model version the provider has retired, records the truth and cannot restore it, so retention on the artefacts is part of the design rather than a detail. And it says nothing about whether 74.0% is a good score. A comparable number and a meaningful number are different problems, and this chapter has only addressed the first.

Still, the first is the one that was silently missing. A template edited in place leaves no evidence, so a score measured against it is a number with no configuration attached, and no amount of repository discipline supplies one. A content hash costs nothing and cannot be forgotten; a manifest costs a few hundred bytes per run and turns an unanswerable question into three replays; and the same identifier on every production request is what connects the score to what customers actually received.

What the topic leaves you with is a rendering path you can now describe end to end. The template is loaded from a file and identified by its content; the trusted values are bound as arguments and the untrusted ones are typed, escaped for the output context they land in, and wrapped in delimited sections; the prompt that goes to the model has a structure no input can alter; the record that comes back names every value's source and the revision that shaped them; and the system prompt holds nothing whose disclosure would cost anything. The next prompt you render is where that becomes real, and the first decision in it is the one from chapter 1 — which characters are template, and which are data.