Snapshotting the rendered prompt

A snapshot test calls a function with fixed inputs, compares the result against a stored copy in the repository, and fails when the two differ. Applied to prompt rendering it answers a question most review processes get wrong, which is what the model is actually sent. Teams treat the .jinja or .md file under prompts/ as the artifact under change control, and it is not the artifact: it is one of the ingredients of a function whose return value is the artifact. Reviewers watching that directory are watching the wrong file, and a change that alters the wire string without touching a template passes review unread.

This chapter puts the rendered string into the pull request. It works through what a realistic renderer draws on, how a stored copy turns an invisible change into four visible lines, and what has to be held fixed for the technique to survive — a clock, an ordering, a float format. It is also worth being precise about the claim being made, because the assertion is narrower than the name suggests and gets rejected in review by people who think it is claiming something else. The chapter ends on the two ways the technique degrades: large snapshots that nobody reads, and branches nobody wrote a case for.

The template is one input among several#

A realistic renderer, of the shape the previous chapter pulled out into its own function so the finished string exists as a value, draws from more sources than most people would guess:

def render_answer_prompt(
    question: str,
    chunks: Sequence[Chunk],
    tools: Sequence[ToolSpec],
    now: datetime,
) -> str:
    context = "\n\n---\n\n".join(
        f"[{c.doc_id}] {truncate(c.text, MAX_CHUNK_CHARS)}" for c in chunks
    )
    return TEMPLATE.render(
        system=SYSTEM_PREAMBLE,
        today=now.strftime("%A, %-d %B %Y"),
        tool_list="\n".join(t.one_line_summary() for t in tools),
        context=context,
        question=question,
        examples=load_examples("answer_fewshot.yaml"),
    )

Six inputs, and the template file governs only the arrangement. SYSTEM_PREAMBLE lives in a constants module. truncate and MAX_CHUNK_CHARS live in shared utilities that other subsystems also use. one_line_summary() is a method on the tool spec, so adding a parameter to a tool changes the prompt. The few-shot examples — worked question-and-answer pairs pasted into the prompt itself to show the model the shape of a good response — are a YAML file that a domain expert edits without touching code at all. The separator between chunks is a string literal in this function.

Seven of the eight rows are invisible to a reviewer watching the prompts directory, which is why "we review prompt changes carefully" is not a control.

The reason it is not a control has a shape worth carrying to other problems. Review attention is aimed at paths: a reviewer subscribes to a directory, a code-owners file names a directory, a checklist says "look at the prompt". The thing anybody actually cares about is not a path but a value — the string that leaves the process — and that value is computed from several paths at once. Any one of them can move it. So a control expressed over one path can only ever catch changes that happen to enter through that path, and the six rows between the first and the last are the ones that enter somewhere else. Widening the set of watched paths does not fix it either, because the Jinja row moves the value without touching any source file the team wrote — a dependency upgrade did it. The only control that covers every row is one expressed over the value itself.

Storing the string puts it in the diff#

In Python with syrupy the test is one assertion:

def test_answer_prompt_renders(snapshot):
    prompt = render_answer_prompt(
        question="Why was I charged twice in March?",
        chunks=[
            Chunk(doc_id="kb-114", text=BILLING_CHUNK),
            Chunk(doc_id="kb-207", text=DUPLICATE_CHARGE_CHUNK),
        ],
        tools=[REFUND_TOOL, LOOKUP_TOOL],
        now=datetime(2026, 3, 14, 9, 30, tzinfo=UTC),
    )
    assert prompt == snapshot

The stored copy goes in the repository next to the test. When a shared truncate() is reimplemented to cut on a word boundary and append an ellipsis, this is what the pull request carries:

   [kb-114] Duplicate charges usually appear when a saved card is retried after
-  a soft decline. Check the authorisation log before issuing a refund. [src:kb-114]
+  a soft decline. Check the authorisation log before issuing a…
 
   ---
 
   [kb-207] If two identical amounts post within 90 seconds, the second is a

Four minutes of review is enough to see it. The reviewer does not need to know anything about retrieval, ranking or the citation instruction, because the missing [src:kb-114] marker and the ellipsis are visible in the text, and the prompt tells the model to cite the marker at the end of each chunk.

How four lines of diff got in front of a reviewer who never asked for them is a chain worth following, because every link in it is a design decision somebody could have made differently. The test calls the renderer with inputs that are fixed in the test file, so its output depends on nothing but the code. Any change anywhere in the render path — a constant, a shared helper, a YAML row, a templating library — changes that output. A changed output no longer equals the stored copy, so the test fails. The only way to make it pass again is to write the new string into the stored file, and the stored file is tracked by version control, so writing it produces a diff. That diff has to travel in the same commit as the change that caused it, because the branch will not go green without it. The result is that a reviewer who was looking at a text-utility refactor is shown the exact bytes it moved, without anyone having predicted that this particular refactor would move them.

One link in that chain carries a condition rather than a fact, and it is the first: the output depends on nothing but the code. That is true only because the renderer is a pure function of its arguments, which is a property the previous chapter had to arrange rather than one that came free. A renderer that reads a clock, or samples its own examples, does not have it, and the consequence of not having it is set out two paragraphs below.

Two details of the stored file affect whether the diff is readable. Store the snapshot as an external file rather than an inline string in the test source, once the prompt is longer than a few lines, because a 60-line inline snapshot buries the test. And make whitespace visible in review, since a change from \n\n to \n \n or a switch to \r\n line endings renders as an identical-looking diff in most viewers while changing the bytes the tokenizer sees. Configuring the snapshot format to escape trailing spaces, or adding a .gitattributes entry pinning line endings for the snapshot directory, prevents an afternoon of confusion over a diff where both sides look the same.

Fixing now in that call is not decoration. A renderer that reads datetime.now() internally produces a different string every day, so the snapshot fails every midnight and the team learns to run the update command reflexively, which destroys the mechanism. Anything nondeterministic in the render path has to be a parameter: the clock, any sampled examples, any iteration over a set or a dictionary whose order is not stable, and any float formatted without a fixed precision. The clock injection from the previous chapter is what makes the first of those tractable.

What the assertion is, exactly#

A stored prompt cannot tell you the prompt is good. It cannot tell you a change was an improvement, and it has no opinion about the model's output at all — it never calls the model. What it asserts is narrower and worth stating precisely: the string sent to the model is a pure function of the inputs shown here, and that function's output has not changed without a human agreeing to the change. It is a change-detector on an input, with a review step attached.

The name causes trouble, because "snapshot test" is also what people call the thing from the first chapter: storing a model's output and comparing future outputs against it. Those two tests share a library and nothing else. One compares a deterministic function's return value against a stored copy and fails only when someone edits code; the other compares a sampled generation against a stored copy and fails when the model feels like it, which is the failure mode this topic opened with. A team burned by the second is likely to reject the first on sight, and the distinction to insist on is which side of the network boundary the stored string comes from.

That description sounds thin until you compare it with the alternative on offer. Suppose a pull request joins retrieved chunks with a blank line instead of the --- separator, the snapshot changes by one line, and the author argues the snapshot is golden-file noise and the evaluation suite is the real ground truth. A golden file is the general name for an expected output stored in the repository and compared byte for byte on every run, and the complaint against the genre is that it fails often and says little. An evaluation of the sort this team runs takes 40 minutes and costs $12.

They answer different questions, and the evaluation's answer is the weaker one to gate on, because a two-point score movement across a 200-case dataset is inside the noise band of most evaluation setups, and even when it is real the report tells you the score moved without telling you which of the pull request's seven commits moved it. The snapshot names the cause and says nothing about the effect. Deleting it to rely on the evaluation trades a fast, precise, cheap statement about the input for a slow, imprecise, expensive statement about the outcome, and puts the expensive one on the critical path of every merge. The right answer for the separator change is to accept the update, because the change is intentional and the reviewer has now seen the exact bytes that changed; approving a snapshot update is the moment the reviewer's attention was purchased.

A cheap companion assertion makes the stored string carry one more signal. Counting tokens on the rendered output and asserting a bound turns a size regression into a failure with a number attached:

def test_answer_prompt_stays_within_context_budget():
    prompt = render_answer_prompt(**STANDARD_INPUTS)
    assert count_tokens(prompt) < 6_000

That test caught a real problem on one team when a few-shot file grew from four examples to eleven: the rendered prompt went from 3,900 to 9,400 tokens, which was inside the model's context window and therefore produced no error, and more than doubled the per-request input cost across every call in production. The snapshot would have shown the seven new examples; the token assertion is what makes the consequence legible without reading them.

Those figures are worth doing rather than reading past, because the arithmetic is what turns a YAML edit into a budget line. Seven new examples added five and a half thousand tokens, which is around eight hundred tokens apiece — a plain statement about the average size of an example that nobody would have arrived at by looking at the file. The ratio is the part that reaches the invoice: 9,400 against 3,900 is nearer two and a half times than twice, and since input tokens are billed per token, the multiplier on the prompt is the multiplier on the input cost, applied to every request the service serves. Nothing about the change looked expensive at the moment it was made, because its cost is a product of a small number and a large one, and only one of the two was in the diff.

The bound in the assertion is the other half. Setting it just above the prompt's current size would fail on the next legitimate edit and be raised reflexively; setting it at the model's context limit would never fail at all, because a prompt that fits is exactly what the provider is happy to accept. The figure in the test sits between the two, with enough headroom above 3,900 for ordinary growth and far enough below 9,400 that a jump of this size cannot pass. It is a budget rather than a limit, and the difference is that somebody chose it.

How a stored prompt degrades#

Snapshot tests fail badly when they are large and numerous. A single stored prompt of 7,000 characters, updated by a command that rewrites every snapshot in the repository, produces pull requests where the diff is 300 lines and nobody reads it. The habit that follows is running pytest --snapshot-update before looking, at which point the test asserts nothing except that the renderer is deterministic.

Three things keep it useful. Snapshot the pieces separately where the pieces are separately meaningful, so that the context block, the tool list and the instruction body are three small files and a change to one produces a diff of a few lines. Keep the fixture inputs small enough that the whole rendered string fits on a screen, which usually means two short chunks rather than the eight a production request would carry. And treat an updated snapshot as a reviewable change in its own right, which some teams enforce by requiring a line in the pull request description whenever a file under __snapshots__/ is modified.

The other honest limitation is scope. A stored copy pins one input combination, so a prompt with a branch — a different preamble for enterprise accounts, an extra instruction when the user is on a mobile client — needs one per branch, and the branch nobody wrote a case for is unprotected. This is ordinary test-coverage arithmetic rather than anything peculiar to prompts, and it is worth noticing because prompt code accumulates conditionals faster than most code, and each one doubles what full coverage means.

What a stored rendering buys is a review step on an input that no other mechanism in the repository observes. It runs in milliseconds, needs no key, calls no model, and reports a cause rather than an outcome: this line of the prompt changed, and you approved it. That is a smaller claim than the word "test" usually carries and a more useful one than an evaluation score can make about a specific commit.

There are now two things in the repository that a team might loosely call a mock. One is the fake client returning a response object you wrote, which exercises your parsing and branching. The other is the stored copy of the string you send, which exercises nothing at runtime and instead catches unreviewed change. Neither has any connection to what the provider's endpoint actually returns, and a third artifact — a response captured from a real call and replayed from disk — does. Those three get conflated constantly, in code review and in interviews, and telling them apart is what decides which one to reach for when something breaks.