Fakes, cassettes and prompt snapshots

A cassette is a file holding a real request and the real response a provider gave it, captured once during a live run and served from disk by every run afterwards. The technique comes from a Ruby library called VCR, released in 2010 and named after the videotape recorder, built for testing against third-party HTTP APIs whose rate limits made live tests impractical. Python inherited it as vcrpy, usually driven through pytest-recording; Simon Willison's write-up "Using VCR and pytest with pytest-recording" is the shortest path to a working setup, and @node-llm/testing covers the same ground in TypeScript with API-key redaction handled for you. None of these were designed with language models in mind, and the mismatch shows up in exactly one place, which is how a request gets matched during playback.

By this point in the topic a repository following it holds three artifacts: a hand-written fake implementing the LLMClient port, a stored copy of a rendered prompt, and — once you record one — a cassette. They are routinely all called "mocks" in review comments, which flattens three genuinely different guarantees into one word. This chapter separates them, works through what a recording has to preserve and what has to be scrubbed out of it before it is committed, explains where in the stack to intercept and how a request is matched on the way back, and deals with the property that makes a cassette different in kind from the other two: it is a photograph of a system that keeps changing, and its realism has an expiry date that nothing in the suite announces.

Three artifacts, three different claims#

The aging row is the only failure in that table that produces a false green, which is why the rest of this chapter spends more time on the middle column than on the other two.

Why that row reads Yes for one artifact and No for the other two is the most useful thing in the table, and it is a property of what each one is compared against rather than of how carefully it was written. A hand-written fake is compared, on every run, against the code that calls it: the call site passes arguments and reads fields, and if the fake no longer offers what the code expects, the run raises. A stored prompt is compared, on every run, against the renderer that produced it. In both cases the two sides of the comparison live in the same repository, so a divergence between them is a divergence between two things a single test run can hold at once, and the run reports it.

A cassette has no such second side. It is the recorded behaviour of a system that is not in the repository, and nothing in the repository holds a competing account of that behaviour to check it against. The provider changes; the file does not; and no run can notice, because noticing would require asking the provider, which is the precise thing replay exists to avoid. Every test using the cassette continues to pass, and it passes for a good reason — your code really does handle that response correctly. What has stopped being true is that the provider still sends it. Detection therefore cannot come from inside the suite, and the only remaining source is a run that talks to the provider on purpose, which is what the scheduled re-record later in this chapter is.

What the recording has to keep#

The instinct when recording is to capture the response text, since the text is what the code parses. The text is the part that matters least, because your code branches on almost everything else.

interactions:
- request:
    method: POST
    uri: https://api.anthropic.com/v1/messages
    headers:
      authorization: [REDACTED]
      anthropic-version: ['2023-06-01']
    body: '{"model":"claude-sonnet-4-5","max_tokens":512,"tools":[...],
            "messages":[{"role":"user","content":"Summarise this support..."}]}'
  response:
    status: {code: 200, message: OK}
    headers:
      anthropic-ratelimit-requests-remaining: ['3']
      anthropic-ratelimit-tokens-reset: ['2026-03-14T09:31:00Z']
      request-id: [req_011CQk8]
    body: '{"id":"msg_01X","model":"claude-sonnet-4-5-20250929",
            "stop_reason":"tool_use","usage":{"input_tokens":1204,"output_tokens":87},
            "content":[{"type":"tool_use","name":"create_refund",
                        "input":{"order_id":"A-9931","amount_cents":4200}}]}'

Everything in that recording that your code reads is a condition it can branch on. The status code decides whether the error path from topic 2.11 fires at all. stop_reason decides whether you parse text or dispatch a tool call — the model's structured request that your code run a named function with the arguments it supplies, arriving instead of ordinary prose. The presence of a tool_use block decides which handler runs. model is the resolved dated build rather than the alias you asked for, which is what makes a cassette able to prove which build a behaviour was observed against. The rate-limit headers matter if anything in your code reads them, and a retry-after header on a 429 matters a great deal, which is why a useful recording set includes failures and not only the happy path.

Redaction is the other half, meaning the step that replaces credentials and personal data in the captured interaction with placeholders before the file is committed. The authorization header is the obvious one and is not the only secret in a request to a language model, because the request body is a prompt, and the prompt frequently contains a customer's support transcript, an internal document, or an email address. A cassette is a file in your repository that anybody with read access can open, so recording a real production-shaped request into it is a data-handling decision rather than a testing decision. The configuration that covers both:

@pytest.fixture(scope="module")
def vcr_config():
    return {
        "filter_headers": [("authorization", "REDACTED"), ("x-api-key", "REDACTED")],
        "before_record_response": scrub_pii,
        "record_mode": "none",
        "match_on": ["method", "scheme", "host", "path", "llm_body"],
    }

Where the interception happens decides what is being tested#

There are three plausible layers to intercept at, and the choice determines how much of your stack the cassette covers. Patching the SDK's messages.create to return a recorded object is the shallowest: it is easy, and it skips the SDK's own request construction, retry logic and response parsing, so it is really a fake wearing a cassette's clothes. Intercepting at the HTTP transport — installing a custom httpx transport or letting vcrpy patch the connection classes — is the usual choice, because the SDK builds a real request, serialises real JSON and parses a real response body, so the adapter from the second chapter is genuinely exercised. Intercepting at the socket layer, or running a local proxy the tests point at, covers connection handling and TLS as well, at the cost of a setup nobody on the team will want to debug.

The middle layer is right for almost everything, and knowing which one you are on matters when a test passes. A cassette applied at the SDK method level cannot catch a serialisation bug in your tool schemas, because your JSON never gets serialised.

Matching is where LLM cassettes differ#

During playback the harness decides which recorded interaction answers the request the code just made, using a match key. The defaults — method and URI — are enough for a REST API where the URL identifies the resource. Every call to a chat completions endpoint goes to the same URL, so the default key matches the first recorded interaction regardless of what you asked, and a test sending a summarisation prompt will happily receive the tool-call response recorded for a different test.

The reason the default was ever adequate is worth seeing, because it explains why this is not a bug in vcrpy. A REST API puts the identity of the thing being asked for into the path: an order lookup is /orders/ followed by that order's identifier, so two requests for two different orders are two different strings. Method plus URI is then very nearly a unique key, and matching on it is matching on what the call is about. A chat completions endpoint puts none of that in the path. The path is a constant, the identity of the call lives entirely in the body, and so the default key evaluates to the same value for every request the suite ever makes. A key that never distinguishes anything degenerates to "return the first one", which is what the harness does — without complaint, because from its point of view the request matched.

Adding the body to the match key is the obvious fix and it is brittle in a specific way: the body contains the rendered prompt, so a change to a separator, a truncation limit, or today's date changes the body, and the request no longer matches. That is the same fragility the previous chapter turned into a feature, except here it produces a confusing failure in a test that is nominally about parsing. The workable middle is a custom matcher that hashes a normalised projection of the body — model, max_tokens, tool names, message roles, and a hash of the last message's first 200 characters — so that a genuinely different call misses while a cosmetic prompt edit does not.

A matcher is a function from a request to a key, and two requests match when their keys are equal, so designing one is entirely a question of which differences to erase before comparing. Erase too little — match on the raw body — and every cosmetic edit to the prompt is a miss. Erase too much and two genuinely different calls collide, at which case a test receives a response recorded for something else and passes on it. The projection above is a set of bets about where the distinguishing content of a request lives: in the model name, the token bound, the tool names, the sequence of message roles, and the opening of the last message. Each is a field that changes when the call changes purpose and stays fixed when the wording is tidied. The bet in the last one is the least safe, since two requests differing only past the first couple of hundred characters of the final message hash the same, and the way to find out whether that describes your suite is to look at what your calls actually differ by rather than to reason about it.

What the harness does on a miss is the setting people get wrong, and the cost of getting it wrong is the whole point of the exercise:

The third branch reintroduces everything the topic set out to remove. With record_mode left at its default, a test whose request stops matching silently falls through to the network, and on a runner that has a key in its environment it will succeed, cost money, take a second and a half, and be as flaky as the live test from the first chapter. It will also rewrite the cassette file, so the first sign of trouble is often an unexplained diff in a pull request. Setting record_mode: none in continuous integration turns a miss into a loud failure naming the request that did not match, and recording becomes something a developer does deliberately on a laptop with an explicit flag.

Noticing that the tape has gone stale#

Because a stale cassette passes, staleness has to be detected by something outside the cassette, and the mechanism that works is re-recording on a schedule and diffing.

A nightly job runs the same tests with recording forced on and a real key, writes the interactions to a temporary directory, and compares them against the committed cassettes after normalising the fields that legitimately change — request-id, timestamps, token counts, the generated text itself. What remains is structural: a new field, a renamed field, a stop_reason value you have never seen, a header that disappeared. The failure does not block anyone's merge; it opens a ticket.

The cost is small and worth stating: eleven interactions re-recorded once a day is 330 calls a month, which for the traffic profile in these examples is a few dollars. That is the entire maintenance budget for keeping the most realistic artifact in the suite honest. The figure is one multiplication — eleven calls, thirty-ish days — and it is worth putting next to the arithmetic in the first chapter, where the same count of live calls ran on every push to a busy repository and came to a few hundred runs a month instead. The calls are identical. What changed is what decides when they happen, and a clock that ticks once a day is a far cheaper trigger than a team that pushes several times a day — which is the general reason expensive checks get moved onto schedules, and the subject of the last chapter of this topic.

Teams running several services against the same provider often move this a layer outwards, putting an internal gateway or proxy in front of every model call and having it record and replay centrally, so that one re-recording job maintains a shared body of real wire traffic and a schema change is noticed once rather than five times. The same position gives you a place to enforce redaction rather than trusting each repository's fixture configuration, which is the harder half of the problem to get right by convention.

Which one to keep if you keep one#

Consider a service whose only interaction with a model is a single classification call: a support message goes in, one of six labels comes out, and the code maps the label to a queue. The reflex is to keep the cassette, on the grounds that it is the most realistic. Realism is what makes it the weakest single choice here, since it is a frozen copy of a format that changes, it can pass forever against an endpoint that no longer behaves as recorded, and it says nothing whatsoever about the prompt.

The belief underneath that reflex is that realism and safety are the same property, and separating them is most of the value of this chapter. Realism is a statement about the moment of recording: on the day the file was written, it was an exact account of what the endpoint did. Safety is a statement about the future, and the two come apart the instant the recording is committed, because one of them is fixed by the file and the other depends on a system that goes on changing. A hand-written fake is less realistic than a cassette on its first day and more trustworthy on its two hundredth. It is a worse model of the provider than the cassette is, and it never claimed to be one. What it claims to model is your port, and your port sits in the repository where every run can check it.

The decision turns on where the risk lives in that particular service. The code is thin, so the fake protects little: parsing a label out of a short response has maybe two branches, one for an unrecognised label and one for a refusal. The prompt is where the behaviour is — the six label definitions, the tie-breaking instructions, the examples — and it is edited far more often than the parser, frequently by someone iterating on classification quality. For that service the stored rendered prompt is the one to keep, because it is the only artifact that observes the thing that actually changes. Add branches to the parser, or a second call, or a tool, and the answer moves to the fake. The cassette earns its place when your parsing of the wire format is genuinely complex, and even then it needs the nightly re-record to stay meaningful.

Three artifacts, then, and one rule for reading them: the two that fail when they are wrong can be trusted without maintenance, and the one that passes when it is wrong needs a scheduled job pointed at it. That is the entire practical consequence of the aging row, and it costs a few dollars a month and one configuration line to obtain.

None of the three has anything to say about a response that arrives four seconds late, a 429 carrying a retry-after of 12 seconds, or a wrapper that is supposed to give up after 30 seconds in total. Replay hands back its recorded 429 instantly, so a test exercising the retry path through a cassette proves that a retry happened and proves nothing about when.