Replay without captured responses is only a second run

A replay feature offers to take a run that has already happened and make it happen again. The claims-triage agent this topic has followed — it reads an insurance claim, pulls the documents attached to it, estimates a repair and writes a decision — shipped one, and what the tool was doing was re-executing. It read the trace to recover the original user input and claim id, then ran the agent from scratch: live calls to the provider, live calls to fetch_claim, ocr_document and estimate_repair. The trace was a lookup for the starting parameters and nothing else. That is a perfectly reasonable thing to build and a poor thing to call replay, because the word carries a promise about determinism that the implementation does not make.

That gap matters because the promise is what people act on. Somebody with a candidate fix replays a run that failed, sees it come out green, and closes the ticket. The inference is sound only if the replay held everything except the fix constant, and re-execution holds nothing constant: the provider samples afresh, and every tool reads a live system that other people have been changing since the run happened. A cassette, in the vocabulary borrowed from the HTTP record-and-replay libraries this technique comes from, is a recording of the responses the original run received, and having one is the difference between a controlled comparison and a fresh sample.

What the second run actually observed#

Trace 9b40c1d5 is the run from chapter five, where a validation error at step nine — a payout that came out negative — traced back to an ocr_document result at step four that read a damage area of 205 square centimetres at a field confidence of 0.38, off an assessor's form whose thousands separator had been lost across a fold in the scan. Putting the original run beside its re-execution shows what the second one reproduced and what it could not.

The last line is the whole answer for this particular run. The OCR service shipped a new model on 5 March, and it reads the folded page correctly. The trajectory that failed on 4 March cannot recur on the 9th regardless of what the agent does, and the patch was never exercised.

Written as a chain, each link is checkable on its own. The failure depended on one specific tool response — a damage area of 205 at a confidence of 0.38. That response was not a property of the agent or of its input; it was produced by a live service reading a scanned page. That service was redeployed with a model that reads the fold correctly. So the value that produced the failure no longer exists to be fed to the agent, and no change to the agent, patched or unpatched, can bring it back. The trajectory was unreproducible by re-execution before anybody wrote the fix, and the fix had nothing to do with it.

That conclusion costs one extra run, which is why a control arm is the cheapest thing in this chapter and the thing the team did not have. A replay against the patched build that comes out green is consistent with two hypotheses: the patch worked, or the trajectory no longer reproduces. One observation cannot separate them, because both predict green. Running the same replay against the unpatched build does separate them, because the two hypotheses make opposite predictions about that run — under the first it should still fail, and under the second it should not. A single green is weak in a second way too, since the outcome is not deterministic: re-running a replay several times and counting the results distinguishes a trajectory that now reliably succeeds from one whose outcome is close to a coin toss.

Where the divergence actually comes from#

The reflex fix is to make the model deterministic — pin the model to an exact snapshot, set temperature to zero — and then treat any remaining variance as a provider problem. It is wrong twice over.

The reflex is tempting for a reason worth naming, because it recurs wherever somebody sets out to make a system reproducible. Temperature and model version are the two quantities here that an engineer can actually hold fixed: parameters on a request, living in a config file, set by a one-line change whose effect is visible in a diff. Holding a quantity fixed feels like removing a source of variation, and here it removes almost none of it, because the quantity is constant while the dangerous property varies. What differs between the original run and its re-execution is what the tools return, and no setting on the model request reaches that. Believing a pinned model makes a run reproducible is a belief that a defence works, and this defence is guarding the smaller half.

It is wrong mechanically on current models. claude-opus-5 rejects temperature, top_p and top_k outright with a 400; the knob no longer exists. Thinking is on by default and adaptive, so the model decides per request how much reasoning to do. Even where sampling parameters are accepted, temperature zero has never guaranteed byte-identical output: floating-point accumulation order varies with how requests are batched on the serving fleet, mixture-of-experts routing, which sends each token through a subset of the model's parameters chosen at inference time, is affected by what else is in the batch, and an alias resolves to whatever snapshot is current. Determinism was always a strong tendency rather than a contract.

It is wrong in proportion, which matters more. The claims team instrumented their replay harness to record where the replayed trajectory first diverged from the recorded one, over two hundred replays:

Tool responses dominate, and they dominate for a structural reason: a tool reads a live system that other people are changing. Claims get settled, policies get amended, documents get re-scanned, upstream services get redeployed. The model is a fixed function that drifts slowly; the world behind the tools moves every hour. A replay strategy that hardens the model call and leaves the tools live has addressed the smaller half of the problem.

Release cycles against business hours is the comparison doing the work there. A model's parameters move when a provider ships a snapshot, and a pin freezes even that, while the state behind a tool moves whenever anybody does their job — which is why the ordering of that table is a structural fact rather than a measurement that might come out the other way on another agent.

The third row is the cheapest to fix and the most embarrassing to find. A timestamp in the system prompt makes every request byte-different, which breaks the prompt cache as well as replay — one defect showing up on two dashboards.

It shows up on two because both mechanisms are the same comparison over the same bytes. Prompt caching, under which the provider keeps the tokens of a prefix it has already processed and charges a reduced rate to read them back, matches on that prefix being identical; a cassette matches on a hash of the request being identical. A value that changes on every request fails both comparisons, and fails them at the front of the prefix, which is the position that costs the most on the billing side and invalidates the most on the replay side.

Capture, with a hard failure on a miss#

Replay that means anything records the responses at the time of the original run and serves them back on demand. The recording is keyed by the position in the trajectory and a hash of the request, and the single most important design decision is what happens on a miss.

Both halves of that key are load-bearing. The position — this is the fourth model call, this is the second call to ocr_document — is what lets one request appear twice in a run and receive two different recorded answers, which is the retry case from chapter one. The hash turns the lookup into a check rather than a fetch: if the replayed run asks something the recording has no answer to, the question has changed, and the run has already stopped being the run that was recorded. Keyed on position alone, the transport would hand back the recorded answer to a different question and the divergence would pass unnoticed.

class RecordedTransport:
    """Serves captured responses. A miss is an error, never a live call."""

    def __init__(self, cassette: dict, strict: bool = True):
        self._responses = cassette["responses"]   # {key: serialised response}
        self._strict = strict
        self.divergence: str | None = None

    @staticmethod
    def key(kind: str, index: int, request: dict) -> str:
        canonical = json.dumps(request, sort_keys=True, separators=(",", ":"))
        return f"{kind}:{index}:{hashlib.sha256(canonical.encode()).hexdigest()[:16]}"

    def fetch(self, kind: str, index: int, request: dict):
        k = self.key(kind, index, request)
        if k in self._responses:
            return deserialise(self._responses[k])

        # The request differs from the recorded one at this position: the replay
        # has already diverged. Report where, and stop.
        recorded = [r for r in self._responses if r.startswith(f"{kind}:{index}:")]
        self.divergence = (
            f"{kind} #{index}: request hash {k.split(':')[-1]} not in cassette "
            f"(recorded: {[r.split(':')[-1] for r in recorded]})"
        )
        if self._strict:
            raise ReplayDivergence(self.divergence)
        return None      # non-strict: caller may fall through to live, knowingly

Raising on a miss is what separates a replay from a second run. A transport that quietly falls through to the network when it has no recording produces exactly the tool the claims team shipped: it replays the prefix that happens to match and then silently goes live for everything after the first divergence, and the output looks like a successful replay. The exception is the feature. ReplayDivergence carrying the step index and the two hashes is the most useful thing the harness produces, because "the replay diverged at step 4 because the request to ocr_document differed" is itself a finding.

What that raise licenses is worth stating exactly, along with what it does not. Because a miss can never fall through to the network, any replay that runs to completion is a re-execution of the agent's own code over recorded observations, so every difference in the outcome is attributable to a change in that code — the inference the opening case wanted and could not support. The residual is that the guarantee covers what was recorded and says nothing about what was not. A cassette exists only for runs somebody chose to record, and a full-fidelity one expires with the seventy-two-hour tier the redaction policy put it in, so replay is something you can do on a fresh incident and not on a month-old one — a property of the retention rule rather than a shortcoming of the harness.

Capture costs less than teams expect if it is selective. A full cassette for a median six-step run — every request and response for the model calls and the tools — serialises to about 48 KB. Recording every run at 6,000 a day would be 288 MB/day. Recording only runs that error, plus a 1% sample of the rest, brings it to roughly 90 MB/day, and the sample is what gives you a control group when a replayed failure comes out green. Replaying from a cassette takes about 340 ms against 6.8 s live, which changes what the tool is for: at that speed you can replay a hundred failed trajectories against a candidate fix in under a minute, and that is a regression suite rather than a debugging aid.

Both of those figures repay doing rather than reading. The daily volume is a per-run size times a daily count: forty-eight kilobytes a run at six thousand runs a day is the 288 MB/day, small enough that the argument against recording everything is about the payloads and never about the disk. The speed figure does more work, because three hundred and forty milliseconds against 6.8 seconds is a ratio of roughly twenty, and a factor of twenty changes what the tool is. A hundred trajectories at the live speed is most of a coffee break, which is something you do during an incident; a hundred at the cassette speed fits inside a build, which happens on every commit without anybody deciding to.

The policy from chapter two decides which of three you get#

A cassette is, by construction, every payload the run touched — the exact thing the redaction policy confined to a 72-hour break-glass tier. That constraint is not an obstacle to route around; it is what forces the choice, and there are three coherent answers.

Read those columns as three answers to one question rather than as a ranking. The question is what a replay has to reproduce, and each column pairs an answer with a recording the redaction policy will actually let you keep. Structural replay reproduces the shape of a run and can sit in the ninety-day tier, because a shape is not customer data. A full-fidelity cassette reproduces the run and cannot outlive the seventy-two-hour tier, because it is every payload the run touched. The third reproduces nothing, and is honest where the payloads should not be copied at all. What decides between them is the gap between what your bugs depend on and what your policy will store, and the first is a question about your own code most teams have never put to themselves.

Structural replay is more useful than it sounds, because a large share of harness bugs are indifferent to content. A tool result appended as a separate user message instead of alongside its siblings, a retry that re-sends the tool result without the assistant turn, a step cap that is off by one, a parser that chokes on a decimal — all of those reproduce faithfully with "a"*147 in place of the incident description. The claims team's structural cassettes catch about 70% of what full cassettes catch, at a twelfth of the storage and with no access controls needed at all.

About four kilobytes against about forty-eight is where the twelfth comes from, and the ratio is that large because a shape reduces a value to a handful of fields whatever its size — so the saving is largest on exactly the payloads that made the security review difficult.

The third column deserves to be a real option rather than an admission of defeat. For runs where the payloads are GP correspondence, the honest position is that they will never be replayable, and the substitute is a maintained set of synthetic claims with hand-written narratives that exercise the same paths. Writing forty of those took two days and they have the enormous advantage of being usable in CI, in a demo, and by a contractor.

Determinism and fidelity are different goals with different tools#

Deterministic replay is what you want for regression tests of your own code, and there the correct behaviour is total intolerance of variance: strict cassettes, a raise on any miss, and a failing build when a request hash changes. Fidelity is what you want for understanding one incident, and there a divergence is information rather than a test failure — knowing that ocr_document now returns 2050 where it returned 205 is the finding.

One event therefore wants opposite handling in the two modes, which is why a single harness rarely serves both without a switch. The same ReplayDivergence is a build failure in one and a result in the other, and the strict flag is where a team records which mode it is in.

There is a third thing people mean by replay that wants neither, and it is worth naming because it is often what the request actually is. A checkpoint is a saved snapshot of the agent's state as it stood after a given step, and checkpoint-based forking restores that state at step N and continues live from there with something changed — a different prompt, a different tool result, a different model. Divergence after the fork point is the entire purpose. LangGraph's Time travel: replay and fork agent runs is the clearest description of the mechanism, and the distinction from cassette replay is worth holding: a cassette records observations and re-executes from the beginning, while a checkpoint records state and resumes from the middle. They answer different questions, and a team that builds one and is asked for the other will be told their feature does not work.

Those questions are worth putting into words, because the distinction is easy to agree with and hard to apply under pressure. A cassette holds the world fixed and varies your code, so it answers whether a change breaks a run. A checkpoint holds your code fixed up to the fork and varies the world after it, so it answers what a run would have done otherwise. Neither can be tuned into the other: a cassette with the world allowed to vary is a second run, and a checkpoint with nothing varied after the fork is a shorter re-execution of a run you already have.

Replay of any kind is also not evaluation. A replayed run that now succeeds establishes that the patch does not break this one trajectory. It says nothing about the other 179,999 runs that month, and the confidence threshold of 0.55 that fixed CLM-91077 will reject legitimate low-confidence fields on some population of claims that nobody has sized. Sizing it is a job for a suite of runs scored against expected outcomes, which is a different apparatus with a different cost.

The distinction the chapter turns on is small, and everything downstream follows from it. A replay either serves back what the original run observed, or it goes and observes the world again. The first is a controlled comparison in which your own code is the only thing that moved, so a change in the outcome is attributable to it. The second is a fresh sample sharing a starting input with the run you care about, and whatever it reports is a fact about today rather than about the failure.

What the topic leaves you with is four artefacts and a habit of reaching for the right one. The span hierarchy answers what happened. The reconciled per-step cost answers where the money went, and it is only true while the daily gap check keeps firing — schedule that alert before anything else, because it is the one thing that detects your instrumentation silently rotting when a new tool ships with a model call inside it. Shape analysis over many runs answers which classes of behaviour are expensive. The causal walk plus a cassette answers why one run went wrong. On-call for this agent now works in that order, and the first question on the runbook is not "what does the trace say" but "is this one run or a class of runs", because those two lead to entirely different pages.