Recomputing a generated id on resume invents a second reality

The checkpoint built in the previous chapter is atomic, has exactly one owner, and is correct about position. None of that prevents the failure in this chapter, and it is worth being clear why before anything else: a checkpoint records what the steps returned, and the damage here is done by a value that was never a step's return value. It was computed at the top of the run, used, sent outward, and never written down, because it did not look like a step result to anyone.

The consequence has a specific shape that is worth naming, since it is what makes the bug hard to see. A run that regenerates such a value is not corrupted and does not fail. It is perfectly consistent with itself and inconsistent with everybody else, so every error it produces surfaces in somebody else's system — an unmatched webhook, a duplicated allocation, a reconciliation that will not balance — and arrives at your team as a report about their system rather than as an exception in yours.

The order that existed twice#

Supplier sup_881 raised a ticket in March. Their portal had received purchase order PO-2026-0041 at 09:12, acknowledged it, and allocated stock against it. At 09:47 the same buyer's order appeared again as PO-2026-0067, for the same 240 line items and the same €14,200, and the supplier's team assumed the buyer had double-ordered and allocated a second pallet. Neither number existed in the buyer's ERP — the enterprise resource planning system, the internal system of record that owns orders, stock and invoices — which by then held a single order under a third reference.

What happened was one worker restart. The order reference was minted at the top of the run with a call to the ERP's sequence generator, the run crashed during the shipment booking, and the resumed run called the sequence generator again because that line of code sat above the resume point. The checkpoint recorded that resolve_supplier and quote had completed, and it did not record the order reference. Three references now existed for one order: two the supplier holds and one the ERP holds, and no system anywhere has a record connecting them.

What "already acted on" means#

The rule is narrow enough to apply mechanically. Any value the run obtained nondeterministically and then made externally visible has to be captured in the checkpoint and read back on resume rather than regenerated.

Both halves of that condition are doing work, and dropping either one produces the wrong answer in a predictable direction. A deterministic value needs no recording, because determinism means that the same inputs produce the same output, so recomputing it from recorded inputs reproduces it exactly and nothing can disagree. A value nobody outside the run ever saw needs no recording either, because there is no external record for a fresh value to contradict — the run is free to invent a new one and no party is holding the old. Only a value that is both nondeterministic and externally visible creates the possibility of two parties holding different answers to the same question, which is the entire failure.

Externally visible covers more ground than an API call: a value written to your own database, printed into an email, embedded in a log line an auditor will read, or returned to a caller has left the run and is now part of somebody's record. That is the half of the test people under-apply, because "external" reads as "another company" and the buyer's own ERP is external to the run.

The complement of the rule is what keeps checkpoints small. The tax on the quote is round(subtotal * rate), and recomputing it from a recorded subtotal and a recorded rate gives the same cents every time, so it does not belong in the checkpoint. The saving compounds: once the inputs of a derivation are recorded, every value downstream of them is free, so the record only has to hold the leaves of the computation rather than all of it.

One comparison is close enough to be worth offering as an analogy, with its limit stated. Recording the values a run acted on works the way a game replay file works: record the seed and the player's inputs, re-run the simulation, and everything derived comes back identically, while anything the simulation sampled outside that record diverges and desynchronises the replay two minutes in. The analogy breaks on what a divergence costs. A desynchronised replay is a spoiled playback and nothing more, because the simulation is the only thing that ever existed; a resumed run has already told a supplier's portal, an ERP and a payment gateway about its outputs, so a divergence is not a corrupted recording but a second set of facts that other parties have already acted on. There is no re-record.

Deciding which side of the line a value falls on is where the work is, because the interesting cases are not obviously random.

The first two now() rows are the same function landing on opposite sides, which is the clearest evidence that the test is about what happened to the value rather than about how it was obtained. Neither call is more random than the other. One was printed into a document a buyer is holding and one was divided by another timestamp and thrown away.

The last two rows of that table hide most of the arguments teams have. The attempt counter looks like bookkeeping and it is control flow, because a resume that reads attempt 0 will happily give a step four more tries after it already had four — and the whole purpose of an attempt limit is to bound the total, so a limit that resets is not a limit. The supplier price is worse, because the correct answer depends on what the run has already promised. If the quote has gone out at €59.16 per unit, the resumed run must charge €59.16 even if the supplier raised the price at midnight, and it must record that it is doing so. If the run crashed before the quote was sent, re-reading the price is the better behaviour, and pinning the old one commits you to a number the supplier will reject. The same value, the same API call, and the deciding fact is whether an email has left the building.

Recording at the step boundary#

The tempting implementation is a set of helper functions — a checkpointed_uuid, a checkpointed_now — sprinkled wherever a nondeterministic value appears. It works and it decays, because the twentieth call site is written by somebody who does not know the rule and reaches for uuid.uuid4() directly, and nothing fails in test or staging where processes do not die.

The structural version puts the memoisation — storing a result under a name the first time it is computed and handing back the stored copy on every later call — at the step boundary, so a step is a named unit whose entire result is written once and read back on replay. A value that matters is a value inside a step's result envelope, and the only way to obtain one is to run the step or read it from completed.

async def step(ctx: RunContext, name: str, fn):
    if name in ctx.completed:
        return ctx.completed[name]           # replay: no execution at all
    result = await fn()
    commit_step(ctx.conn, ctx.run_id, ctx.worker, ctx.epoch,
                name, result, next_cursor(name))
    ctx.completed[name] = result
    return result

The wrong version of the order step and the right one differ by where the generator call sits.

# wrong: the reference is minted above the memoised boundary
order_ref = erp.next_purchase_order_ref()
supplier = await step(ctx, "resolve_supplier", lambda: resolve(order_ref))

# right: minting is itself the step, so its output is recorded
async def open_order():
    return {
        "order_ref": erp.next_purchase_order_ref(),
        "opened_at": datetime.now(timezone.utc).isoformat(),
        "quote_valid_until": (datetime.now(timezone.utc) + timedelta(days=7)).isoformat(),
    }

order = await step(ctx, "open_order", open_order)

Follow the replay through the wrong version and the defect is mechanical rather than subtle. A resume re-enters the function and the interpreter runs the statements in order. The first statement is the generator call, which is an ordinary function call that step knows nothing about, so it executes and returns a new reference. Only on the second line does control reach a memoised boundary, and by then the damage is a local variable holding a value the outside world has never heard of. The fix is not to record more; it is to move the boundary so that the minting happens inside a step. After the restart, open_order is a dictionary lookup and order["order_ref"] is still PO-2026-0041, which is the reference the supplier's portal is holding stock against. The three values that travelled outward now travel outward once.

There is a third approach between sprinkled helpers and step envelopes, and it is worth knowing because the durable engines use it. Instead of asking authors to route nondeterminism through a recorded boundary, the runtime intercepts the sources: the clock, the random number generator and the uuid function are replaced by versions that consult the run's history, so datetime.now() inside workflow code returns the recorded instant on replay without the author doing anything. That buys safety at the cost of a rule about where your code may run, since the interception only applies inside the runtime's own execution context and a helper function called from a library will happily read the real clock. The step-envelope approach keeps the rule visible in the code and puts the burden on review; the interception approach hides the rule and puts the burden on a determinism checker that fails runs at replay time. Chapter 7 shows the second one in operation, including what its failures look like.

Model calls carry more than the text you wanted#

Model calls need the same treatment and a little more of it, because the artefact you must preserve is the whole assistant message rather than the text you cared about.

The chain that makes this compulsory runs through the provider's own validation. A model that decides to call a tool returns an assistant message containing a tool-use block with an identifier the provider generated. Your next request has to include that assistant message and a tool-result block naming the same identifier, because that pairing is how the provider knows which call the result answers. A resumed run that re-samples gets a fresh identifier, and a resumed run that reconstructs the transcript from a stored summary has no identifier at all. Either way the request contains a tool result naming something the conversation does not contain, and the provider rejects it. That failure arrives as a 400 on resume, with an error about an unmatched tool call, and is usually misdiagnosed as a serialisation bug — reasonably enough, since the visible symptom is that a rebuilt message structure was refused.

{
  "shipping_decision": {
    "model": "claude-sonnet-4-6-20260514",
    "assistant_message": {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "Consolidating into one pallet; standard freight is adequate."},
        {"type": "tool_use", "id": "toolu_01Qk3fD9", "name": "book_freight",
         "input": {"class": "standard", "pallets": 1}}
      ]
    },
    "usage": {"input_tokens": 14210, "output_tokens": 96}
  }
}

Recording the sample rather than re-drawing it also removes a source of drift nobody controls, and this is where a widely held belief needs giving up. Temperature zero feels like a determinism switch, and within one snapshot of one model it does remove the sampling randomness — which is exactly what makes it tempting. What it does not fix is that the thing being sampled from can change. The provider can change kernels, batching or the model snapshot behind an alias, and a run resumed nine days after it started may be sampling from something that no longer answers the same way. The setting stays at zero while the function underneath it moves, so a value that was safe to recompute on Monday is not safe to recompute on Thursday and nothing in your code or configuration records the difference.

The size of the drift was measured on this system rather than assumed. On the shipping-class prompt, a re-sample disagreed with the original in about 4 out of 100 replays at temperature 0.3, which sounds small until it is 4 orders in 100 whose emailed shipping class no longer matches the booked freight.

Where the rule stops being useful#

Recording is a commitment to a value's continued validity, and that commitment expires. A run that pauses for three days holds a price read on Monday, and replaying it on Thursday is correct with respect to the quote already sent and wrong with respect to the supplier's current catalogue. The resolution is not more recording; it is an explicit revalidation step that re-reads the price, compares it to the recorded one, and either proceeds or routes the difference to a human. Chapter 5 makes that concrete, and the general point is that "replay the recorded value" answers what the run should believe about the past, and says nothing about what it should do about the present.

Nor does capturing a value protect the action taken with it. The order reference survives the restart, and the resumed run can still submit that same reference to the supplier's portal a second time, which is a duplicate submission carrying a correct identifier instead of a wrong one. That is a strictly better position to be in, because a stable identifier is what makes deduplication possible at the far end, and it is not by itself a fix. Chapter 4 is about turning a stable identifier into an actual guarantee, and this chapter's contribution to that is the identifier.

The rule also does not mean recording everything the run touched, and the cost of over-recording lands in a place that surprises people. A completed map that stores the full 240-item price lookup response for every step grew to 1.9 MB per run on that service, and at 1,900 runs a day that is 3.6 GB of JSONB, the binary JSON column type Postgres stores these documents in, written daily to a table that is also the hot path for every lease renewal. Those two facts multiply rather than add. Postgres does not edit a row in place; an update writes a new version of the whole row and marks the old one dead, so a heartbeat that touches a 1.9 MB row rewrites 1.9 MB and leaves 1.9 MB of dead version behind it. Two behaviours follow: autovacuum, the Postgres background process that reclaims the space held by superseded row versions, falls behind, and a resume that has to load a 1.9 MB row takes 90 ms instead of 4 ms. Record what left the run and what the remainder needs. Everything else can be re-derived or re-fetched.

There is a compliance edge here that is easy to walk into. A checkpoint that captures model transcripts, buyer contact details and card metadata is a personal-data store with the same retention and deletion obligations as any other, and it is one that most teams forget when they answer a deletion request, because it does not look like a database of customers. Deciding up front what the checkpoint may hold — references rather than payloads, for anything sensitive — is much cheaper than retrofitting redaction into a table that also has to be replay-correct.

Finally, this rule is about values and not about code. Capturing the order reference does not make the surrounding function deterministic; a run that iterates a Python set, or reads a feature flag inline, still takes different paths on replay even though every recorded value matches. That distinction stays harmless while you control the resume path yourself, because your resume path only re-enters at a named step and never re-runs the code above it. It becomes load-bearing in chapter 7, where an engine re-executes your workflow function from its first line and treats any divergence as an error.

With position recorded atomically and the run's nondeterministic values pinned, the resume path finally has enough to work with. What it does not yet have is a decision procedure: given a checkpoint, which steps count as done, which count as untouched, and what to do about the one that was halfway through a vendor call when the pod died.