A checkpoint records execution position, not just data

Durable execution is the practice of writing down a run's progress as it happens, into a store that outlives the process doing the work, so that when a pod is replaced, a machine fails, or a run sits waiting three days for a human, some other process can pick that run up and carry it forward from the point it had reached. What it writes down is a checkpoint: a durable entry, made at each step boundary, saying which step the run is on and what the earlier steps produced. Beside it sits an execution lease, a claim written into the same record naming the one worker currently entitled to advance this run and the moment that claim lapses, which is what stops a run from being driven by two workers at once when the first has stopped answering without having stopped running.

Most backend engineers have built a partial version of this before meeting the vocabulary for it. A jobs table with a status column and a cron job that re-drives whatever is stuck, an overnight pipeline with a restart file, an order state machine that support staff can nudge forward by hand — each of those is somebody's checkpoint, written because a unit of work outgrew a single request. The idea also arrives as a product, since workflow engines such as Temporal and AWS Step Functions, agent frameworks that ship a checkpointer such as LangGraph, and database-backed libraries such as DBOS all sell some version of the guarantee, and chapters 6 through 8 weigh those against writing it yourself. This chapter builds the record by hand at the level of one Postgres row, because everything those systems offer rests on the two properties developed here.

The deploy that restarted 1,900 runs#

The deploy went out at 14:07 on a Tuesday and replaced forty worker pods over about ninety seconds, which is the normal way that service ships. At that moment 1,900 purchase-order runs were in flight on those pods. Each run is a nine-step agent: it parses a buyer's request, resolves the supplier, builds a quote, runs a credit check against an external vendor, charges a card, books a shipment, and sends two notifications. A run takes roughly six minutes of wall-clock time, and almost all of that is waiting on somebody else's API. The pods drained on a thirty-second grace period, which is enough for a run that is two steps from the end and nowhere near enough for one that has just opened a call with a ninety-second timeout.

The damage was legible the next morning. All 1,900 runs came back through the queue and restarted at step one, because that is what the code did with a message it had not yet deleted. The credit vendor charges €0.40 per enquiry and had already been called on 1,100 of those runs, so €440 of that day's €760 credit spend bought nothing and left 1,100 duplicate hard enquiries on other companies' files. Re-parsing the requests cost about 14,000 input tokens per run a second time. Sixty runs had already charged a card, and forty-one of those charged it again.

The credit spend was an annoyance and the double charges were a Friday, and they are two different bugs. The first is a failure to remember where the run had got to, and it is what this chapter fixes. The second is a failure to make an external write safe against being attempted twice, and it survives every fix in this chapter, which is why chapter 4 exists. Keeping them apart from the start is worth the effort, because a team that conflates them builds a checkpoint and expects the double charges to stop.

Built properly, the record is what lets a worker starting up after the deploy learn that six of the nine steps for PO-2026-0041 are finished and that the €0.40 credit enquiry is among them, and run only the three that remain.

Position is a fact that the outputs do not contain#

The obvious remedy is to save what the run has produced so a restart can pick it up. That is half of the record and it is the half that is easy to agree on, because everybody already has a runs table with a JSON column in it somewhere. Storing outputs on their own leaves the restarted worker with a genuine ambiguity: it knows the run holds a supplier record and a quote, and it does not know whether the run was about to charge the card or had just come back from charging it.

State the reason as a property rather than as an example, because it decides which steps need explicit markers. Recovering position from data means running the map from positions to stored states backwards. That is only possible if the map is one-to-one — if every distinct position leaves a distinguishable state behind. It is not. The notification step returns an SES message id that nobody reads, so the state before it and the state after it are the same bytes. The step that marks the buyer's account as active writes a row that was already there in nine cases out of ten, so its before and after are again identical. Two distinct positions mapping to one state means the inverse is not a function, and no amount of care in the reader will recover what the writer never distinguished.

What that licenses is a test you can apply per step while designing: does executing this step change the stored state in a way an outside reader could detect? If yes, the data carries its own marker. If no — and empty returns, idempotent writes and pure notifications are all "no" — the step needs position recorded as its own fact. The residual is that the test is about your stored state and says nothing about the world: a notification step leaves no trace in your database and has certainly changed somebody's inbox.

So position has to be written down as its own fact: which step the run is on, whether that step has started, and how many times it has been attempted.

This distinction is old and was not invented for agents. Long-running batch systems have written restart records since the 1970s, and a scientific simulation that runs for a week on a supercomputer has always dumped its arrays plus an iteration counter, because restoring the arrays without the counter tells you nothing about what to do with them. The agent case reintroduces the problem to people who had stopped needing it, because a web request that dies is simply retried by the caller and a database transaction that dies is rolled back by the engine, and neither pattern prepares you for a unit of work that is minutes long, partially externally visible, and impossible to roll back.

The same applies inside a step. The quote step iterates over line items, and a large order has 240 of them, each requiring a price lookup that takes 180 ms. Position within that loop is a cursor, and a checkpoint that records only "step: quote, status: pending" throws away the better part of a minute of lookups every time a pod moves. The unit of the record is therefore a triple: where the run is, what it has produced, and what it needs to keep going.

{
  "run_id": "run_8f3c1a",
  "epoch": 4,
  "cursor": {"step": "credit_check", "status": "started", "attempt": 2, "item": null},
  "completed": {
    "parse_request": {"buyer_id": "acct_5512", "urgency": "standard"},
    "resolve_supplier": {"supplier_id": "sup_881", "terms": "net30"},
    "quote": {"total_cents": 1420000, "currency": "EUR", "line_items": 240}
  },
  "input": {"order_ref": "PO-2026-0041", "channel": "portal"},
  "lease_owner": "worker-17",
  "lease_expires_at": "2026-08-08T14:07:31Z"
}

The completed map is what stops replay from redoing work, the cursor is what tells the resume path where to re-enter, epoch is a claim counter that rises by one every time a worker takes the run, and input is there because a run that resumes on a different pod three hours later cannot assume the original request is still sitting in a queue message. All three live in one row.

CREATE TABLE run_state (
  run_id            text PRIMARY KEY,
  epoch             bigint NOT NULL DEFAULT 0,
  cursor            jsonb  NOT NULL,
  completed         jsonb  NOT NULL DEFAULT '{}',
  input             jsonb  NOT NULL,
  lease_owner       text,
  lease_expires_at  timestamptz,
  updated_at        timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON run_state (lease_expires_at) WHERE cursor->>'status' <> 'done';

Everything in that table living in one row is a design decision rather than an incidental one, and it is the decision the rest of the chapter defends. A single row can be updated in a single statement, and a single statement in Postgres is atomic: it either takes effect entirely or not at all. That is the only property available anywhere in this topic that closes a gap for free, and spending it on the pair of facts that must never disagree — position and ownership — is what the layout buys.

The lease is what makes ownership decidable#

A row that records position is enough for a single worker that never has a twin. Nothing in the delivery path guarantees that. The queue redelivers a message whose visibility timeout lapsed, an operator reruns a stuck order by hand, and a pod that has stopped answering health checks has not necessarily stopped running code. So the row carries a lease: a worker identity and an expiry, claimed with a conditional update and extended by a heartbeat every ten seconds against a thirty-second TTL, the time-to-live after which an unrenewed claim lapses and another worker may take the run.

Concurrency control on shared rows is territory from topic 2.12 and the mechanics are the same here. What is specific to durable execution is what the lease is a claim on. It is not a lock on the data, and the difference is structural rather than terminological. A database lock is held inside a transaction and released when that transaction ends, so its lifetime is bounded by one unit of work the engine controls. This claim has to survive across many transactions and, in between them, across calls to systems the database has never heard of. Nothing in a lock manager can express that, so the claim has to be expressed as data in the row and enforced by every writer checking it.

That framing matters because the lease and the state now have to agree with each other at every instant, and they are two different columns that somebody will eventually put in two different stores.

Choosing the TTL is a trade between two costs that pull in opposite directions, and the trade has two measurable endpoints rather than a taste. A lease that is too short is one shorter than a pause a live worker can suffer while still healthy, because then an ordinary garbage-collection pause becomes a recovery event and a worker loses runs it is still working on. A lease that is too long is one longer than the delay you can tolerate before a genuinely dead worker's runs are picked up, because the run sits stranded for that long after a hard kill. So the TTL has to sit above the longest pause you actually observe and below the recovery delay the product can absorb.

Both endpoints were measured on that service. The p99 pause observed in the runtime was 4.2 seconds and the SLA on order acknowledgement was five minutes, which leaves a wide gap, and thirty seconds with a ten-second heartbeat was chosen inside it: the recovery delay was invisible to buyers and false takeovers stayed near zero. Those two numbers are the ones to measure before copying anybody's defaults. The residual on the argument is that a p99 is not a maximum — a rarer pause will exceed any TTL you pick — which is why the next section's mechanism is needed whatever the TTL is.

Two writes and one crash#

The first way this breaks is the tempting layout where state goes somewhere spacious and the lease stays in the database. The completed map for an order with a long model transcript runs to 400 KB, which is unpleasant in a JSONB column, so the results go to object storage and Postgres keeps the cursor and the lease. Now every checkpoint is two writes, and the crash lands between them.

Both orderings fail, and they fail symmetrically, which is the observation that stops people looking for a third ordering. Write the object first and the row second, and a pod killed in between leaves an object saying the credit check completed with a score of 712 while the row still says the step is pending. The resume path believes the row, re-runs the credit check, and spends another €0.40 and another entry on the buyer's credit file. Write the row first and the object second, and the same crash leaves a row claiming the step is done next to storage that has no result under that key, so the resume path either raises on a missing key or, in the version somebody wrote to make the pager stop, falls through to a default score and approves an order it should have declined. Neither ordering has fewer crash windows than the other, because the window is the interval between two writes and both orderings have exactly one.

The second way it breaks does not need two stores. A single worker can lose its lease without noticing, and the belief that a recent successful heartbeat proves current ownership is the one to give up. A heartbeat is a claim made at an instant; the write that follows it happens later; and between those two moments a network can partition, a runtime can stall, or a hypervisor can suspend the machine. Nothing the worker can observe about itself distinguishes "I still own this run" from "I owned this run thirty seconds ago", because the evidence for ownership lives in a row it currently cannot reach. From inside, both look like ordinary execution.

Worker-17 claims run_8f3c1a at t=0 with epoch 4, heartbeats at t=10 and t=20, and then its node loses network for thirty-four seconds — a routing change, a full GC pause, a hypervisor stall, the cause does not change the shape. The lease expires at t=30. Worker-23 sweeps for expired leases at t=41, claims the run at epoch 5, and starts the credit check. Worker-17's packets start flowing again at t=44, and as far as its own code is concerned nothing happened at all: the vendor call it issued before the stall returns normally, and it writes its checkpoint.

Without the epoch in the WHERE clause, worker-17's write lands, and it overwrites epoch 5's cursor with epoch 4's view of the world. The run's position moves backwards, worker-23 continues from state it did not write, and the two of them proceed to interleave for as long as both survive.

UPDATE run_state
SET cursor = $1,
    completed = completed || $2,
    lease_expires_at = now() + interval '30 seconds',
    updated_at = now()
WHERE run_id = $3 AND lease_owner = $4 AND epoch = $5;

Reading that statement as a mechanism rather than as a query is what makes the guarantee visible. Taking a run increments the epoch, so the value in the row is a count of how many times ownership has changed. A worker holds the value it saw when it claimed. Putting that value in the WHERE clause makes the update conditional on nothing having changed hands since — the row is read and written in one statement, so no other worker can slip between the check and the write. If a takeover happened, the row's epoch has moved on, the condition is false, and the statement modifies nothing. The epoch is the thing that makes the stale writer's commit fail rather than succeed, and it costs one integer.

def commit_step(conn, run_id, worker, epoch, step, result, next_cursor):
    with conn.transaction():
        n = conn.execute(COMMIT_SQL, [next_cursor, {step: result},
                                      run_id, worker, epoch]).rowcount
    if n != 1:
        raise LeaseLost(run_id, step, epoch)

Checking the row count is the whole enforcement, and the row count is the only signal available: a statement that matched no rows is indistinguishable from a successful one at the connection level unless somebody reads the count. A worker that gets zero rows stops touching the run immediately and reports that it performed the step's side effect without recording it, which puts the run into the in-doubt category that chapter 3 has to resolve. What the epoch buys is exactly that at most one worker's write lands per epoch, with no coordination between the workers and no message passing; what it does not buy is anything about the world, since worker-17's credit enquiry has already been made and the vendor has already billed for it.

The limits of a single atomic row#

It is not a backup. The checkpoint is the live position of a run and it is overwritten on every step, so a bug that writes a wrong cursor destroys the only copy of the truth. Teams that care keep an append-only log of transitions alongside the mutable row, which is a different artefact with a different purpose, and which chapter 7 shows an engine treating as the primary record rather than the secondary one.

It is not a transaction spanning the run. There is no way to hold a database transaction open across a ninety-second call to a credit vendor without destroying the connection pool, so durability here is a chain of small transactions with unprotected gaps between them. Everything difficult in this topic lives in those gaps.

It is not idempotency, which is the property that an operation performed a second time leaves the world in the state it would have reached after the first. A perfectly atomic checkpoint still cannot tell a restarted worker whether the credit request that was in flight at 14:07 reached the vendor's servers, because that fact is not in your database and never was. Atomicity gives you a record you can trust about your own execution; it says nothing about the outside world's record of you.

It is also not free, and the cost is worth knowing before somebody checkpoints every loop iteration. The conditional update above ran 3 to 6 ms on that cluster, so nine steps add about 40 ms to a six-minute run and nobody notices. Checkpointing each of 240 line items adds 240 commits and roughly 1.2 seconds, which is still fine, and doing the same inside a step that runs 50,000 times is where people quietly turn checkpointing off and lose the thread. The granularity question is a real design decision, and the rule that falls out of it is to checkpoint at the boundary of anything expensive or externally visible, and to let cheap deterministic work re-run — because re-running cheap deterministic work costs milliseconds and re-running a credit enquiry costs €0.40 and somebody's credit file.

The row now survives a restart and has exactly one owner. What goes inside it is a separate question, and a run that stores its supplier record and its quote faithfully can still come back from a restart with a different order number than the one it already sent to the supplier's portal. That failure has nothing to do with atomicity and everything to do with which values a resumed run is allowed to compute again.