A three-day approval cannot live in a process or a callback

A durable pause is a run's position expressed as stored state rather than as a suspended function: the run is not running anywhere, no process is holding anything open, and something on disk says what it is waiting for and what would satisfy it. It is the case that separates a system which survives restarts from one which survives restarts and also waits for people, and the two are further apart than they look.

Orders above €10,000 need a named approver, which is a control the finance team wrote into policy long before the agent existed. That single sentence of policy is what turns a six-minute run into a three-day one, and every mechanism in this chapter follows from the gap between those two durations. A run whose longest wait is a ninety-second API call can be held in memory; a run whose longest wait is a weekend cannot be held in anything that a deploy is allowed to replace.

The dictionary that went with the pods#

The first implementation of the approval control was fifteen lines. The run reached the charge step, posted a message into the approvals channel with two buttons, created a future, and awaited it. A small HTTP handler received the button click, looked the approval id up in a module-level dictionary, and resolved the future. It worked in staging, it worked in the pilot with two suppliers, and it went to production in June.

On 14 July at 11:20 a deploy replaced the workers. Twenty-three orders were sitting at the approval step, worth €412,000 in total. The dictionary went with the pods. At 13:05 a finance manager clicked Approve on a €38,000 order and got a 404 from an endpoint that no longer had any idea what she was approving; by 16:00 there were nine such clicks and a thread in the channel asking whether the buttons were broken.

Those two symptoms are one failure, and seeing why is the whole lesson of the section. A future is a data structure in the memory of a process. The dictionary that maps approval ids to futures is in the same memory. Replacing the pods discards that memory, so the run's continuation and the lookup table that would have resumed it die in the same instant and for the same reason. There was never a moment when one could have rescued the other. The runs themselves were not merely paused, they were gone: the future was the only thing holding the run's continuation, and nothing on disk said that order PO-2026-0193 was waiting for anybody.

What the waiting state has to contain#

Three things go in the row, and the failure above lost all three at once.

The proposed action comes first, fully specified rather than described. The approver is authorising a charge of €38,000 to supplier sup_412 against card ending 4417 for order PO-2026-0193, and every one of those particulars has to be in the record.

The reason is that an approval is a statement about an action as it was described, rather than a resumption token for a function. Store a pointer saying "resume run_8f3c1a at step charge" and the actual amount has to be recomputed when the run continues, which reopens chapter 2's problem in its most expensive form: the approver said yes to €38,000 and the run charges whatever the supplier's price feed says on Thursday. The thing approved and the thing done are then two different objects that happen to share a step name. What licenses the whole control is that they are the same object, and the only way to keep them the same is to make the description durable.

The timer comes second. Something has to fire when nobody clicks, and it cannot be a sleep in a process, a setTimeout, or a queue message with a delay, since queue delays are capped in the range of minutes and the process is exactly what is not going to survive. A durable timer is a fire_at column and a sweeper, meaning a query that any worker runs every few seconds to find the rows whose time has arrived and wake the runs behind them. Nothing is held open between the pause and the fire; the passage of time is checked rather than waited on.

The authorisation context comes third and is the one most often missed. The run acts on somebody's behalf, and by Thursday the token it was carrying on Monday has expired, been rotated, or belongs to an employee who left. What gets recorded is the grant rather than the bearer credential, meaning the token a request physically carries: the principal the run acts as, which is the identity its requests are made in the name of, the scopes it was granted, which are the named permissions attached to that identity, the approval policy version in force at the time, and after the decision, who approved and when. The resumed run mints a fresh credential from that grant, and if the grant is no longer valid the run stops rather than continuing under whatever ambient permissions the worker happens to have.

{
  "pause_id": "pause_5d21",
  "run_id": "run_8f3c1a",
  "step": "charge",
  "status": "waiting",
  "proposed_action": {
    "kind": "payment",
    "amount_cents": 3800000,
    "currency": "EUR",
    "supplier_id": "sup_412",
    "payment_method": "pm_4417",
    "order_ref": "PO-2026-0193",
    "quote_valid_until": "2026-07-21T11:19:00Z",
    "effect_key": "run_8f3c1a:charge:0"
  },
  "authorisation": {
    "acts_as": "svc:procurement-agent",
    "on_behalf_of": "user:1182",
    "scopes": ["payments.create", "orders.write"],
    "policy_version": "fin-approvals-v4",
    "requires_role": "finance_approver"
  },
  "created_at": "2026-07-14T11:19:02Z",
  "fire_at": "2026-07-17T11:19:02Z",
  "on_expiry": "escalate",
  "decision": null
}

The record also has to carry the explanation the approver reads, generated at pause time and stored, rather than regenerated when the request is rendered. Agents produce their justification from a model call over a context that will not exist on Thursday, and re-running that call to populate a Slack message on Wednesday produces a second explanation which may not match the first, at a cost of 14,000 input tokens per render. That is the same rule as the proposed action one level up: the explanation is part of what the approver was told, so it is part of what was approved. Storing the rendered summary alongside the structured action means the approver sees the same three sentences every time they open the request, and it means the audit trail contains what the approver actually saw rather than a reconstruction of it.

-- the sweeper: one query, any worker, no process holds the pause open
SELECT pause_id, run_id
FROM run_pause
WHERE status = 'waiting' AND fire_at <= now()
ORDER BY fire_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

FOR UPDATE SKIP LOCKED lets any number of workers sweep the same table without stepping on each other, and the mechanism is worth one sentence because it is what makes the sweeper a plain query rather than a leader-elected job. FOR UPDATE locks each row the query returns; SKIP LOCKED tells the query to pass over rows another transaction has already locked instead of waiting behind them. So two sweepers running the identical statement at the identical moment take disjoint sets of due rows, and adding a third just partitions the work further. The ORDER BY fire_at and the LIMIT 100 are there for the same reason as each other. Ordering by due time means the oldest overdue pause is picked up first, so a backlog drains in the order things became due rather than in whatever order the planner returns rows. The limit bounds how much one sweep can take on, which keeps a single worker from locking a large set of rows and holding them while it wakes the runs behind them. The run they wake is claimed with the same lease and epoch mechanism from chapter 1, so a woken run has exactly one owner like any other. The pause is a durable state of a run rather than a separate system.

One field in that record is worth reading against the chapter that produced it. effect_key holds the idempotency key the charge will be made under, minted at pause time so that the approved action and the key that guards it are recorded together. That is correct and it is not protection, for the reason the last section of this chapter gives: by Thursday the gateway has forgotten the key. What the recorded key is good for is the lookup — it names the effect precisely enough for the ledger and the provider's metadata search to answer whether it happened. A field that looks like a defence turns out to be a handle, and knowing which it is decides whether the resume path may rely on it.

Deciding what happens when nobody answers#

The approval that never comes is the normal case rather than the edge case. A design with no answer for it leaves runs waiting forever, holding stock allocations at suppliers and quotes that expire underneath them.

There are three answers and the choice is a product decision made in advance, written into the pause record as on_expiry so the sweeper does not have to consult anybody at 03:00.

Expiry with automatic denial suits low-value, reversible actions where doing nothing is safe; the run cancels the order, releases the supplier allocation and tells the buyer. Escalation suits the case where the action must happen: at the deadline the request moves to the approver's manager and the timer is reset. Cancellation is different from both because it comes from outside — the buyer withdraws the order while it is waiting — and it needs the same conditional-update discipline as everything else, since a withdrawal and an approval can arrive in the same second.

That race is worth being precise about. The sweeper fires at exactly fire_at, the approver clicks at fire_at minus 200 ms, and both paths try to move the pause out of waiting. Making every transition a conditional update on the current status gives exactly one winner:

UPDATE run_pause
SET status = 'approved', decision = $1, decided_at = now()
WHERE pause_id = $2 AND status = 'waiting';
-- rowcount 0 means someone else moved it first; show the approver what happened

This is the epoch trick from chapter 1 with status in place of the counter, and it works for the same reason. Whichever statement commits first changes the status away from waiting, so the second statement's condition no longer matches, it modifies zero rows, and its caller learns it lost by reading the count. Neither path needs to know the other exists.

The loser has to produce a sensible outcome for a human who just clicked a button, which means the UI reports that the request expired two seconds ago rather than showing a generic error. That is a small piece of work and it is the difference between a control people trust and one they route around.

The three days in the middle are not free#

A pause is a window during which the world changes, and everything the run recorded on Monday is a claim about Monday. The general form is worth naming before the three instances, because it generalises chapter 2's rule: every recorded value has a validity horizon, meaning a period after which the fact it captured may no longer hold. Recording a value freezes what the run believes; it does not freeze what is true. So a pause of a given length obliges the resume path to re-check every recorded value whose horizon is shorter than that length, and the design question is which values those are.

Three of them go stale here, on three different clocks owned by three different parties.

The quote is the obvious one, and its horizon belongs to the supplier. The supplier's price moved 2.4% overnight in eleven of the 340 approvals sampled that quarter, and in two cases the quoted validity window had passed before the approver clicked. The resumed run therefore re-reads the price and compares it against proposed_action.amount_cents, and a difference beyond a declared tolerance sends the pause back to waiting with a fresh request rather than charging a number nobody approved. That revalidation loop is the arrow in the diagram above, and it is the reason the amount had to be in the record in the first place.

The idempotency key is the second, and its horizon belongs to the payment gateway. Chapter 4 named it: payment gateways forget keys within a day, and Stripe's window is 24 hours. A charge whose key was minted on Monday and replayed on Thursday gets no protection from the key at all, because the provider no longer remembers it and treats the request as new. So the resume path leans on the effect ledger and a lookup by metadata[run_id] before it issues anything, and it mints a fresh key for the new attempt. Teams that learn about durable pauses before they learn about key expiry build a system that is correct for two-hour approvals and quietly wrong for weekend ones — which is the worst available shape, since the short pauses are the common ones and the system will look fine for months.

The credential is the third, and its horizon belongs to your identity provider. A run holding a delegated OAuth token — one a named person authorised the agent to use on their behalf — with a one-hour lifetime cannot present it on Thursday, and a run holding a long-lived service credential can present it and should not, because the person who delegated the authority may no longer have it. Those two failures point in opposite directions: the short-lived token fails loudly and the long-lived one succeeds when it should not. Re-minting from the recorded grant and re-checking the grant's validity converts an implicit assumption into a check that can fail loudly in both cases.

A queue message is not a waiting state#

The pattern that gets proposed instead is a queue message with a long visibility timeout, and it is worth naming so it can be dismissed for this purpose rather than left as a nagging alternative. It is tempting for a good reason: the message is genuinely durable, and while it is invisible the work does look safely parked. What stays constant is that a message exists and is hidden; what varies underneath is everything about the wait, and none of it is recorded anywhere.

The concrete limits follow. Amazon's Simple Queue Service, SQS, caps the visibility timeout — the window during which a message handed to one consumer is hidden from every other consumer — at 12 hours, so a message held invisible for three days is not expressible. Holding one for even 12 hours occupies an in-flight slot, and it still tells you nothing about what the run was waiting for, who may satisfy it, or what should happen if nobody does. A queue can carry the wake-up signal, and it cannot be the waiting state.

It is not a scheduled job that scans for stale rows and guesses what to do with them, either. The sweeper above executes a decision that was made when the pause was created, which is why on_expiry is a stored field rather than a rule in the sweeper's code. The difference shows up on the day the policy changes: a rule living in the sweeper applies to every row it sweeps, including rows created months earlier under the old policy, so changing the code retroactively changes what those pauses do. In an approval system that is a compliance problem rather than a refactoring preference, because the pause is a record of an authorisation made under a stated policy version.

It also does not make the human faster or more attentive, and it is worth being clear that the mechanism solves durability rather than latency. Runs that wait 41 hours at the 95th percentile hold real resources, and the useful responses are product ones: raising the approval threshold, batching approvals into a single morning digest, or letting the agent proceed on the reversible part of the order while only the payment waits. Durable pauses make those options available by ensuring a waiting run still exists tomorrow, and they do not choose among them.

By this point the system has a leased checkpoint, recorded nondeterministic values, a recovery boundary with three states, keyed effects with a ledger, durable timers and a pause table with expiry policy. All of it was written by hand, against a queue that delivered a message and retried it. That inventory is the right thing to hold up against what a queue actually promises, because the gap between the two is precisely the product a durable engine is selling.