A three-day wait cannot live inside an in-memory await

An approval gate is a check placed in front of one action that suspends a run, sends a request to a person, and lets the run continue only when a decision comes back. The interval between those two events is the subject here. For a gate on an outbound reply it is seconds or minutes. For cancel_subscription — the action that ends a customer's contract, so their API keys stop working and every seat on the account loses its dashboards — a team reasonably gives an operator three business days, because the action is rare, consequential, and not one anybody should decide in a hurry. Three days is an ordinary thing to ask of a person and an impossible thing to ask of a process.

A request that has been issued and not yet decided is pending, and the design question is where that pending state is kept: on a stack inside a process that deploys and autoscaling will replace within hours, or somewhere with a longer life than they have. Over its first month one such gate produced ninety-six requests, and fifty-one of them reached no terminal state at all — no approval, no rejection, no expiry, no escalation, and no error in any log.

A wait of days inside a process that lives hours#

Two durations are in play, and the shorter one decides the outcome. The window is 72 hours. Comparing it against how long a replica lasts needs a model of how replicas die, and the useful one is the exponential lifetime: the assumption that a process is ended by an event arriving from outside at a roughly constant rate, rather than by wearing out.

That assumption licenses everything below it. A pod here is ended by a deploy or by a scale-in decision, and neither of those consults the pod's age. A replica that has been up for six hours is no likelier to be replaced in the next minute than one that came up thirty seconds ago, since what ends it is a release or a traffic change happening to the whole fleet. That property — the chance of surviving another interval does not depend on how long you have already survived — is exactly what the exponential distribution encodes, which is why it is the right model here rather than a convenient one. Something that failed by accumulating wear would need a different one.

Under that model the chance a process is still alive after some interval is the exponential of minus that interval divided by the mean lifetime, which is the average time a replica survives before something replaces it. Both quantities are in hand. The service deployed four to six times a day, and horizontal autoscaling recycled pods through the afternoon peak. The mean pod lifetime was 3.4 hours. Treating that as an exponential lifetime, the probability that any given pod survives 72 hours is exp(-72/3.4), which is about seven in ten billion.

The substitution teaches more than the answer does. Seventy-two hours divided by a mean lifetime of 3.4 hours is a ratio of a bit over twenty, so the window is a bit over twenty mean lifetimes long, and each mean lifetime that passes multiplies the chance of still being alive by the same fixed factor — a shade over a third — because that is what a constant hazard rate does. Twenty-odd multiplications by roughly a third is the seven-in-ten-billion figure. The exponent does all of the work, which is why the conclusion survives a large error in the mean: doubling 3.4 hours would still leave the window more than ten mean lifetimes long.

So this is not a gate that usually works and occasionally loses a request. The gate was not unreliable, and it was not a race condition; a three-day wait implemented inside a process with a mean lifetime of 3.4 hours will never complete, and the only surprising number in the incident is the 41 approvals that did land, all of which were answered within the first three hours. Those forty-one are the confirmation rather than the exception, since three hours is inside a single mean pod lifetime and that is precisely the subpopulation the model says survives. Requests that landed and requests that vanished are one distribution, cut where the hosting process stops existing.

What the implementation looked like#

pending: dict[str, asyncio.Future] = {}

async def request_approval(run, tool: str, args: dict) -> Decision:
    digest = payload_digest(run.id, tool, args)
    fut = asyncio.get_running_loop().create_future()
    pending[digest] = fut                       # lives in this process only
    await notify_owner(run, card_for(tool, args))
    return await asyncio.wait_for(fut, timeout=3 * 24 * 3600)

@app.post("/approvals/{digest}")
async def decide(digest: str, decision: Decision):
    pending[digest].set_result(decision)        # KeyError on any other replica

Four things are wrong with that, and only one of them is the timeout. The dictionary is per-process, so a decision routed to a different replica raises KeyError and the operator sees an error page after doing the right thing. The awaiting coroutine holds the run's entire state — the message list, the partial results, the plan — on a stack that no other process can read. The wait_for timer is itself in-process, so the expiry policy from the previous chapter — a window, a rule for what happens when it closes undecided, and a named owner — is only enforced if the process lives long enough to enforce it, which is precisely the case where it is not needed. And the memory is held for the duration: at 200 concurrent pending approvals with transcripts averaging 60 KB, that is 12 MB of resident state that cannot be evicted, plus a task per approval, in a process that is also serving traffic.

Two hundred approvals each carrying a transcript of sixty kilobytes is two hundred times sixty kilobytes, which comes to twelve thousand kilobytes — the twelve megabytes above. What makes an otherwise unremarkable number matter is that none of it can be released while the futures are outstanding, so it is a floor the process carries rather than a peak it touches, and it grows with the number of people who have not answered yet.

The suggestion that gets made first in these reviews is to raise the timeout and add a retry. Raising a timeout does nothing about a process that is not there, and a retry makes the situation worse: the retry re-runs the run from the beginning, re-issues a card for an action a human may have already approved, and produces a second effect if the first one landed. The failure is structural, in that the wait outlives the thing implementing it, so the fix has to move the wait somewhere with a longer life than the process.

Both suggestions are one-line changes to a configuration file, and both are the correct treatment for the failure this one resembles: an operation that took longer than expected and then gave up. The timeout is a quantity that stays constant while the dangerous property varies. A team can move it from three days to thirty and the survival arithmetic above does not shift at all, because the timer the timeout configures runs inside the process whose absence is the failure. What actually decides the outcome — how long a replica lives — moves with deploy cadence, with traffic, and with the scale-in policy, none of which appear in the file where the timeout is set. A defence tuned on one axis while the danger moves on another is not a weak defence; it is not connected to what it is defending against.

The retry's three steps are each worth following. Re-running from the beginning prompts the model again, which reaches the gate again, which puts a second card in front of an operator who has already seen that payload — the situation the previous chapter's rule against rendering reminders as cards exists to avoid. And if the first attempt's effect reached the billing system before the process went away, nothing in a retry knows that, so the effect happens twice.

Persist the pending state, let the process exit, resume from storage#

The durable version splits one long-lived function into three short-lived ones: the worker that reaches the gate and stops, the endpoint that records a decision, and the worker that resumes the run afterwards. Nothing in the system is awaiting a human.

A requirement and a design decision are tangled together in that diagram. The requirement is forced by the arithmetic: the pending state must outlive the process, because the process will not be there. Everything after it is chosen. That the cursor and the transcript live in a table in your own database, rather than in a durable execution engine's store or a graph framework's checkpointer, is a design decision, and all three satisfy the requirement. Which is which matters when somebody proposes a different store, because a design decision can be reopened on cost and a requirement cannot.

The row is the unit of durability, and what goes in it is everything needed to reconstitute the run.

CREATE TABLE approvals (
  id            uuid PRIMARY KEY,
  run_id        uuid NOT NULL,
  digest        text NOT NULL,              -- binds tool + resolved args
  cursor        jsonb NOT NULL,             -- step index + loop position
  transcript    jsonb NOT NULL,             -- the message list to resume from
  preconditions jsonb NOT NULL,
  card          text NOT NULL,              -- exactly what the operator saw
  owner         text NOT NULL,
  state         text NOT NULL,              -- pending|approved|rejected|expired|superseded
  decided_by    text,
  decided_at    timestamptz,
  expires_at    timestamptz NOT NULL,
  UNIQUE (run_id, digest)
);
CREATE INDEX ON approvals (state, expires_at) WHERE state = 'pending';

The decision endpoint's whole correctness lives in one statement, because the interesting cases are all races: a double-clicked button, a webhook the notification platform retries, an approval arriving in the same second the expiry sweeper fires.

async def decide(approval_id: str, decision: str, actor: str) -> str:
    rows = await db.execute(
        """UPDATE approvals
              SET state = $2, decided_by = $3, decided_at = now()
            WHERE id = $1 AND state = 'pending'
        RETURNING run_id""",
        approval_id, decision, actor,
    )
    if not rows:                       # already decided, expired or superseded
        current = await db.fetch_state(approval_id)
        return f"no change — this request is already {current}"
    await queue.enqueue("resume", run_id=rows[0]["run_id"], key=approval_id)
    return "recorded"

A conditional update whose WHERE clause includes the expected state gives the state machine from the previous chapter — pending, moving on to approved, rejected, expired or superseded — a single writer per transition, and the row count tells you whether you were the one who made it. The expiry sweeper runs the same shape of statement against state = 'pending' AND expires_at < now(), so whichever of the two arrives first wins and the loser gets an accurate message instead of a second effect. The key=approval_id on the enqueue is what stops two deliveries of the same webhook producing two resumes.

Being exact about what that statement licenses matters, because it is easy to credit it with more. Several writers may reach the row at once, and each issues an update that matches only while state is still pending. The database serialises them, so exactly one finds that condition true and the rest find it false — made false by the first writer's own update. The licence is a single claim: exactly one transition out of pending ever happens, and the row count tells its caller whether that transition was theirs. The residual is what the claim leaves untouched. The statement makes the decision happen once, and says nothing about the resume happening once, which is the queue key's job, or about the effect happening once, which belongs to the idempotency key derived from the digest.

What has to be serialisable, and what usually is not#

Resuming from storage requires the run's continuation to be data. The hand-built loop from the earlier topic — call the model, run whatever tool it asked for, append the result, go round again — makes this easy, since its entire state is a list of messages plus a step counter, and both serialise to JSON without ceremony. What does not serialise is a Python stack: local variables in a partially executed function, an open database transaction, a generator suspended mid-iteration, a closure capturing a client object. Any design where the gate is a function call in the middle of a longer function has that problem by construction, which is why durable implementations tend to look like state machines with an explicit cursor rather than like straight-line code.

That table reads as gaps rather than as levels, and the rows are not ranked. The column that decides is the second one, which splits the six rows into a group that cannot do the job at all and a group that can; every other column — where the state lives, what a three-day wait costs, what adoption costs — is a choice made inside the second group. Reading down the last column instead produces the opposite ordering and the wrong answer, since the cheapest thing to adopt is the thing that does not work. The second row shows that most clearly. Sticky routing plus a long timeout, which pins every decision for an approval back to the replica that issued it, is real work to set up and leaves behind a replica you cannot drain, and a pinned replica is still replaced like any other. That is what "Low, and it buys nothing" records: a genuine cost, and no movement in the only column that decides.

The storage cost is small enough to ignore and the storage contents are not. A transcript at the point of a gate runs from a few kilobytes on a short run to a couple of hundred on a long one, so a thousand pending approvals is tens of megabytes, which is nothing. What matters is that the blob is a copy of everything the agent read, including whatever customer data came back from the tools, now living in a table with a different retention policy and a different access pattern than the systems it came from. The workable arrangement is to store references where the data is retrievable — invoice ids rather than invoice bodies — and to give the table a retention rule that deletes decided rows after the audit window, keeping the decision record itself, which is small and is the part anyone will ask for.

Migrating an existing in-memory implementation does not have to be a rewrite, because the two can coexist for a while. Writing the row at the moment the card is issued, before touching the in-process future, gives you a durable record and a dashboard immediately, and the resume path can continue to work through the future while it exists. The row is then the source of truth for expiry and for the audit trail, and the second phase — deleting the future and resuming from the row — is a change with a known blast radius, because by then the rows have been recording what the in-memory path was actually doing. Teams that attempt both phases at once tend to discover halfway through that some of their run state was never serialisable.

LangChain's Human-in-the-loop with interrupts documents the checkpointer version of this, including the re-entrancy trap that the code after an interrupt runs again on resume; the OpenAI Agents SDK's Human-in-the-loop page does the same with a serialisable RunState you store and rehydrate. Both are worth reading before hand-rolling, because both have already enumerated the gotchas that turn up in week three of doing it yourself.

Durable execution engines such as Temporal, Restate and DBOS solve the general problem by recording an event history and replaying it to rebuild state, which imposes a determinism requirement that agent code violates naturally: a model call returns something different every time, so it has to be recorded as an event rather than re-executed during replay. That is a solved problem in every one of those systems, and it is the first thing to get wrong when adopting one.

Two steps sit under that requirement. Replay rebuilds state by running your function again from the top and feeding it the recorded results of everything it did last time, so for the rebuilt state to equal the state that existed before the crash, every step that is re-executed rather than replayed has to return what it returned originally. A model call cannot meet that condition, since it is a fresh sample and identical inputs give different text. A model call left on the re-executed side of that line therefore rebuilds a state that has already diverged from the one being rebuilt, which is the failure the whole mechanism exists to prevent. Moving it to the recorded side is the repair, and it is why each of these systems draws an explicit boundary around effects.

The parts durability does not fix#

Persisting the pending state does not decide what happens when nobody answers; it only makes the expiry policy from the previous chapter executable. A durable row with no sweeper is a run that waits forever with excellent bookkeeping, and the 51 lost runs would have become 51 rows sitting at pending for a year.

Durability also does not make effects idempotent. A resume worker that crashes after calling the payment API and before writing its result will resume again and call it a second time, and the only thing standing between that and a duplicate refund is the idempotency key derived from the approval — the same digest that binds the payload. The three mechanisms compose deliberately: the conditional update makes the decision happen once, the queue key makes the resume happen once, and the idempotency key makes the effect happen once. Removing any of the three leaves a hole that only appears under a restart at exactly the wrong moment, which is a category of bug that surfaces in production and not in tests.

The row does buy one thing beyond correctness, which is that pending work becomes queryable. A single index scan answers how many approvals are outstanding, how old the oldest is, who owns them and which action classes they belong to, and that query is the dashboard the 51 lost runs needed: an alert on any row whose age exceeds its window by more than a few minutes would have fired on the first afternoon rather than at the end of the month. Nothing equivalent is possible when the pending set is a dictionary inside one of nine replicas.

The one that surprises teams is that resuming is not the same as continuing. Three days have passed, and if the resume path re-prompts the model with the transcript plus the approval, the model may produce a different call than the one that was approved, because it is a fresh sample. This is not hypothetical for any implementation that resumes by adding a message such as the human approved your refund; proceed and letting the loop run. The execution-time digest check catches it, and the cleaner design does not create the situation: the resume executes the stored payload directly and only returns to the model afterwards with the result. The model's job ended when the gate fired.

Which of those two you build is a design decision rather than a requirement, and saying so changes the argument. The requirement is that the call executed is the call approved. The digest check meets it by detecting a mismatch after a fresh sample has produced one; executing the stored payload meets it by never generating a mismatch to detect. Only the second leaves nothing to handle when a three-day-old transcript is replayed into a model that has been upgraded underneath it.

Preconditions are the last thing the resume has to honour, for the same reason. Three days is long enough for the invoice to be paid, the subscription to have been cancelled by the customer, the account to have been closed, or the ticket to have been merged. Revalidating at resume converts those into a superseded outcome with an explanation, rather than into an action that was correct on Tuesday.

The gate that lost fifty-one requests was correct in every respect a design review inspects: a window, an expiry action, a named owner and an escalation chain, each enforced by code that worked when it ran. What it lacked was anywhere for a three-day wait to live, and the comparison that settles whether that matters is available before a line is written — how long the window is, against how long the thing implementing it survives. Where the second number is the smaller one, the wait belongs in a row rather than in a stack frame, with the run's cursor, transcript, preconditions and card alongside it. The window, the owner and the expiry action become enforceable at exactly that point, and are decoration before it.

With the wait durable and the decision recorded, the gate now produces something it did not before: a stream of decisions, each with an actor, a latency, an outcome and a card. That stream is the only evidence available about whether any of this is working, and the numbers it produces are read wrongly more often than not.