Checkpoints and idempotent recovery

A checkpoint here is a durable record of what an agent intended and what happened, written to a store outside the agent's process. It exists because the alternative record is the model's context window, which holds a complete and accurate account of the run right up to the moment the process dies, and then does not exist. Anything that stops an agent — an out-of-memory kill, a deployment, a timeout, a kill switch someone pulled deliberately — leaves three questions behind: how far did it get, what already happened in the outside world, and is re-running safe.

Those three are unanswerable after the fact unless the design answered them beforehand, which is what makes an interruption a design problem rather than an operational one. The subject here is the recovery half of that problem; the verification half — whether a run that did complete was right — is a separate class of control, covered earlier in this topic.

A journal written before the action, not after#

The record is a durable statement of intent, written before each action and completed after it. Its two-phase shape is what makes the ambiguous case visible: an entry with an intent and no outcome is a step that may or may not have happened, and knowing precisely which step that is turns a forty-file investigation into a one-file investigation.

Only one step can ever be in unknown, because the agent writes one intent at a time and does not begin the next until the previous outcome is recorded. That constraint costs a round trip per step — about 15 ms against a local Postgres — and it is what bounds the ambiguity to a single action.

That claim is the one piece of reasoning the rest of the chapter applies, so it is worth proving rather than accepting. Suppose two entries lacked outcomes at the same instant. Every entry has its intent written before its action and its outcome written after, so the later of the two had its intent written at some moment, which means the agent had started that step. But the agent does not start a step until the previous step's outcome has been recorded. So the earlier entry must already have carried an outcome, which contradicts the supposition that it lacked one. Two is impossible, and one is therefore the most the journal can ever be holding. What that licenses is a bound with the plan's length divided out of it: a run of ten steps and a run of ten thousand both leave at most one action in doubt, so the work of resolving an interruption is a constant rather than something that grows with the size of the job.

Price the constraint while the proof is fresh, because the ratio is the argument for paying it. About fifteen milliseconds a step, over a plan of forty-two steps, is well under a second of machine time spread across the whole run. The alternative, measured on a real morning later in this chapter, is about four hours of a person's attention and a telephone call to a supplier. It is one of the few controls in this topic whose cost and whose benefit can be stated in the same sentence without either of them needing an estimate.

import json, hashlib, datetime as dt

def key_for(task_id: str, step: int, action: str, args: dict) -> str:
    payload = json.dumps([task_id, step, action, args], sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

def run_step(db, task_id: str, step: int, action: str, args: dict, execute, probe):
    k = key_for(task_id, step, action, args)
    row = db.fetch_one("SELECT * FROM journal WHERE idem_key = %s", k)

    if row and row["outcome"] is not None:
        return json.loads(row["outcome"])            # already done, skip

    if row and row["outcome"] is None:               # died mid-flight
        observed = probe(args)                       # ask the target system
        if observed is None:
            db.execute("DELETE FROM journal WHERE idem_key = %s", k)
        else:
            db.execute("UPDATE journal SET outcome=%s, resolved_by='probe' "
                       "WHERE idem_key=%s", (json.dumps(observed), k))
            return observed

    db.execute(
        "INSERT INTO journal (task_id, step, idem_key, action, args, before_state, started_at) "
        "VALUES (%s,%s,%s,%s,%s,%s,%s)",
        (task_id, step, k, action, json.dumps(args), json.dumps(probe(args)), dt.datetime.utcnow()),
    )
    result = execute(args)
    db.execute("UPDATE journal SET outcome=%s, finished_at=%s WHERE idem_key=%s",
               (json.dumps(result), dt.datetime.utcnow(), k))
    return result

The probe argument is the part that carries the design weight, because it is what lets a resumed run establish truth from the target system rather than from its own memory. For a rename it queries Drive for a file with the intended new name and returns its metadata or None. For an email it looks up the provider's message list filtered by a header the send step stamped with the idempotency key. Every action in a plan needs a probe, and writing them is the exercise that reveals which actions are unverifiable — those are the ones to redesign, because an action whose completion cannot be established after the fact is one that can never be safely resumed.

Idempotency then falls out of the journal for most actions and has to be constructed for a few. Renaming is naturally idempotent when it is expressed as desired state — "this file should be called X" is safe to apply twice, whereas "append _v2 to this file" is not — and expressing steps as target states rather than as deltas is a free win taken at planning time. Sending mail has no such property and no idempotency key — a caller-supplied identifier that a service uses to collapse a repeated request into the first one — in the Gmail or Microsoft Graph APIs, so the mechanism is the ledger check plus a custom header carrying the key, which makes the probe possible and makes a duplicate detectable even if the network failed in the worst place. Payments and API writes that do support idempotency keys, such as Stripe, should be handed the journal's key directly rather than a freshly generated one, so that a resumed run and its predecessor collide by design.

Why one of those is safe twice and the other is not comes down to a single property. "This file should be called X" names an end state, and an end state that already holds is unchanged by asking for it again — the second application finds nothing left to do. "Append a suffix to this file" names a transformation of whatever happens to be there, so the second application runs on the output of the first and produces something neither the plan nor the operator ever asked for. Sending mail is a delta in exactly that sense. There is no end state "this message has been sent once" that the provider will check on your behalf, so a second send is a second message rather than a no-op. That is why the mechanism has to be supplied from outside — a ledger row plus a stamped header — and why writing the plan in end states at the moment it is generated costs nothing while retrofitting the property afterwards costs a redesign.

What the same failure looks like with the journal in place#

Run a forty-file rename and two emails, kill the process partway, and the journal holds 32 rows: 31 renames with intents and outcomes, one rename with an intent and no outcome, and nothing beyond step 32, because the emails were steps 39 and 40 in a plan of 42.

Every entry in that right-hand column comes from arithmetic on two numbers, which is what makes it a record rather than a reconstruction. The journal holds thirty-two rows; thirty-one of them carry outcomes and one does not; those add back to the row count with nothing left over, so "thirty-one confirmed, one to probe" is counted rather than inferred. The email row is the one to dwell on. It says provably none, and the proof is a comparison of two step numbers: the last row in the journal is step thirty-two, the first irreversible step in the plan is thirty-nine, and thirty-two comes before thirty-nine. No mailbox was opened to establish that and no provider was asked. It follows from an ordering fixed before the run began.

Set that against what the same question cost on the morning described later in this chapter, when it was put to the mail provider instead. The sent-mail folder was empty, which looked conclusive and was wrong, because the send step used a queueing endpoint and a queued message is not a sent message. The folder was answering a question about delivery while the question on the table was about submission. Two instruments disagreed, and the habit worth forming is to ask what each one measures before deciding which to believe. The journal records what the agent attempted, which is the quantity that decides whether a re-run can duplicate anything. The sent folder records what the provider has finished with, which lags, and can lag indefinitely. Where the two can disagree, the record written before the action is the one to build on — which is the entire reason the intent is written first rather than the outcome being written twice.

The ordering of the plan is a second lever that costs nothing and is rarely used. Emails, payments and anything else irreversible sat at steps 39 to 42 in this plan because somebody put them there deliberately, so an interruption anywhere in the first 38 steps could not have sent anything. Sorting a plan so that reversible work happens first and irreversible work happens last, after the verification gates have passed, converts most interruptions into resumable ones by construction. Plans that interleave sends with renames get the opposite property, and the difference is a sort order.

The kill switch and the person who decides#

A kill switch has to work when the agent is not cooperating, which rules out the version most teams build first — a flag the agent checks between steps, which is fine for an agent behaving normally and useless when the agent is stuck in a loop or when the concern is the model's judgement rather than its liveness. The mechanisms that hold are outside the process: revoke the short-lived credential at the broker, so the next call fails regardless of what the agent intends; kill the container; drop the egress rule. The in-band flag is worth having as well because it produces a clean stop with the journal in a tidy state, and it should be the first thing tried, with the out-of-band mechanisms behind it.

The failing version deserves a diagnosis rather than a dismissal, because it is what everybody builds first and it is not a foolish thing to build. A flag the agent checks between steps is a control whose operation depends on the agent checking it, which makes its reliability a function of the very thing it is meant to control. Run the two cases. Where the agent is healthy and somebody wants a tidy stop, the flag works and works well. Where the agent is looping, wedged on a page that never loads, or doing something the operator disagrees with, the flag is read late or not at all. Those are precisely the two cases the switch exists to tell apart, and the flag is at its most reliable in the one where it is least needed. The out-of-band mechanisms have the opposite dependency. Revoking the short-lived credential at the broker, killing the container and dropping the egress rule are all performed by components the agent has no influence over, so their behaviour does not vary with the agent's state at all. That independence is the property being bought, and it is the reason the in-band flag belongs first in the runbook and never in the argument that the system can be stopped.

Whatever stops the agent must also stop whatever restarts it. The nightly scheduler in a supplier-sync design would otherwise start run 123 on top of the wreckage of run 122, and a killed run that reappears an hour later is the version of this that turns a contained incident into a repeated one. The stop has to set a state that the scheduler reads, and the default when nobody has looked is to stay stopped.

The rollback owner is a person on a rota with three properties: the authority to decide, a runbook that tells them how to read the journal and resume or unwind, and the ability to reach whoever owns the affected system at three in the morning. "The team" is not an owner, because a Slack channel at 03:14 produces no decision. The decision they make is one of resume from the checkpoint, unwind the completed steps using the before_state the journal recorded, or stop and hand it to daylight — and the third option is legitimate and underused, because a half-renamed folder is usually not urgent while a hasty unwind at 03:30 can be.

Testing the path before you need it at 03:14#

A recovery path that has never been exercised is a design document. The test is mechanical and takes an afternoon: run the task against a staging target, kill the process at a uniformly random step, resume, and assert that the end state matches the end state of an uninterrupted run and that no action was performed twice. Repeat it fifty times in CI and it will find the two or three steps whose probe is wrong, which is the usual result, because probes are written from an assumption about what the target system returns and some of those assumptions are false. On one document agent this found that the Drive probe matched on filename alone, so a resumed run treated a file the supplier had independently renamed as evidence that its own rename had succeeded.

The plan itself has to be durable alongside the journal. An agent that regenerates its plan on resume produces a different sequence of forty-two steps, and step 32 of the new plan is not step 32 of the old one, which makes the journal's step numbers meaningless. Persist the plan when it is produced, resume against the stored version, and treat a change in the plan as a new task with a new identifier. The plan is also what makes the audit trail readable afterwards, since a list of clicks and keystrokes without the intent they were serving is difficult to reconstruct six weeks later when somebody asks what happened to a specific file.

What a recovery path does not decide#

None of this makes the actions correct. A journal will faithfully record 31 renames to the wrong scheme, resume cleanly, and complete the remaining nine. Verification decides whether a run should resume; the recovery path only decides whether it can, so the resume path has to re-run the precondition gates before continuing rather than trusting that they passed twelve hours ago.

Idempotency is also not a transaction. There is no atomicity across Drive, a mail provider and an ERP, so partial states are the normal outcome of an interruption and reconciliation is how they are closed. Teams that reach for two-phase commit here, the protocol in which every participant promises to commit before any of them does, spend weeks and arrive back at a journal plus probes, because the third-party systems will not participate in anything else.

And a kill switch stops future actions rather than in-flight ones. An HTTP request already accepted by the mail provider will be delivered, and revoking the credential a second later changes nothing about it. This is a further argument for ordering irreversible actions last: the window in which the kill switch is ineffective is exactly the window in which the irreversible things happen, and a plan can control when that window opens.

Whether an interruption becomes an incident is decided before the run, by whether anything durable was written outside the process. A two-phase journal costs a round trip per step and buys a named stopping point, a bounded ambiguity of exactly one action, a probe that resolves it against the target system rather than against the agent's memory, and a re-run that is provably safe from that point on.

Everything in this topic converges on a small set of artefacts that can be produced for a real design in about a week. The containment statement says what a code-execution tool reaches, with a probe asserting it in CI. The authority map says what a browser agent inherits, which is the enumeration exercise rather than a susceptibility rate. The costed comparison says whether the capability beats the feed, the scraper or the twenty minutes. The grant records say what is permitted, brokered and expiring. The verification gates say whether a completed run was right. The journal says where an interrupted one stopped.

The order to build them in is not the order they were taught. Start with the journal, because it is the cheapest of them and every other investigation depends on the record it leaves — enumeration, verification and post-incident reconstruction are all reading it. Then the containment statement and the probe, which take an afternoon and settle the question a review will ask anyway. The grants and the verification gates take a week between them and are where the argument with other teams happens, so they benefit from arriving after the first three have made the exposure concrete. Take the design currently sitting closest to production, write its containment statement this week, and see how much of it you can fill in without asking anybody.