A ledger records intent; only the tree records what happened

Step five of the auth extraction takes about ninety seconds: two files edited, go build ./... at nine seconds, go test ./internal/auth/... at twenty-two. Somewhere inside that ninety seconds the harness kills the process, and where inside it decides what the ledger says.

The implementation writes the status after the check passes, which is the ordering most people choose because it is the one that means anything. A kill at second eighty-nine leaves the edits applied, the check passed, and "status": "in_progress" on disk. The resumed run reads that, retries step five, and re-registers the route — the panic from the previous chapter, arrived at through a durable file rather than through a lost conversation.

Reverse the ordering and the error reverses with it. Writing "status": "passed" before running the check, or marking the step optimistically as soon as the edits are dispatched, produces a kill at second forty that leaves a ledger claiming success over a tree where one of two files was written. The resumed run skips step five and builds five subsequent steps on top of a half-applied change.

There is no third ordering. The edits and the flag are two writes to two different places with no transaction spanning them, so a kill in the window between them is misrecorded, and the only choice available is which direction the mistake goes.

The window is thirty-one seconds wide on a ninety-second step, which is a third of the step spent in a state the ledger describes incorrectly. Shrinking it is possible and pointless; a window of two hundred milliseconds is still a window, and long runs are killed thousands of times across a fleet.

Ask the tree, not the file#

The repair is to stop reading status as a fact. Every step already carries a command that decides whether it worked, and that command runs against the working tree, which is the only artifact that survived the kill. Status becomes something the resume computes rather than something it loads.

def reconcile(ledger, run):
    """Rebuild step status from the tree. Returns the first uncertified step."""
    for step in ledger["steps"]:
        if not touched_since(step):          # tree unchanged where this step lives
            step["status"] = step.get("status", "unknown")
        certified = True
        for cmd in step["check"]:
            result = run(cmd)
            step.setdefault("observed", []).append(
                {"cmd": cmd, "exit": result.exit, "secs": result.secs})
            if result.exit != 0:
                certified = False
                break
        step["status"] = "passed" if certified else "uncertified"
        if not certified:
            return step
    return None                              # every step certified; task is done

Running this against the killed run produces the answer the ledger could not. Step five's check passes, because step five genuinely completed, so the resume does not re-register the route. Step seven's check fails, because four of its six files were rewritten and internal/api no longer compiles, so the resume does not skip it. Both of the previous chapter's failures are corrected by the same mechanism, and neither correction required knowing when the kill happened.

The touched_since guard is the one piece of the loop that is allowed to skip work, and it is worth being precise about why it does not reintroduce the problem it is avoiding. It asks git whether anything under the step's touches paths differs from the tree the step's evidence was recorded against:

git diff --quiet 8f3c1ad -- internal/auth/register.go cmd/api/main.go

That is a question about the tree, answered by the tree, and its answer is a fact rather than a claim. A step whose files are byte-identical to the state its check passed against does not need the check re-run. This matters because reconciliation is not free: twelve steps at roughly forty seconds of checking each is eight minutes before the resumed run does any work, and on a repository where the scoped suites are slower — a Java monolith where a module's tests take four minutes — a naive full reconciliation costs the better part of an hour. With the guard, a resume of a run killed at step eight re-runs the checks for the two or three steps whose files were plausibly disturbed and skips the rest, which brings the cost to about a minute.

What half a step leaves behind#

The concrete case is worth walking, because the design decision it forces is not obvious. Step seven migrates token-parsing callers across six files. The kill lands after four.

$ git status --short
 M internal/api/handler_login.go
 M internal/api/handler_refresh.go
 M internal/api/handler_logout.go
 M internal/api/middleware.go
$ go build ./...
internal/api/session.go:22:14: not enough arguments in call to parseSessionToken
internal/api/admin.go:41:9: not enough arguments in call to parseSessionToken

Four files speak the new signature, two speak the old one, and the package does not compile. Reconciliation reaches step seven, runs go build ./..., gets exit 1, and returns step seven as the first uncertified step. That is the correct diagnosis, and it leaves the resumed run holding a partially mutated tree.

Two options exist and one of them is a trap. Continuing forward — inspecting the four modified files, working out what remains, finishing the other two — is tempting because it preserves work, and it puts the model back in the position that caused every failure so far, inferring from ambiguous evidence what a previous process intended. The four modified files might be correct, or one of them might be the reason the step was going to fail anyway.

The safer default is to discard back to the step's rollback boundary and re-run the step cleanly:

git checkout s06 -- internal/api/

The cost is bounded by the step, which is the point of having made steps small enough to certify. Redoing step seven costs the four minutes it was going to cost anyway. Where a step is genuinely expensive — a codemod, meaning one scripted mechanical edit applied across the tree, run over eight hundred files and taking forty minutes of machine time — the partial work is worth keeping, and the way to keep it without contaminating the retry is to preserve it somewhere the retry cannot see: git stash push -m "s07 partial @ 09:41" or a scratch branch named for the run, referenced from the ledger, discarded when the step eventually certifies.

Making the tree carry the record#

The touched_since guard needs a tree reference per step, which raises the question of where those references come from and whether they survive the workspace being destroyed. Both problems close if each certified step ends in a commit, and the commit carries the evidence in trailers:

git commit -m "s05: move session middleware registration into auth.Register" \
  -m "Agent-Run: r-4471" \
  -m "Agent-Step: s05" \
  -m "Agent-Check: go test ./internal/auth/... -count=1 exit=0 secs=22.6" \
  -m "Agent-Plan-Version: 3"

The branch is pushed after each step, so the durable record of progress is the same object every engineer on the team already knows how to read. A resumed run whose container is brand new — no ledger file, no scratch directory, nothing but a clone — rebuilds the ledger's skeleton from history:

$ git log --format='%h %(trailers:key=Agent-Step,valueonly)%(trailers:key=Agent-Check,valueonly)' \
      --reverse origin/agent/r-4471
3b91e0c s02  go build ./... exit=0 secs=9.1
7c02f14 s03  go test ./internal/auth/... -count=1 exit=0 secs=19.8
8f3c1ad s05  go test ./internal/auth/... -count=1 exit=0 secs=22.6

Reading that log tells you which steps once passed and against which trees, and reconciliation still re-runs the checks rather than trusting the trailers, because a trailer is another claim written by the same process. What the commits give that a status field cannot is a set of tree references that are certain — the tree at 8f3c1ad is exactly what the check ran against, by construction, because the commit is that tree.

Notice that step four is absent from the log. Step four's check passed and its commit failed to write, or the push failed, or it was folded into step five's commit by an implementation that batched them. Reconciliation handles this without special-casing: with no recorded tree for step four, touched_since returns true, the check runs, and step four is certified or not on the evidence of the moment.

Why not make every step idempotent instead#

The standard alternative, proposed by anyone who has built retry-safe job queues, is to sidestep the whole question: write every step so that applying it twice has the same effect as applying it once, and then a resume can simply run everything from the beginning without caring what already happened.

It works for some steps and fails for the ones that matter. A codemod expressed as a fixpoint — rewrite every call to parseSessionToken(x) into parseSessionToken(x, policy) unless it already has two arguments — is genuinely idempotent, and steps of that shape are the easy half of a migration. The other half is not. Step five moves a registration from one file to another; running it again against a tree where it already ran means looking for something that is no longer there, and the implementations that handle that gracefully are the ones that check first, which is reconciliation wearing a different name. Deletions are worse: a step that removes a deprecated package, re-run after the package is gone, either errors or silently succeeds, and neither tells you whether the deletion you wanted happened or whether the package was never there.

Idempotence also only addresses one of the two directions. It makes re-running a completed step harmless, which fixes the duplicate-registration failure. It does nothing about a resumed run skipping a step it wrongly believes complete, because a skipped step is never run at all and idempotence has no opinion about work that does not happen.

The two combine well, which is the practical position. Where a step can be written as a fixpoint cheaply, write it that way, because it makes the cost of a wrong reconciliation decision lower. Where it cannot, reconciliation is the mechanism, and the effort that would have gone into contorting the step into idempotence goes into making its check fast and scoped instead. Between the two, the check is the one you cannot do without, since it is also what tells you the step succeeded.

The limits of reconstructing from checks#

A reconstructed status is exactly as trustworthy as the checks that produce it, in two directions that fail differently.

Checks have to be deterministic. A step certified by a suite containing one test that fails four times in a hundred on a timing race gives reconciliation a four percent chance of deciding, on any given resume, that a completed step needs redoing. That converts a flaky test from an annoyance into a source of duplicated edits, which is a far more expensive failure than a red build, and it is an argument for holding certifying checks to a stricter standard than the rest of the suite — hermetic, no network, no clock dependence, no shared fixtures.

Checks also have to be scoped tightly enough that they answer about their own step. A step certified by the full suite is certified by a command that fails for reasons belonging to other steps, so reconciliation attributes an unrelated breakage to the earliest step whose check happens to include it, and the resume rolls back work that was fine. Scoping is what makes the first-failure-wins rule in reconcile sound.

None of this establishes correctness. Reconciliation determines which steps reached the state their checks describe, and the previous chapters' ceiling still applies: a step whose check is a build and 41 passing tests can be certified while having dropped a header-casing quirk that nothing tests. Reconstructing status from the tree removes an entire class of bookkeeping error and leaves the verification problem exactly where it was.

The mechanism has a quieter implication that is easy to miss. Everything here has been framed as recovery — something killed the run, and the resume reconstructs. But the run that spends four hours and three hundred thousand tokens on twelve steps has also lost its early context by step nine, with nobody killing anything, and the model at step nine is in a position uncomfortably similar to the resumed run's: working from a plan it barely remembers writing. That suggests the reconstruction should not wait for a failure.