A step nothing can verify is a failure nobody can attribute

The task is a client migration. Forty call sites across twelve Go packages move off internal/httpx, an in-house wrapper around a transport library that stopped receiving security patches in March, and onto pkg/transport, which the platform team shipped last quarter. The agent is given the repository, a two-paragraph ticket, and a budget of twenty attempts. It produces a plan in six phases before touching anything, which is the right instinct and, in this case, the wrong plan.

phases:
  - 1: inventory every import of internal/httpx; write the call-site list to MIGRATION.md
  - 2: add pkg/transport to go.mod; vendor it; confirm it builds in isolation
  - 3: update the shared wrapper internal/httpx/client.go to delegate to
       pkg/transport, and adjust all 40 callers to the new signature
  - 4: move retry configuration from per-caller literals into the wrapper
  - 5: add the RetryPolicy type and backoff defaults that the wrapper and
       callers now reference
  - 6: delete internal/httpx; run the full suite

Phase three changes Do(ctx, req) into Do(ctx, req, *RetryPolicy) and rewrites forty call sites to pass a policy. RetryPolicy does not exist until phase five. From the moment phase three's first file is written until phase five's last one, go build ./... fails with undefined: httpx.RetryPolicy — twenty-six times, once per file that references it. No test in the repository can run, because the package they live in does not compile. The agent knows this and proceeds anyway, because its plan told it to, and it has no rule that says a phase must end somewhere checkable.

The run dies eleven minutes into phase four on a wall-clock limit. What is left in the working tree is 940 changed lines across twelve packages, a build that has not succeeded since phase two, and no evidence anywhere that distinguishes "phase three did its job" from "phase three mangled nine call sites and phase four piled onto the wreckage". The only honest description of the state is that some subset of a 940-line diff is correct.

Length is the thing that broke this#

Everything above is a consequence of size. A long-horizon code task is one piece of work that a coding agent cannot finish in a single pass — dozens of edits across many files, tens or hundreds of model turns, and a run measured in hours rather than minutes. The migration here is a modest example. A framework upgrade across a monolith, extracting a module into its own package, a test-suite migration, a dependency bump with a mechanical fix at four hundred call sites: these are the shapes you meet in practice, and they are the tasks teams most want to hand to an agent, because they are tedious, largely mechanical, and expensive in engineer-hours.

Short tasks need none of the machinery in this topic, since the loop supplies it. An agent fixing one failing test edits a file, runs the test, reads the result and corrects itself inside a few turns, so a mistake made at turn two is caught at turn three while everything about it is still in view. Stretch the same loop over four hours and three of its assumptions stop holding, because the run outlives the model's memory of its own early decisions, outlives the process it started in, and outlives anyone's attention. Errors then compound rather than being caught, which is how a recoverable mistake in phase three turns into 940 lines nobody can attribute.

Three controls answer that, and they are what this topic is about. Decomposition breaks the work into steps that can each be checked on their own. Progress tracking keeps a record of what has actually been done, somewhere that survives the run. Stopping rules decide when a loop that is no longer getting anywhere should hand back to a person instead of spending the rest of its budget. The rest of this chapter is about the first of the three, and the other two rest on it, since a step nobody can check is a step no record can honestly report and no stopping rule can reason about.

Attribution is a property of the plan, not of the failure#

When a build breaks after four phases and the last green build was three phases ago, the search for the cause is a search over everything that happened in between. Bisecting the diff by hand is possible and takes an engineer somewhere between forty minutes and half a day depending on how interleaved the edits are, because the phases were not written as separable units and their edits overlap in the same files.

The rollback boundary follows the same logic. Rolling back means returning to a state that some check certified, and the most recent such state is the end of phase two. Everything after it goes, including the parts that were fine. A migration that was structured as six phases has, in practice, one: it succeeds entirely or reverts entirely.

The property that would have fixed both is not "fewer changes per phase". It is that every phase ends in an observable artifact — a tree that builds, a binary that exists, a file with known contents — and that some named command, the phase's certifying check, can be run against that artifact and returns pass or fail without a human interpreting it. Attribution comes free once that holds, because the first phase whose check fails is the phase that broke it. So does the rollback boundary, because the last phase whose check passed is a state worth returning to.

Rewriting the migration under that constraint produces twelve steps instead of six, and the reordering matters more than the count. RetryPolicy and the backoff defaults move to step two, before anything references them, so the type exists before it is used. The wrapper is changed to accept an optional policy — a variadic parameter, so the old forty call sites still compile — before any caller is touched. Then callers move in four batches of ten, grouped by package, each batch ending in a build and a scoped test run. The variadic form is deleted last, once nothing depends on it.

- id: s04
  title: "migrate billing + invoicing callers (10 sites)"
  touches: [internal/billing/**, internal/invoicing/**]
  depends_on: [s02, s03]
  check:
    - go build ./...
    - go test ./internal/billing/... ./internal/invoicing/... -count=1
  artifact: "tree compiles; 84 tests in 2 packages pass"
  rollback_to: s03

Each step now answers three questions that phase three could not answer about itself: what it is allowed to touch, what has to be true before it starts, and what command decides whether it worked. The check field is the load-bearing one. go build ./... takes 9 seconds on this repository once the cache is warm, and the two scoped test packages take 31 seconds, so certifying a step costs about 40 seconds — which is the price of knowing, at the moment step four ends, whether step four worked.

What twelve steps buy over three, and what they cost#

The same argument shows up on any large task. Take extracting an authentication module out of a monolith. Plan A has three steps: extract the package, rewire the callers, delete the old code. Plan B has twelve, each ending in a package that compiles and a named test subset that passes. The instinct that B is better is right, and the usual reason given for it is wrong, so it is worth laying the two side by side on the dimensions that actually differ.

Nothing in that table says "smaller is safer". Step size is a consequence; the per-step check is the mechanism. A twelve-step plan whose steps cannot be individually verified is a three-step plan with extra ceremony — it has the paperwork of granularity and none of the properties, since the first state anyone can certify is still the end.

The cost column is real and is the reason A survives. Ordering twelve steps so that each one is independently compilable took the constraint that produced the variadic wrapper, and the variadic wrapper is a temporary construct that exists only to keep the tree green mid-migration and has to be removed afterwards. That is genuine extra work in service of verifiability. Eleven intermediate states each have to make sense to the next step, and each one is a place the run can be interrupted, which the following chapters will show is not free either.

A is the right choice when the task is small enough that bisecting its diff by hand is cheap — under roughly two hundred lines in a single package, where reading the whole change is faster than reasoning about which step produced which hunk. It is also right when the tree can be held green throughout regardless of how the work is sliced, because then the coarse plan already has a check running continuously and slicing finer buys nothing but overhead. Renaming a symbol across a codebase with a working refactoring tool is the clean example: every intermediate state compiles by construction, so there is nothing for per-step checks to catch that the final check would miss.

The step whose check cannot exist#

Occasionally a step resists all of this. Consider step nine of the auth extraction: "move session-token parsing from the request middleware into the new package, preserving current behaviour." The code compiles before and after. There are no tests over the middleware, because it was written in 2019 and its behaviour is defined by production traffic. No command in the repository distinguishes a correct move from one that dropped a header-casing quirk.

That step has two honest fates, and inventing a third — declaring it verified because the build is green — is what produces migrations that pass every gate and change behaviour in production.

The first fate is to fold it into a neighbour whose check does cover it. If step ten adds an integration test that exercises the login path end to end, then nine and ten become one step certified by that test, and the rollback boundary widens to include both. That widening is the cost, and it is worth paying explicitly rather than pretending the boundary is narrower than it is.

The second is to manufacture a check. A characterisation test pins current behaviour without claiming it is correct, which is exactly the guarantee a "preserve current behaviour" step needs:

// characterisation only: asserts what parseSessionToken does today,
// not what it should do. Delete when the extraction lands.
func TestParseSessionToken_Characterisation(t *testing.T) {
    cases := []struct{ header, want string }{
        {"Session=abc123", "abc123"},
        {"session=abc123", "abc123"},          // casing quirk, deliberate
        {"Session=abc123; Path=/", "abc123"},
        {"Session=", ""},
        {"", ""},
        {"Session=a=b", "a=b"},                // no split on second '='
    }
    for _, c := range cases {
        if got := parseSessionToken(c.header); got != c.want {
            t.Errorf("parseSessionToken(%q) = %q, want %q", c.header, got, c.want)
        }
    }
}

Six cases, written by reading the old implementation, take about ten minutes and turn an unverifiable step into a certified one. The lowercase-session case is the sort of thing a clean reimplementation drops silently.

Where behaviour is too wide to enumerate, the same move works at a coarser grain: a temporary assertion compiled into the binary that fails loudly if the old and new paths disagree, removed when the plan completes; or a snapshot of the artifact the step is supposed to produce — the generated OpenAPI document, the compiled query plan, the rendered config — diffed byte for byte against the pre-step version. A snapshot check is weak in the sense that it proves only sameness, and sameness is precisely the claim a mechanical move makes.

Catching phase three before it runs#

The six-phase plan is not a stupid plan. It is a narrative one, ordered the way an engineer would describe the migration to a colleague — swap the wrapper, tidy the configuration, add the types the tidying implied, delete the old thing. Narrative order groups work by topic, and compilability is a constraint on order that topics ignore. Models plan in narrative order by default because that is what the prose they learned from does, so the failure is systematic rather than occasional, and it is worth a check that runs before the first edit.

The check is mechanical, because the defect is structural. Every step must name a check, every named check must be a command that exists, and the checks must be runnable in the order the plan proposes without depending on a symbol a later step introduces. The last of those is the hard one to prove in general, and a cheap approximation catches most of it: for each step, take the identifiers it introduces and the identifiers it references, and flag any step that references something no earlier step in the plan produces and the current tree does not contain. Run that over the six-phase plan and phase three reports RetryPolicy as referenced-but-unproduced, which is the eleven-minute failure found in under a second. It is the same class of pre-run consistency check that a plan's dependency graph gets, applied to the vocabulary of the steps rather than to their edges.

The weaker version of this rule, and the one worth writing down because it is nearly free, is that a step whose check field is empty or whose check is a command already passing before the step ran has not been planned. It has been described.

What this does not give you#

A certifying check confirms that a step reached a state, and says nothing about whether that state is desirable. Step four's check is a build and 84 passing tests; a batch of call sites that compiles and passes while silently dropping the retry policy satisfies it completely. Decomposition localises failures. It does not detect wrongness that the checks were never looking for, which is why the strength of the suite sets a ceiling on everything in this topic — the same ceiling that showed up when evaluating coding agents, where the oracle's strength bounded what any score could mean.

It is also not the same thing as commit granularity, though the two should line up. A step is a unit of verification; a commit is a unit of history. Making every step a commit is a good default because it gives the rollback boundary a name that git understands, and it is not required — a step can end in a stash, a tagged worktree state, or a patch file, as long as something can return to it.

And decomposition alone does not survive the run being interrupted. The plan above knows that step four depends on steps two and three, and knows the command that certifies each. What it does not have is any record, outside the agent's own context, of which steps have already been done. When the wall-clock limit killed the six-phase run eleven minutes into phase four, everything the run knew about its own progress died with the process. That record, where it should live and what it has to contain, is the next problem.