Order the validation ladder around the moment of writing

The same failing edit, run through two orderings of the same seven checks, produces these two traces. The edit is the jitter patch with the missing import random, and the check that catches it is the linter's undefined-name pass.

# write first, then validate
00.000  write billing/gateway.py                        (file on disk is now broken)
00.004  post-write hook: ruff format --fix              rewrites the broken file
00.010  editor language server reindexes                214 phantom diagnostics
00.052  ruff check → F821 Undefined name `random`
00.052  FAIL → snapshot restore, index restore, verify   1 rollback
00.310  report to model

# validate on the candidate, then write
00.000  ast.parse(candidate)                            ok      3 ms
00.003  ruff format - < candidate                       ok     25 ms
00.028  ruff check --stdin-filename billing/gateway.py  F821   40 ms
00.068  FAIL → discard candidate                        0 writes, 0 rollbacks
00.068  report to model

The difference in wall time is a quarter of a second and does not matter. The difference in what happened to the repository is the whole design: in the first trace a file that does not compile existed on disk for 52 milliseconds, a formatter ran against it and rewrote it, a language server — the background process that holds an editor's view of the project's symbols and types, and answers its questions about them — indexed it and produced two hundred diagnostics that will sit in the engineer's editor until the next save, and recovery required the rollback machinery from the previous chapter to execute correctly. In the second trace nothing happened. The edit was a string that got thrown away.

Two kinds of check, and the write between them#

The checks available divide cleanly by what they need in order to run. Some accept content on standard input and need nothing on disk; some need a file, or a package, or a whole build graph.

Parsing runs on a string, as established in chapter 3. Formatters run on strings too — ruff format -, prettier --stdin-filepath src/cart.tsx, gofmt reading stdin — which matters because formatting is a mutation rather than a gate, and running it on the candidate means the bytes that eventually reach disk are the formatted ones, so no post-write hook has to fire and no second version of the file exists. Linters mostly accept stdin as well, with a filename passed alongside so that per-path configuration still applies: ruff check --stdin-filename, eslint --stdin --stdin-filename, golangci-lint being the awkward exception that wants a package.

Type-checking is where the boundary sits, because a type checker needs the rest of the module graph and normally reads it from the filesystem. Two ways across: a language server with in-memory overlays, where tsserver or pyright is told "for this session, billing/gateway.py has these contents" and answers against the overlay without the file changing, or an overlay directory that shadows the repository for a single invocation. The overlay approach costs about 240 ms warm against a tsserver that is already holding the project, and 6 seconds cold, which is the difference between having a persistent language server in the loop and not.

Everything below that — compiling the package, running the module suite, building the affected targets — needs real files, because it shells out to tools that open paths. That is where the write goes.

The ordering above the write is by cost ascending, and it is also an ordering by precondition: a lint result on a file that does not parse is noise, and a type error reported on a file with a syntax error is usually a cascade of phantom errors that will consume the model's context and mislead its next attempt. Every rung assumes the rungs above it passed, so a failure at any rung stops the ladder rather than collecting a full report — the model gets one actionable error rather than forty-one, and the forty that would have followed were mostly artefacts of the first.

The shares come from 118 rejected edits over a month in the repository from chapter 5. Eighty per cent of failures are caught by checks that cost under a quarter of a second and never touch the filesystem, which is the empirical case for the ordering rather than the aesthetic one. The compile row reading 0% is not an error; in a Python and TypeScript repository the type check subsumes it, and it stays on the ladder only for the Go packages, where it catches about a third of that language's rejections.

One ladder, several files#

Most interesting edits touch more than one file, and the ladder has to say what that means at each rung. The string rungs run per file, because a parser and a linter each take one file's contents, and a failure in any of them fails the whole edit. The type check is the rung where per-file processing breaks down: a patch that adds a parameter in platform/schemas/order.py and updates its caller in platform/orders/build.py will fail a type check of either file in isolation, since each one is only correct in the presence of the other. The overlay has to carry every candidate at once, which is a small change to how the check is invoked and a significant change to how it must be reasoned about — the unit being validated is the patch, not the file.

The write then becomes a barrier with a specific shape. Every candidate passes every string rung, then all files are written in one tight sequence with no model turn between them, then the disk rungs run once over the union of affected packages rather than once per file. Interleaving the writes with the disk checks — write file one, compile, write file two, compile — guarantees a red result in the middle for exactly the reason above, and burns a compile to produce it.

Keeping the type check fast enough to sit above the write depends on a warm language server, and the lifecycle of that server is part of the design rather than an implementation detail. A pyright or tsserver process started at session start and held for its duration answers overlay queries in around 240 ms and costs 400–900 MB of resident memory on a repository this size; the same query against a cold process costs six seconds, which pushes the rung below the write on cost grounds and changes the whole shape of the ladder. The server has to be restarted, or at least told to reload, when the edit changes the dependency graph rather than only file contents — a new import of a package not previously loaded, a change to tsconfig.json or pyproject.toml, a new file in a package. Missing that restart produces the worst failure available at this rung, which is a stale green: a type check that answers against a graph the repository no longer has.

The duplication argument, taken seriously#

The objection is that CI runs lint, type-check, build and test after the branch is pushed, so running them again inside the loop is the same work done twice. The premise is accurate and the conclusion does not follow, for reasons that are worth separating.

The first is about when a signal can be acted on. CI reports minutes after a push, and by then the loop has exited, the model's context is gone, and the failure has become a notification in somebody's inbox. The entire retry mechanism the agent is built around — re-read, re-plan, re-emit — is only available while the loop is running, so a check that reports after the loop has finished is a check that produces no automatic behaviour at all. That is not duplication; the two runs feed different consumers, one a machine that can act, one a person who has to.

The second is about scope, which cuts against the objection numerically. CI validates a whole repository against a merge target; the loop validates one package and, when chapter 5's classification says so, its reverse dependencies. In the repository above, the loop's per-iteration checks cost a median of 41 seconds against CI's 11 minutes, and installing the ladder cut CI minutes by 34% over the following month, because branches that would have failed stopped being pushed. The loop is not doing CI's work twice; it is doing a small subset early enough that CI's work happens less often.

The third concession is real and should be made rather than argued away: the definitions are duplicated, and duplicated definitions drift. A loop that reimplements the lint rules, or hardcodes a tsc invocation that diverges from the one CI uses, ends up rejecting edits CI would accept and accepting edits CI rejects, which is worse than either policy alone. The fix is that every rung invokes the same target CI invokes, with the same configuration file, differing only in scope:

# ladder.yaml — one definition, two callers
rungs:
  - name: parse
    mode: in_memory
    impl: tree_sitter          # grammar chosen per file extension
  - name: format
    mode: stdin
    command: ["ruff", "format", "-"]        # CI: make fmt-check
    mutates: true
  - name: lint
    mode: stdin
    command: ["ruff", "check", "--stdin-filename", "{path}", "-"]  # CI: make lint
  - name: typecheck
    mode: overlay
    command: ["pyright", "--outputjson", "--project", "{package}"] # CI: make types
  - name: write
    mode: commit
  - name: module_tests
    mode: disk
    command: ["pytest", "-q", "{package_tests}"]                   # CI: make test
  - name: affected_build
    mode: disk
    tier_min: 2
    command: ["bazel", "build", "{rdeps}"]                         # CI: bazel build //...

Each rung carries the CI command it corresponds to in a comment because the two must be kept in step by a person; the honest version of this is a test that asserts the loop's lint command and CI's lint command resolve to the same configuration, which is twenty lines and prevents a category of confusion that otherwise recurs every few months. PatchPilot's account of a cost-efficient generate-apply-validate-refine loop makes the same trade from the other direction, spending most of its budget on cheap validation and reserving expensive verification for candidates that survive it.

The rung to drop first#

Under latency pressure the affected-target build goes first, and it goes by moving rather than by deleting. Instead of running it on every iteration of a tier 2 edit, run it once, after the loop is otherwise satisfied and before it reports done. That saves 2 minutes 10 seconds on each of the 2.4 additional iterations a typical tier 2 task takes, which is most of the ladder's cost, and it keeps the check that would have caught the stale generated client.

What that costs is attribution. A cross-package failure discovered at the end of a task arrives after several intervening edits, so the model has to work out which of them broke the dependent service rather than being told immediately, and the re-plan is correspondingly more expensive — in practice about 2.1 extra turns compared with catching it in the iteration that caused it. The version of this move that is not acceptable is dropping the rung entirely rather than deferring it, because a loop that exits without ever compiling its reverse dependencies reproduces the failure from chapter 5 exactly, and the whole point of the classification was to know when that check was owed.

The two rungs that never come off are parse and the base-hash comparison. Parse costs three milliseconds and every check below it is meaningless without it. The hash comparison costs a tenth of a millisecond and is the only thing standing between the loop and an edit applied to a file it never read.

What remains is to make the ladder yours rather than this one. Log, for every rejected edit, which rung caught it, how long the rung took, and how many turns the model needed afterwards; a month of that data tells you which rungs are earning their runtime and which are catching nothing that a cheaper rung above them did not already catch. Rungs with a zero catch rate and a real cost should move below the write or off the ladder; rungs that keep catching failures the model then fixes on the first retry are the ones worth paying more for. The design in this chapter is a starting configuration, and the instrumentation is what turns it into the right one for a particular repository.