A clean apply plus a parse is a claim about one file

The second attempt at the jitter ticket went the way the machinery was built to make it go. The guard confirmed the hunk landed inside charge, the base hash matched, the candidate parsed, and the loop wrote the file and reported:

billing/gateway.py
  hunk #1  applied at 718  offset 0  fuzz 0  symbol charge  ✓
  base     sha256:2b71e8…  unchanged since read              ✓
  parse    ast.parse ok (3 ms)                               ✓
  tests    pytest tests/billing -q → 214 passed (41 s)       ✓
ACCEPTED

Four green marks, and the module was broken. The patch introduced random.uniform(0.5, 1.5) into the retry backoff, and billing/gateway.py had no import random at the top. Python resolves global names when the line executes rather than when the module is compiled, so the file compiled, imported, and served traffic normally. The only path that touches the new line is the except TransientError branch, which the 214 tests never enter, because none of them provoke a transient gateway failure. Eleven days later the gateway had a bad thirty seconds, every charge that hit the retry path raised NameError: name 'random' is not defined from inside the exception handler, and a retryable blip became a wall of 500s. The bug was six characters wide and every check the loop ran was telling the truth.

Two claims, stated precisely#

It is worth writing down what those green marks assert, in the narrowest form that is still accurate, because the narrow form is surprising to read after you have been treating them as approval.

A hunk applying at offset zero with fuzz zero asserts that the context lines in the patch matched the file byte-for-byte at the position the patch stated. That is a claim about string equality between two pieces of text. It contains no information about what either piece of text means, whether the replacement is an improvement, or whether the patch's author understood the function it landed in. It is the strongest possible version of the edit went where it was aimed, and it stops there.

A file parsing asserts that its bytes form a well-formed program under the language's grammar. It is a claim about structure — brackets balanced, blocks closed, statements terminated — and nothing in a grammar knows what a name refers to. random.uniform is a syntactically impeccable attribute access on a name, and so is randam.uniform, and so is self.nonexistent_method(), and so is a call passing four arguments to a function that takes two. Python's grammar accepts all of them, and so does TypeScript's, and so does Go's for the subset it cannot decide locally.

Neither claim touches the four things that actually break programs. Names have to resolve to definitions, which is a scope question. Types have to line up across call boundaries, which is a whole-program question in most languages. Imports have to exist and not cycle, which is a module-graph question. Behaviour has to be right, which no static check answers at all. The jitter patch failed the first of those four while satisfying both claims completely, and it did so in a way that produced no observable symptom for eleven days.

The module's own suite is a third local claim#

The natural objection is that the test run should have caught it, and the test run is the check people trust most. What a green suite establishes is that the code paths the suite executes behaved as the suite asserts. A suite's coverage of a file is the share of that file's executable lines which ran at least once while the suite was running, and a test runner will report it per file if asked. Coverage on billing/gateway.py at the time was 87%, which sounds like most of it. The uncovered 13% was concentrated exactly where uncovered lines usually are — error handlers, retry branches, fallbacks — and the edit was inside it. A suite cannot object to a line it never runs, so a green result after an edit to an unexercised branch carries no more information about that edit than not running the suite at all.

There is a sharper version of the same limitation that survives even perfect coverage. The suite in question is tests/billing, which imports billing.gateway and a handful of its neighbours. If the same edit had deleted a re-exported name that only orders/api.py imports, tests/billing would still be green, because nothing in it loads orders/api.py. A test suite bounds its claims by what it imports, and the module's own suite is bounded to roughly the module.

The row that would have caught the jitter bug is not in that table, which is the point. ruff check --select F runs an undefined-name pass over the file's scopes in about 40 ms and prints exactly one line:

billing/gateway.py:722:38: F821 Undefined name `random`
Found 1 error.

Forty milliseconds, one file, no test database, no network. The reason it was not in the loop is not that anybody weighed it and declined; it is that the loop's four checks all came back green and green felt sufficient.

The step people usually reach for instead is importing the module, on the grounds that importing is closer to running it than parsing is. It is closer, and it is not close enough, and it is not free. python -c "import billing.gateway" executes every module-level statement in that file and in everything it imports, which in this repository means constructing a settings object, opening a connection pool to Postgres, and registering three signal handlers. That is a check with side effects, running in whatever environment the agent happens to have, and its failure modes include "the database is not up" — a red result that says nothing about the edit. What it does establish is that module-level names resolve, which the linter established forty milliseconds earlier without executing anything. Names used inside function bodies, which is where nearly all code lives, are exactly as invisible to an import as they are to a parse.

The boundary also moves depending on the language, and a loop that edits more than one has to hold more than one version of it. The identical bug in Go does not reach runtime, because an unresolved identifier is a compile error and go build ./billing fails in under two seconds. In TypeScript, tsc --noEmit reports TS2304: Cannot find name 'random' before anything runs. In Python, Ruby and untyped JavaScript, name resolution is deferred to execution, so the static gate has to be supplied by a separate tool that the repository may or may not have configured. A validation ladder tuned on a Go service and then pointed at a Python one inherits a much wider blind spot than the person who tuned it expects, and the wider blind spot is invisible precisely because the ladder looks the same.

Green marks that are not the same size#

Part of why the boundary blurs is presentation. A status line renders parse ✓ and tests ✓ in the same colour with the same tick, and a model reading its own tool output sees two successful checks and infers the conjunction of two strong properties. The checks differ in strength by a wide margin, and nothing in the output encodes that.

An acceptance record that names the claims rather than counting the passes gives the loop something it can reason about, and gives a reviewer something to argue with:

{
  "path": "billing/gateway.py",
  "established": [
    "text_placement: hunk #1 at line 718, offset 0, fuzz 0, inside symbol `charge`",
    "base_freshness: file unchanged between read and apply",
    "syntax: parses under python 3.12 grammar",
    "local_behaviour: tests/billing 214 passed; changed lines 722-723 covered? NO"
  ],
  "not_established": [
    "name_resolution: no linter run",
    "types: no type check run",
    "consumers: no importer of billing.gateway outside tests/billing was loaded"
  ],
  "decision": "hold — changed lines are not covered by the suite that passed"
}

The changed lines covered? field does most of the work in that record. It is cheap to compute, and the computation is worth spelling out because most teams assume it is harder than it is. Run the suite once with coverage collection — pytest --cov=billing --cov-report=json, which added 3.1 seconds to the 41-second run — and the resulting coverage.json contains, per file, the list of line numbers that executed. The patch already knows which lines it wrote. Intersecting the two sets is a dozen lines of code and answers a question the summary line cannot:

touched = set(patch.changed_line_numbers("billing/gateway.py"))   # {722, 723}
executed = set(cov["files"]["billing/gateway.py"]["executed_lines"])
uncovered_changes = sorted(touched - executed)                    # [722, 723]

Two out of two changed lines never ran during a suite that reported 214 passing tests. That converts a green suite from a global signal into a statement about this edit, and it is the difference between "the tests pass" and "the tests exercised what you changed". An edit whose changed lines were never executed has not been tested, whatever the summary line says, and a loop that knows the difference can escalate — run the linter it skipped, ask for a test, or hand the edit to a person — instead of accepting.

The same intersection is useful in the opposite direction. When the changed lines are covered and the suite is green, the acceptance record can say so precisely, and a reviewer reading "changed lines 722-723, both executed by 4 tests, all passing" is being given evidence rather than a colour. Recording the count of covering tests is nearly free once coverage is being collected with context, and it is the first thing anybody asks when they are deciding how carefully to read a diff.

What this is not an argument for#

None of this says the cheap checks are weak. Placement, freshness and syntax are the fastest and most reliable signals available, they catch the failures from the previous three chapters, and a loop without them corrupts files. Their value is precisely that they are cheap and definite, and definite claims are only dangerous when they are read as broader than they are.

It is also not an argument that tests are the real answer and the static checks are decoration. A green suite is a bounded claim in the same way, bounded by coverage and by imports rather than by grammar, and a loop that replaces "it parses" with "the tests pass" has swapped one local claim for another local claim that costs 41 seconds instead of 3 milliseconds. The useful move is not picking a favourite check but knowing, for each one, the set of failures it is blind to, and then deciding which of those blind spots this particular edit can afford.

That decision is where the cost lands. The obvious response to everything above is to run more checks — lint, type-check, compile, the wider suite — on every iteration, and for a repository of any size that answer collapses under its own runtime. It also spends the same budget on a one-line comment fix and on a change to a shared response model, which is where the next chapter starts: an edit that passed every local check, correctly and honestly, and broke two other services because the checks were local and the change was not.