A 96% pass rate at 41% coverage grades almost nothing

The slide said 96%, and the proposal underneath it was that agent patches with a green run should merge without human review. The reasoning was not careless. Review was the measured bottleneck — median 19 hours from patch ready to merged, against 4 minutes of agent time — and the argument was that the team had spent two quarters making the tests a trustworthy verifier precisely so that a machine verdict could stand on its own. Everything in this topic had been built to make that verdict mean something.

The number that belongs next to the 96% is 41%, which is line coverage on the repository. It is not evenly spread: ledger/utils sits at 78%, the adapter packages at around 60%, and ledger/services — where most tickets land, and where money moves — at 22%.

What the number is a measurement of#

The 96% is the share of tasks that reached a green run inside the attempt budget. That is a statement about the loop's ability to satisfy the assertions that exist, and it is a genuinely useful statement: it went from 71% to 96% over two quarters as the feedback format, selection ownership and test-edit policy came in, and if it drops next month something in the loop has broken. As a metric of the loop it is well-constructed. As a metric of correctness it is measuring the loop and the suite together and reporting a single number, which lets the weaker of the two hide behind the stronger.

The breakdown by package is where that becomes visible. Its third column is a mutation score, which the section below returns to and which can be read for now as the share of small deliberate corruptions of a package's code that some test catches:

The pass rate is flat across four packages whose coverage varies by a factor of three and a half. If the tests were doing the grading, the pass rate would fall where the grading is harder, because a change to a thinly tested settlement path is not easier to get right than a change to a date helper. It does not fall, which means the pass rate in the bottom row is being produced mostly by the absence of assertions rather than by the presence of correct code. The final column is the same claim from the other direction: patches to ledger/services pass at the same rate and fail in production at nine times the rate.

The mutation score column is worth more than the coverage column for this purpose. Coverage says a line was executed; mutation testing changes the line — flips a comparison, alters a constant, removes a call — and asks whether any test notices. A mutation score of 26% means that of the small mutations introduced into service code, three quarters produced a program the suite accepted. Running it is not cheap, since each surviving mutant needs a suite run, and on this repository a mutation run restricted to files changed in the last 30 days takes about 4 hours on a spare runner overnight. That is affordable once a week and it produces the single number that says how much a green run binds.

Converting green into correct#

The conversion requires a label the suite did not produce. Three sources are available, and they answer slightly different questions.

A blind review sample is the most direct. Take 40 accepted patches at random, strip the fact that they were green, and have two engineers independently judge each one against its ticket: correct, incomplete, or wrong. This team's result was 31 correct, 6 incomplete — the ticket's requirement partially implemented, usually an edge case — and 3 wrong. Against a 96% green rate, roughly 78% of accepted patches were actually correct, so the conversion factor from green to correct is about 0.81 on this repository as a whole, and it is worse than that in ledger/services and better in ledger/utils. That single number is what the VP's proposal is actually arguing about.

Post-merge defect attribution is slower and more honest, since it counts what reached users. It needs patches tagged at merge and defects traced back over a window long enough to catch the slow ones — a quarter, in practice, because a rounding error in a monthly report is found by finance at month end.

Seeded defect escape, from the subset experiments earlier, measures the suite rather than the agent, which makes it the right instrument for deciding what the suite can be trusted to gate. The 14 seeded defects that no test in the repository caught are the concrete form of the 26% mutation score.

Answering the proposal#

The verifier-first argument that motivated all of this depends on the verifier being independent of the generator and strong enough to reject bad work. The first property has been engineered here — the harness selects, tests are read-only, changes require falsification. The second has never been established for the packages where most of the work lands, and the 96% cannot establish it, because the pass rate is high partly because the bar is low. Approving the proposal as written applies a strong argument to a verifier nobody has audited.

There is a second-order effect that matters more than it first appears. At present a green run gets the patch to a reviewer, so the value of gaming the suite is bounded by what a human notices. Under auto-merge, green gets the patch to production, which raises the payoff on every surface from the accommodation chapter at the same time as it removes the last check on the failures nothing mechanical catches — source-only special-casing, and patches that satisfy the ticket's words while missing its intent. Whatever the accommodation rate is today, it is not a constant that survives changing what green is worth.

The bottleneck is nonetheless real, and answering a 19-hour review latency with "keep reviewing everything" is not a serious response. What can be said is that the review bottleneck is not uniform: ledger/utils patches are being reviewed at the same depth as settlement patches, and the data above says one of those is nearly free to automate.

A version worth agreeing to#

Auto-merge scoped by measured verifier strength is defensible, and the eligibility predicate is short enough to read:

def auto_mergeable(patch, run, package_stats):
    return all([
        package_stats[patch.primary_package].mutation_score >= 0.55,
        package_stats[patch.primary_package].coverage >= 0.70,
        run.diff_coverage >= 0.90,
        run.suite == "full" and run.green,           # not the impacted subset
        not patch.test_paths and not patch.config_paths,
        patch.changed_executable_lines <= 80,
        not patch.touches_any(NO_SUBSET_PATHS),      # migrations, settings, deps
        patch.reverts_cleanly(),
    ])

On the last quarter's traffic that predicate matches 23% of accepted patches, nearly all of them in ledger/utils and the adapter packages, and it removes about a third of the review queue's items while leaving the risky third untouched. Notice that it requires the full pre-merge suite rather than the in-loop subset, which is the moment where the subset's measured 4.2% disagreement rate stops being acceptable, and that it requires the patch to revert cleanly, because the policy's real safety property is that a mistake is cheap to undo rather than that mistakes do not happen.

It should be turned on in shadow first. For six weeks, compute eligibility and let humans review as normal, then compare: how often did the reviewer reject a patch the predicate would have merged? This team saw 2 rejections out of 61 eligible patches, both for the same reason — a correct patch that solved the ticket in a way the team did not want to maintain — which is a judgement no verifier produces and is worth knowing the rate of before you stop collecting it.

Then the monitoring, with thresholds and actions decided in advance rather than during the incident:

The fourth row is the one people forget to instrument. A threshold at 0.90 diff coverage is a target, and targets are met — if the distribution of auto-merged patches clusters just above the line rather than well clear of it, something is optimising for the predicate, whether that is the agent or the humans writing the tickets. A healthy distribution has most eligible patches at 0.97 and above because they are small changes in well-tested code, and the histogram shape is the diagnostic rather than the mean.

The authorisation should also expire. Ninety days, renewed only with the monitoring data attached, because the assumption underneath the whole policy is that mutation score and coverage in the eligible packages hold, and nothing in a codebase holds by default.

Where this leaves the suite#

The honest summary of the whole arrangement is that test execution is a verifier of measurable strength, and the measurement is a property of your repository that nobody can supply for you. The loop from the first chapter is worth building at 22% coverage; it will just be doing less grading than the pass rate suggests, and the gap is filled by review until somebody funds tests for the service layer.

That points at the work rather than at another chapter. Take one ticket that the agent completed last week and run the four-quadrant check on it by hand, which will tell you within an hour whether your green means anything. Compute mutation score on the ten files your agent changes most often, since that is the number the pass rate is hiding. Log the selector and the executed set for a fortnight and see whether they ever differ. Sample twenty accepted patches, review them blind, and write down your own conversion factor from green to correct. Every policy in this topic is a threshold on a number, and the numbers here belong to one repository — the method transfers, the constants do not.