Merged patch count is the wrong number to celebrate

The quarterly review slide had three numbers on it. Merged patches per week had gone from 41 to 127, a 3.1x increase. The agent's checks passed on 94% of the patches it proposed. Adoption was at eleven of fourteen engineers. The fourth number was in an appendix nobody had put on a slide: in the engineering survey, nine of fourteen senior engineers had named code review as the worst part of their job, up from two. The VP running the review wanted to double the programme's budget on the strength of the first number and was genuinely puzzled by the fourth, since more merged changes with a higher pass rate is not an obvious way to make anyone's week worse.

It is a very ordinary way to make someone's week worse. The pipeline has two stages with independent capacities, and only one of them was scaled.

Diagram

Generation went from 45 candidate patches a week to 190. The check stage absorbed that without noticing, because it is machinery that costs seconds. Review capacity is fourteen people with roughly five hours a week each to give it, and that number did not move, so the queue in front of it grew until the only variable that could absorb the pressure did: median time spent per patch fell from 22 minutes to 9. The engineers were not complaining about volume in the abstract. They were describing what it feels like to be the only stage in a pipeline that cannot be sped up, while being asked to go faster anyway.

The number that settles it, and its denominator#

The programme's promise was never more candidate code, because candidate code has been cheap since the first useful model shipped. The promise was less human effort per unit of shipped change, so the number that settles the argument is engineer minutes per merged change, counting review plus rework plus incident time, measured on comparable work before and after.

Two things are true in that table at once, which is why the argument in the room felt unresolvable. The unit cost fell by 23%, so on the measure that matters the programme is doing something real. And the absolute demand on engineering time went from 24.6 hours a week to 58.9, which is where the survey response comes from — nobody experiences a unit cost, they experience a Tuesday. The composition shift is the more serious signal: over half the human time is now spent after the merge rather than before it, which is the expensive end, and it is what a 9-minute review buys you.

Three points about collecting these numbers, because the collection is where this usually falls apart. Review time has to be measured against candidates rather than merges, since a rejected patch consumes review too, and counting only merges flatters the programme by exactly the rejection rate. Rework and incident time have to be attributed to specific merged changes, which means someone tags the revert, the hotfix and the incident with the change that caused it — five seconds of discipline per event, and without it you will be arguing from vibes in six months. And the before number has to come from a period you can still reconstruct, which is a strong argument for computing it in week one of the programme rather than in the quarter when someone asks.

The denominator needs one more correction, and it is the one that makes or breaks the comparison. Merged changes are not interchangeable units. Over these two quarters the mix shifted: chores — dependency bumps, lint sweeps, test additions, type annotations — went from 22% of merges to 51%, because those are the tasks with the cleanest verifier coverage and the agent naturally found them. Comparing 127 merges against 41 is comparing a different basket of work, so the honest version buckets by class and compares within bucket. Done that way, bug fixes went from 18 to 29 a week at 41 minutes each rather than 36, and chores went from 9 to 65 at 6 minutes each. Almost the entire headline improvement lives in the chore bucket, which is worth knowing before deciding what to double.

Answering the self-evident-value argument#

More merged code is not self-evidently value, because merged code is a liability that occasionally carries an asset. Every merged change adds surface that has to be understood by the next reader, kept working by the next refactor, and paid for in the next migration; the value is in the behaviour it delivers, and the behaviour is not proportional to the patch count. A quarter in which 65 dependency bumps merged and four customer-facing bugs got fixed is not obviously better than one with 9 bumps and 12 fixes, and the raw count ranks it higher.

The reason this framing is seductive is that patch count is the only number in the whole system that is free to collect, moves in the flattering direction, and needs no explanation on a slide. Check-pass rate has the same three properties and is worse, since it is the one metric that improves when you weaken the checks — an agent that only ever proposes formatting changes reaches a 99% pass rate, and a team that quietly disables a flaky-but-valuable integration suite gets a bump in it too. Any number the programme reports should be one that gets worse when the programme gets worse, and both of the headline numbers fail that test.

Expanding the programme without touching review capacity has a predictable shape. The queue lengthens, review time per patch falls further, the 52% share of time spent after the merge rises, and the incident line grows fastest because it is downstream of everything else. Hiring reviewers to absorb it converts the whole thing into a headcount cost paid permanently to sustain a throughput number nobody needed, and it does so while spending the exact resource the programme was supposed to free.

Changes worth making that lower the count#

The useful moves reduce merged patches per week and improve every number that matters, which is the clearest evidence that the count was measuring the wrong thing.

Cap the in-flight queue. Set a limit of, say, 60 open agent patches — roughly two weeks of review capacity — and have the agent idle when it is hit rather than generating into a backlog. The patch count falls, the queue latency stabilises, and review time per patch recovers because reviewers stop triaging and start reading. This is the single cheapest change available and it is resisted hardest, because an idle agent looks like waste to anyone reading a utilisation chart.

Make the agent abstain where the verifier coverage is thin. Every service has a measurable coverage profile, and a task in a module with 30% branch coverage — meaning fewer than a third of the routes through its code are walked by any test — no annotations and no integration tests produces patches whose green pipeline means very little. Routing those tasks to humans up front costs you patches and removes the ones most likely to become rework, which is 21 hours a week of the current bill.

Batch by ticket instead of by file. Three patches touching one subsystem cost three context loads from a reviewer; one patch costs one. The count drops by two-thirds on that work and total review minutes drop faster, because the fixed cost of a review is mostly the cost of getting the subsystem into your head.

Spend agent capacity on the check set rather than on features. Point it at the weakest service for three weeks to add type annotations, property tests, and integration coverage. This produces a handful of merged patches and moves defects from the bucket where nothing catches them into the bucket where an 8-second check does, which pays out on every subsequent task in that service forever. It is also the work least likely to be approved, because it is expensive on the metric everyone is watching and invisible on all the others.

What this number does not do either#

Engineer minutes per merged change is gameable in its own way, and it is worth saying so before someone discovers it adversarially. Merging more trivial changes lowers the average, which is why it travels with the class breakdown and with escaped defects per hundred merges — the second number rises when the first is being gamed. Neither survives contact with an org that has decided what answer it wants, and no metric does.

Attribution stays genuinely hard, and honesty here is worth more than precision. You cannot A/B test a whole organisation over two quarters; the codebase changed, the team changed, and a hiring freeze in the middle of Q3 moved the review capacity denominator on its own. The best instruments available are a before-and-after on comparable work with the class breakdown intact, and, where you can afford it, a holdout, which is a comparable slice of the organisation deliberately left out of the change so it can serve as a control: one service of similar size and staffing that does not get the agent for a quarter. Neither is clean, and both are enormously better than a throughput number with nothing underneath it.

The thing to do with all of this is start narrow and instrument on day one: one service with a real check set, the four verdicts wired into a bounded loop with the cheap ones inside and the slow ones at the edge, and a baseline of engineer minutes per merged change captured before the first agent patch lands. The property that makes code the right first domain has been sitting in your repository the whole time, maintained by people who never heard of your agent, and the work in front of you is mostly the unglamorous business of making sure your repository has actually taken delivery of it.