A suite used to steer every change stops being a sample

The chart in the monthly update goes up and to the right without a single reversal: 62% in month one, 71%, 76%, 81%, 85%, 88% in month six. Twenty-six weekly runs, sixty-three harness changes, each one validated against the 187-task suite before it merged. In the same month-six retrospective, three engineers who review agent patches every week said independently that it does not feel any better than it did around month two, and the reviewer-minutes-per-merged-change figure has sat between 58 and 64 for five months. Two instruments pointed at the same system disagree, and the one that moved is the one everything was optimised against.

Nothing dishonest happened. No one edited an oracle, the protected-path enforcement from the mining chapter held, and every one of the sixty-three changes was a genuine engineering improvement in someone's judgement. The mechanism is the selection procedure itself. A change that improved the suite was kept and a change that did not was discarded, sixty-three times, which is a hill-climb on the suite performed by humans over six months. What accumulates is a harness fitted to 187 specific tasks, and the fit is invisible because each individual step was reasonable.

What the fitting looks like when you go and find it#

The specific adaptations are mundane and each has a defensible story attached.

The retrieval ranker's per-service boosts were tuned against the tasks that exist, and 61% of those come from three of nineteen services, so the ranker is now good at finding files in the notifications, catalogue and identity services and no better than it was in month one anywhere else. The context summariser was tuned on failure traces from suite runs, which are traces of the kinds of failure those tasks produce. The per-task timeout was raised to 14 minutes because four suite tasks needed it, and lowered nowhere. The prompt acquired a paragraph explaining how to read a bug report that describes a symptom without naming a file, which is exactly the shape the redaction rule produces and not the shape a real ticket always has. A conventions document injected at every turn grew to 3,400 tokens, much of it added after specific suite failures, and about a third of it concerns modules only the suite touches.

There is also plain leakage — the answer reaching the agent through a channel nobody intended — which is easier to find and less interesting. The reference patches were stored in a directory inside the same repository the agent checks out, so for two months the agent could read the answer to any task whose reference patch it happened to retrieve; four tasks were resolved that way before somebody noticed the retrieved-files log. Task statements are drawn from merged pull requests, so any model whose training data includes the repository has seen the real fix. These are bugs and they get fixed. The fitting described above is not a bug and does not get fixed, because there is nothing to fix — it is what tuning against a fixed set means.

Machine learning has run this experiment at scale already. The ImageNet and CIFAR-10 test sets, the two image-classification task sets the field graded itself on for years, were public and reused by everyone in it for the better part of a decade, and when researchers built fresh test sets from the original data collection procedures around 2019, accuracy on the new sets came in substantially lower than on the old ones across essentially every model, with the gap larger for models that had been developed later. The community had not cheated; it had selected. A 187-task suite steered by one team for six months is the same process at a much smaller scale, which means it happens faster.

The three numbers that separate progress from fitting#

Distinguishing the two requires a measurement the tuning has not touched, and the practical arrangement is three pools of tasks with different rules about who may look at them: a development split the team tunes against freely, a held-out split that nobody may tune against and that is opened only on a fixed cadence, and a pool re-mined every month from recent history, whose tasks have never been seen by anyone before the run that scores them.

The development split and the freshly mined pool diverge by 27 points over five months, and that divergence is the measurement of how much of the climb belongs to the suite rather than to the agent. The held-out split sits between them, closer to the fresh pool, which is what you would expect: it is the same vintage of tasks as the development split and shares its biases about which services are well tested, so it is protected against tuning but not against the suite's composition. The fresh pool is the one that matters most and is the one the arrangement makes cheap.

REMINE_CADENCE_DAYS = 30
FRESH_WINDOW_DAYS = 56          # only merges from the last 8 weeks

def monthly_refresh(repo, suite) -> dict:
    fresh = mine_tasks(repo, since_days=FRESH_WINDOW_DAYS, target=20)
    fresh = [t for t in fresh if validate_task(t)]     # red twice, green twice

    # Tasks green in 10 consecutive weekly runs carry almost no information.
    saturated = [t for t in suite.dev if suite.streak(t) >= 10]
    suite.retire_to_regression_tier(saturated)

    # Fresh tasks enter the dev split; an equal number of dev tasks are archived
    # oldest-first, so the split ages out rather than accumulating.
    suite.dev_admit(fresh[:len(saturated)] or fresh[:5])
    return {"fresh_score": run_suite(fresh), "dev_score": run_suite(suite.dev),
            "retired": len(saturated), "dev_age_p50_days": suite.median_task_age()}

Re-mining monthly costs about half a day of engineering time and roughly 200 agent runs, and it is the single highest-value practice in this chapter because it is the only one that keeps the task distribution moving with the codebase. A suite mined once in January is measuring an agent against a repository that no longer exists by July: services were split, a framework was upgraded, the team's test conventions changed, and none of that reaches a frozen task set. Retiring saturated tasks matters for a different reason, which is information rather than freshness — a task that has resolved in ten consecutive weekly runs cannot distinguish this week's harness from last week's, and keeping fifty of those in the suite mainly serves to make the headline number larger.

One more diagnostic is worth running when the divergence appears, because it turns a suspicion into a list of things to change. Take the sixty-three harness commits, label each with the suite tasks whose outcome it flipped, and count how many distinct tasks account for the improvements. On this programme, 41 of the 63 changes had their entire measured effect concentrated in five or fewer tasks, and eleven tasks appeared in the flip list of more than half of them. Those eleven were doing the steering: they sat near a resolution rate of 0.5, so they flipped often, and a change that nudged them counted as a win. A suite where a handful of borderline tasks decide every decision is not sampling the repository, it is a very expensive way of tuning against eleven pull requests from 2023.

Holding out only slows the clock#

A held-out split is not a solution, and treating it as one is how it gets consumed. Every time somebody runs the held-out tasks, looks at which ones failed and makes a change in response, those tasks have joined the development split, whether or not anyone relabels them. The protection is a budget on looks rather than a property of the data: open the held-out split on a fixed cadence, report the number, and forbid per-task inspection of failures except in a formal review that happens at most quarterly and is written down.

Even with that discipline the split degrades, which is why the table above has it opened four times a year and why the plan should include replacing it entirely once a year from re-mined history. The freshly mined pool is what stays honest indefinitely, because its tasks have never been seen by anyone when they are first run. Its weakness is size: 20 tasks gives an interval of roughly ±20 points on a rate near 60%, so a single month's fresh score is nearly uninformative on its own and only the trend across months carries signal. Reading the fresh column above one row at a time would tell you nothing; reading six rows of it against a development split that climbed 26 points tells you a great deal.

The pattern to distrust#

A rising suite score with flat review load and flat reviewer perception is the specific configuration that should stop a programme rather than reassure it. The reason it deserves that status is that it has exactly one benign explanation and several unpleasant ones. The benign case is that the agent genuinely improved on work the suite represents and that work is a small share of what reviewers see, which is checkable by looking at whether the improving task categories match the categories in the review queue. The unpleasant cases — fitting, saturation, leakage, a task distribution that has drifted away from the repository — all produce the same chart, and all of them are cheap to test for with the three pools above.

The opposite pattern is worth naming too, because it gets misread. A flat suite score with falling review load is usually real progress: patches are becoming easier to review without resolving more tasks, which is what happens when an agent gets better at scoping, at following conventions, and at explaining itself. A suite measures resolution and cannot see any of that, so a flat line there is not evidence of nothing happening.

What all of this leaves you with is an evaluation that has to be operated rather than built. The first month's work is the mining run and the three pools; after that the recurring work is a monthly re-mine, a quarterly held-out opening, a review-load figure published beside the escape rate, and a rule that no harness change ships on a single-run comparison. Start with the mining script against your own history this week, because the 187 tasks it produces are also the fixtures for every experiment the previous chapters described, and the first fresh score you record is the baseline every later month is read against.