Merged commits hand you oracles somebody already reviewed

The first mining run — sweeping the repository's own merge history for changes that can be cut into graded tasks — took 38 minutes over four years of main and returned 1,412 merged pull requests that touched both source and test files. After the exclusion filters ran, 312 remained. After the validation pass — check out the merge commit, revert the source hunks, confirm the shipped tests fail, restore the source, confirm they pass, then do both again to catch flakes — 187 survived. That is a regression suite, a fixed set of graded tasks you re-run after every change to the agent so that a fall in the score points at what the change broke: 187 tasks on your repository, in your language, using your conventions, each with an oracle that a human wrote and another human reviewed before it merged. It cost 34 CPU-hours on the CI fleet and about a day and a half of one engineer's time.

The construction is the same one the benchmark uses, pointed at your own history. A merged change contains, in one artefact, a statement of a problem, a solution somebody accepted, and a check that distinguishes the two states. Splitting that artefact in half gives you a graded task: the description and the pre-change tree are the input, the shipped tests are the oracle, and the source diff is a reference solution you keep for comparison and never show the agent.

Cutting a merge commit into a task#

import subprocess, json
from pathlib import Path

SOURCE_ROOTS = ("src/", "services/", "lib/")
TEST_ROOTS = ("tests/", "src/**/__tests__/")

def build_task(merge_sha: str, repo: Path) -> dict | None:
    files = changed_files(repo, merge_sha)
    src = [f for f in files if f.startswith(SOURCE_ROOTS)]
    tst = [f for f in files if is_test_path(f)]
    if not src or not tst:
        return None

    parent = git(repo, "rev-parse", f"{merge_sha}^1").strip()
    pr = fetch_pull_request(merge_sha)          # title + body + linked issue

    # Working tree = parent, with the merge's test files copied forward.
    git(repo, "checkout", "--detach", parent)
    for path in tst:
        blob = git(repo, "show", f"{merge_sha}:{path}")
        (repo / path).parent.mkdir(parents=True, exist_ok=True)
        (repo / path).write_text(blob)

    return {
        "task_id": f"{repo.name}__{merge_sha[:10]}",
        "base_commit": parent,
        "statement": redact_leaks(pr.title + "\n\n" + pr.body),
        "oracle_tests": select_tests(tst),      # node ids, not whole files
        "protected_paths": sorted(tst),
        "reference_patch": git(repo, "diff", parent, merge_sha, "--", *src),
        "reference_reviewers": pr.approvers,
        "merged_at": pr.merged_at,
    }

Two details in that function carry most of the weight. select_tests resolves the changed test files down to individual test node ids and keeps only the ones whose status actually flips between the two states, which is the difference between "the tests in this file" and a genuine fail-to-pass set — a change that adds one test to a file of ninety should not require the agent to satisfy the other eighty-nine as its primary target. Those other eighty-nine still run, as the regression half of the grade, and a task fails if the agent breaks any of them. redact_leaks strips the parts of a pull request body that were written with the diff in hand, which is discussed below and matters more than it looks.

The validation pass is where most candidates die, and running it twice at each state rather than once is what makes the survivors trustworthy. A single run at the reverted state that shows red proves nothing on its own, because a test can be red for reasons unrelated to the change — an expired fixture date, a dependency that no longer resolves, an ordering dependency on another test file that the copy-forward disturbed. Requiring red twice at base_commit and green twice at the merge commit removed 125 of the 312 candidates on the first run, and the breakdown was instructive: 71 failed because the reverted tree would not build at all without other files from the merge, 38 were flaky in at least one of the four runs, and 16 passed at base_commit, meaning the shipped test did not actually depend on the shipped fix. Each of those would otherwise have entered the suite as a task the agent could never solve, or as one it solves for free.

The oracle only means anything if the agent cannot touch it. An agent asked to make test_proration_uses_downgrade_date pass has an obvious cheaper route available, which is to edit the assertion, and coding agents find that route without being encouraged. The harness has to refuse it structurally rather than by instruction:

def grade(run) -> dict:
    touched = {h.path for h in parse_unified_diff(run.patch)}
    illegal = touched & set(run.task["protected_paths"])
    if illegal:
        return {"status": "invalid", "reason": f"diff touches oracle: {sorted(illegal)}"}

    apply_patch(run.workspace, run.patch)
    checkout_paths(run.workspace, run.task["base_commit"] + "..oracle",
                   run.task["protected_paths"])          # restore, unconditionally
    target = run_tests(run.workspace, run.task["oracle_tests"])
    guard = run_tests(run.workspace, impacted_suite(run.patch))
    return {"status": "resolved" if target.all_passed and guard.all_passed else "unresolved",
            "target": target.summary, "guard": guard.summary}

The restore step is deliberately redundant with the rejection above it. A patch can reach a test file without appearing to, through a conftest.py, a fixture factory, a monkeypatched helper, or a change to the test runner's configuration, and on the first internal suite one team ran, 4 of 187 tasks were "resolved" by an agent that had edited a shared fixture in tests/support/ to return the value the assertion wanted. Restoring the protected paths from the oracle side after applying the patch turns that class of trick into a plain failure. Rejecting the run outright when the diff names a protected file is the version that gives a useful error message; restoring is the version that catches what the check missed.

The two things this method builds into the number#

Only well-tested code produces tasks. The mining filter requires a merged change that shipped tests, so packages with thin coverage contribute nothing, and those are usually the oldest, gnarliest, least documented parts of the repository. On the run above, 61% of the 187 tasks came from three of nineteen services, and the payments-ledger service — 40,000 lines, 14% line coverage, the source of about a third of production incidents — contributed four. The suite is therefore blind in exactly the region where an agent's output is hardest to verify and where a bad patch is most expensive. That is not a flaw to be fixed by mining harder, because there is nothing there to mine; it is a fact about the number that has to travel with it.

The second bias is subtler and inflates scores. The tests in a merged pull request were usually written after the engineer understood the bug, and often after they had written the fix. They encode the shape of the solution, not only its externally visible behaviour, so handing them to the agent as the oracle gives it a specification the original engineer did not have while working. A test named test_fx_rate_snapshot_taken_at_downgrade tells the agent which of three defensible interpretations to implement, which was the entire difficulty of the ticket in the first chapter. The pull request body does the same thing when it was written to describe the diff: "switches get_rate() to look up by downgrade_at rather than subscription.started_at" is not a task statement, it is an implementation plan.

That is what redact_leaks is for, and it should be a review step rather than a regex. The practical rule is that the statement keeps the symptom, the reproduction, the affected data, and the linked issue text as it existed before the fix, and drops any sentence naming a function or file that the reference patch modifies. On the internal suite, applying that rule by hand to 187 statements took four hours and moved the agent's score down by nine points, which is a good measure of how much of the original number was reading comprehension.

What to leave out, and what each exclusion buys#

The large-diff exclusion is the one most often argued about, because dropping the hard tasks looks like flattering the agent. The reason is about oracle strength rather than difficulty. A 900-line change across four services shipped with tests that cover its main path; an agent can satisfy those tests with a patch that gets the main path right and the three edge cases wrong, and the harness will report "resolved". Keeping such tasks does not measure hard work, it measures a weaker oracle, and it does so invisibly. If large changes matter to you, they belong in a separately reported slice with human grading, not mixed into a pass rate.

Reading 71% off this suite#

Suppose the agent resolves 132 of 187 tasks. The claim that number supports is narrow and still useful: on tasks drawn from the well-tested parts of this repository, of a size that historically shipped as one pull request, with a statement redacted to symptom level and an oracle written by the engineer who fixed it, the agent produced a patch that satisfied the shipped tests and broke nothing in the impacted suite, 71% of the time under this harness at this configuration.

Four things it does not support. It is not a prediction for the payments ledger or any other thinly tested area, which the suite barely samples. It is not a statement about tickets, because every task here was derived from a problem that turned out to be solvable and got solved, and the 173 tickets from the previous chapter that produced no patch have no representation. It is not a claim about correctness, since satisfying a test written alongside the fix is weaker evidence than a reviewer reading the diff, and the gap between the two is exactly where the next few chapters live. And it is not comparable to anybody else's 71%, including a vendor's, because the tasks, the redaction rule and the harness are all local.

That last point is about to become concrete rather than theoretical. A suite built this way is a good instrument for measuring your own agent over time, and the first thing anybody wants to do with it is point it at somebody else's agent, which is a different measurement than it looks.