A benchmark task arrives with its own oracle attached

The release note said the new model scored 58% on SWE-bench Verified, up from 41% for the model it replaced. Swapping it into the agent took an afternoon, because the only thing that changed was a model identifier in the harness config and a slightly larger context window. Over the six weeks that followed, 214 tickets were routed to the agent, it opened 41 patches, and 22 of those merged. Over the six weeks before the swap, 198 tickets were routed, it opened 38 patches, and 21 merged. The merge rate moved from 55.3% to 53.7%, which on those sample sizes is not a movement at all, and the median reviewer time on an agent patch stayed at 24 minutes. Seventeen points of public benchmark bought nothing anybody could see.

Before working out why those two numbers disagree, it is worth being plain about what each of them is. Evaluating a coding agent means running it over a set of programming tasks whose outcomes something can grade automatically, and counting how many it got right — where "the agent" is the whole system that turns a problem statement into a patch: the model, the code that retrieves repository context for it, the loop that applies its edits and runs the tests, the retry policy. That surrounding code is the harness, and it is a substantial piece of engineering in its own right. A benchmark is a task set published so that different systems can be scored on the same problems, and the reason one exists at all is that the alternative is anecdote — a demo, a thread, an afternoon somebody spent with a trial licence — and no two anecdotes can be compared. SWE-bench, where SWE is software engineering, is the task set behind almost every coding-agent figure a vendor quotes; the Verified subset is the portion of it that human engineers checked for solvability. You meet these numbers in release notes, vendor decks and leaderboards, and then again a month later when somebody asks in a planning meeting why the upgrade did not show up anywhere.

The other word doing quiet work is oracle, which is whatever decides, with no human in the room, whether a particular attempt at a task is correct. For code the oracle is normally a set of tests: run them against the agent's patch, and green counts as resolved. Nothing can be scored without one, which is why the supply of oracles ends up governing everything in this topic — a benchmark is free to select tasks that already have an oracle attached, and the queue your agent works from is not.

The explanations that get offered first in that meeting are usually contamination, sampling temperature, or some version of "our codebase is just harder than Django". All three are worth checking eventually and none of them is where to start, because there is a structural difference between a benchmark item and a ticket that accounts for the gap before any of those effects are needed. A benchmark task arrives with a machine-checkable definition of done already attached to it. A ticket does not, and the agent's job on a ticket includes inventing that definition, which is a different and harder job that the benchmark never scores.

What a SWE-bench item actually contains#

The task set that most coding-agent numbers are reported against was assembled by scraping merged pull requests from a dozen popular Python repositories, keeping only the ones that closed an issue and shipped at least one test. Each instance is a record that looks roughly like this, with the long fields cut down:

{
  "instance_id": "django__django-11099",
  "repo": "django/django",
  "base_commit": "d26b2424437dabeeca94d7900b37d2df4410da0c",
  "problem_statement": "UsernameValidator allows trailing newline in usernames\nDescription\n\nASCIIUsernameValidator and UnicodeUsernameValidator use the regex\nr'^[\\w.@+-]+$'\nThe intent is to only allow alphanumeric characters as well as ., @, +, and -.\nHowever, a little known quirk of Python regexes is that $ will also match a trailing newline...",
  "hints_text": "",
  "created_at": "2019-03-20T09:20:5",
  "patch": "diff --git a/django/contrib/auth/validators.py ...",
  "test_patch": "diff --git a/tests/auth_tests/test_validators.py ...",
  "FAIL_TO_PASS": [
    "test_ascii_validator_with_newline (auth_tests.test_validators.UsernameValidatorsTests)",
    "test_unicode_validator_with_newline (auth_tests.test_validators.UsernameValidatorsTests)"
  ],
  "PASS_TO_PASS": [
    "test_ascii_validator_with_valid_usernames (...)",
    "test_unicode_validator_with_valid_usernames (...)"
  ],
  "environment_setup_commit": "419a78300f7cd27611196e1e464d50fd0385ff27"
}

The two list fields carry the grading. FAIL_TO_PASS names tests that fail at base_commit and pass once the reference fix is applied; PASS_TO_PASS names tests that pass in both states and must keep passing. The harness applies test_patch — never the agent's diff — to install the oracle, runs both lists, and the instance resolves if every named test in both lists is green. The agent never sees test_patch or the two lists, so it is not being handed the answer. What it is being handed is a problem that was selected for having a crisp, mechanically checkable answer, written by someone who had already decided what correct behaviour was.

Notice how much of the difficulty has been removed before the agent starts. The scope is one repository at one known commit with a reproducible environment. The intended behaviour has been argued out on the issue thread and settled. The change is small enough to have shipped as one pull request. Nobody will ask whether the fix should have been a config flag instead, or whether it needs a migration, or whether the four other call sites that rely on the old behaviour are in scope. The remaining work is localisation, an edit, and verification, which is real work and worth measuring, and it is roughly the last third of what a ticket demands.

The ticket the agent gets on your repository#

Here is a real item from the queue that produced those 214 routings, reproduced as it was written:

LEDG-3318  Wrong proration on annual → monthly downgrade (EUR only)

Finance flagged this during the March close. Two accounts that downgraded
from Annual Pro to Monthly Pro mid-cycle got credited about 8% less than
expected. Both are EUR. USD accounts in the same window look fine.

Thread: #billing-eng from 3 March. Erik thinks it's the FX snapshot date
but there's also that rounding thing we never finished from LEDG-2988.

No repro yet. Two affected invoices attached.

Before anything else, note what happened to this ticket in the funnel. Of the 214 tickets routed to the agent, it opened patches on 41, which means that on 173 of them it either gave up, produced a diff the harness rejected before a human saw it, or ran out of its attempt budget. A benchmark reports resolution over instances that were all deemed solvable by construction, so the equivalent of those 173 does not exist in it. Any comparison that puts 58% next to your merge rate is comparing a number computed over a curated numerator with one computed over everything that walked in the door.

Now the task itself. There is no test that fails today and would pass after a correct change, because nobody has decided yet what "expected" means. The credit could be computed against the FX rate at subscription start, at the downgrade date, or at the original invoice date, and the three produce different numbers that are each defensible. The scope is unsettled: fixing the FX snapshot might be a one-line change, and fixing the rounding might require a data backfill for invoices already issued. An agent that produces a patch making some plausible assertion pass has not resolved this ticket, because the assertion it chose is the thing under dispute.

The last row is the one with a cost attached. On a benchmark, a wrong answer is free to detect. On your repository, a wrong answer that satisfies a plausible assertion consumes reviewer attention, and the agent produces those at whatever rate it produces them regardless of what the benchmark said.

What the agent actually did with LEDG-3318 illustrates that. It found the FX snapshot lookup in billing/fx.py, changed it to use the downgrade date rather than the subscription start date, added a test asserting an 8.4% credit on a fixture invoice, and opened a patch with a confident description. The reviewer spent 40 minutes on it, established that the new snapshot date was probably right, found that the fixture had been constructed from the same wrong assumption the code made, and closed the patch with a question for Finance about which rate the contract actually specifies. A benchmark harness scoring that run would have needed a FAIL_TO_PASS list, and the only test in existence was the one the agent wrote for itself.

What the number is still good for#

None of this makes 58% a fabrication. It is a real measurement of a real capability, under conditions that are documented and reproducible, and the twelve-repository Python distribution it measures over is public. Read narrowly it supports three claims. The model is better at the localise-edit-verify loop than its predecessor was, on medium-sized Python code with test suites and idiomatic layouts. The improvement was large enough to survive a harness that everybody uses in roughly the same way, so it is unlikely to be entirely scaffold luck. And on tasks in your repository that happen to have the same shape — a reported defect with a settled expected behaviour and an existing test file to extend — you have some reason to expect the newer model to do better.

What it does not support is a transfer estimate. There is no defensible arithmetic that turns a 17-point benchmark gain into a predicted change in your merge rate, and the reason is that the two populations differ on the variable that does most of the work. It also does not rank agents under your harness, because the reported figure bundles the model with a scaffold, and the same model has been reported to swing by more than ten points depending on how retrieval, file editing and the retry loop are implemented around it. When a vendor reports a number, some unknown share of it belongs to their scaffold, and you are not buying their scaffold.

The scaffold effect is large enough to change conclusions rather than just add noise. The public leaderboard has repeatedly shown the same underlying model appearing at meaningfully different scores under different submissions, with the differences traceable to mundane engineering: whether edits are applied as unified diffs or as whole-file rewrites, whether the agent can run the test suite mid-task or only at the end, how many attempts the loop allows, and how much of the repository the retrieval step puts in front of the model. Those are all decisions your harness has already made, differently. So the honest reading of a vendor number is that it bounds what the model plus their engineering can do on that task family, and it tells you very little about the model plus your engineering on yours.

Two more limits are worth holding. The benchmark's repositories are public and old enough to be in training corpora, so instance-level memorisation is a live concern that the Verified subset reduces without eliminating; this is a reason to prefer trends over absolute levels. And the whole task family is issue resolution, which is one activity among many. The SWE Atlas work is useful reading here precisely because it argues for task sets covering the rest of the engineering loop — feature work, refactors, dependency upgrades, incident response — where the fraction of the job that is "decide what done means" is much larger.

Where this sits relative to the evaluation you already do#

Topic 10.9 established how to evaluate an agent as a system: define task success so two independent raters agree, score trajectories rather than only end states, hold cost against a budget. That machinery still applies here and this topic does not repeat it. The thing that changes for coding agents is what plays the role of the rater. For a scheduling agent, success was a set of assertions somebody wrote about the world after the run. For a coding agent, the natural oracle is the repository's own test suite, which topic 13.6 already put inside the loop as feedback. A test suite is a much stronger and much cheaper oracle than a judge, and it comes with the constraint that it only exists where somebody wrote it.

That constraint is the whole difficulty of the rest of this topic. A benchmark solves it by selecting for tasks where the oracle already exists — which is exactly why its scores do not predict yours, since your repository does not filter its incoming work that way. You cannot fix the mismatch by finding a better public benchmark, because any public benchmark measures other people's repositories with other people's conventions, other people's test coverage, and other people's definition of an acceptable patch. The measurement you need has to run against your code.

Which turns the problem into a supply question. A suite of five tasks somebody wrote by hand over a weekend is not a measurement instrument, and hand-authoring two hundred tasks with trustworthy oracles is months of work nobody will fund. Your repository already contains thousands of problems whose correct solutions were written, tested and reviewed by people who knew the codebase, and the next chapter is about turning that history into tasks.