pass@1 and pass@5 answer questions for different audiences

The evaluation report has two lines on it. Against the 187-task suite, the agent scores pass@1 of 47% and pass@5 of 82%. A planning document circulated the following week proposes that agent patches which pass CI merge without human review, citing the second figure and observing that a 4-in-5 success rate is better than the team's historical rate of pull requests that need a follow-up fix. The proposal is arithmetically reasonable and it is reading a number that does not describe the system anybody would deploy.

The two figures were computed from the same runs. Each task was attempted 10 times, and the harness recorded, for each attempt, whether the oracle tests passed. pass@1 is the fraction of individual attempts that resolved their task, averaged over tasks. pass@5 is the probability that at least one of five randomly drawn attempts resolves the task, which the harness estimates from the ten it has rather than by rerunning:

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """n = attempts sampled, c = attempts that resolved the task, k = budget."""
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

# suite-level figures
def suite_pass_at_k(results: list[tuple[int, int]], k: int) -> float:
    return sum(pass_at_k(n, c, k) for n, c in results) / len(results)

# task with 10 attempts, 2 of which resolved
pass_at_k(10, 2, 1)   # 0.200
pass_at_k(10, 2, 5)   # 0.778

The estimator comes from the Codex paper in 2021, where it was introduced for a much simpler setting: generate k program completions for a function signature, run the hidden unit tests, count the problem solved if any completion passes. It was designed as a research measure of what a model can produce given a budget, and the "given a budget" part carried an unstated assumption that made sense in that setting and does not survive the move to production.

The 35-point gap is somebody's job#

Consider one task from the suite in detail. LEDG-2041__a9c31f asks for idempotency on a webhook handler, and across ten attempts the agent resolved it three times. The seven failures were not identical: four wrote a deduplication key that omitted the tenant id, two introduced a race by checking and inserting non-atomically, and one produced a patch that the harness rejected for touching a protected test file. Its pass@1 is 30% and its pass@5 is 83%.

For the 83% to be a fact about a running system, something in that system has to look at five candidate patches, three of which are wrong in ways that a careful reviewer would catch and a smoke test would not, and pick a good one. In the evaluation, that job was done by the oracle — the shipped tests from the original merged change, which the agent never saw. In production, the shipped tests do not exist, because the whole reason the task is open is that nobody has written them yet. The gap between 47% and 82% on the suite is the size of the selection problem, measured in points, and it is a measurement of how much value sits in a component the deployed system does not have.

This is where a coding agent differs from the Codex setting in a way that matters. There, the hidden unit tests were a stable artefact of the benchmark and the deployed product was an editor autocomplete where a human picked among suggestions immediately, so k was genuinely a user-facing budget. Here the equivalent human is a reviewer who is being proposed for removal, and the tests that would let a machine do the picking are the very thing the task exists to produce.

What the running loop can actually check#

The agent does have checks of its own. Topic 13.6 established the loop: apply the patch, run the impacted subset, feed failures back, retry. So the deployed system is not choosing blindly, it is choosing by whatever its own verifier says. The question is how strong that verifier is compared with the oracle, and that is measurable directly rather than by argument.

def loop_selected_rate(task_runs) -> dict:
    """Grade the attempt the LOOP would have shipped, not the best attempt."""
    oracle_any = loop_ok = loop_and_oracle = 0
    for task in task_runs:
        attempts = task.attempts[:5]
        oracle_any += any(a.oracle_passed for a in attempts)
        chosen = next((a for a in attempts if a.self_checks_passed), None)
        if chosen is not None:
            loop_ok += 1
            loop_and_oracle += chosen.oracle_passed
    n = len(task_runs)
    return {
        "pass_at_5": oracle_any / n,          # 0.82
        "loop_shipped_something": loop_ok / n, # 0.91
        "loop_shipped_correct": loop_and_oracle / n,  # 0.58
    }

Run over the 187-task suite, that returns 82% for pass@5, 91% for the rate at which the loop produced a patch its own checks were happy with, and 58% for the rate at which the patch it would actually have shipped satisfied the oracle. The 91% is the number to be frightened of, because it is the rate at which the system declares success, and it exceeds the rate at which it is right by 33 points. Under an autonomous merge policy those 33 points are patches that go to main with a green pipeline and a defect in them.

The 58% is the honest operational figure for a five-attempt budget, and it is worth naming separately in the report — the loop-selected rate at k=5 — because it is the only one of the three that describes a system anybody could run. It sits above pass@1 of 47%, which is the correct shape: sampling five times and letting the loop pick does help, just far less than 82% suggests.

It is worth being precise about what "self_checks_passed" covers, because teams differ and the strength of the gate is the whole variable. On this harness it means the patch applied cleanly, the impacted test subset ran green, the linter and type checker were clean, and any tests the agent wrote itself passed. That is a real gate and it rejects a lot; the 91% figure is what remains after it. What it cannot do is notice that the deduplication key omitted the tenant id, because no existing test exercises multi-tenant webhook replay and the agent's own test used a single tenant. A verifier can only reject what it can express, and the strength of a coding agent's verifier is bounded by the coverage of the code it is editing — which is the same constraint that decided which parts of the repository could produce tasks in the first place.

Which number goes where#

For a merge-policy deck, the figure is the loop-selected rate, because merge policy is a question about what the system does unattended. pass@1 belongs alongside it as the per-attempt cost basis, since it determines how many attempts reviewers absorb. pass@5 does not belong in that deck at all, and the reason to say so explicitly is that it is the largest number available and it will be quoted if it is present. It earns its place in a different document, the one where harness engineers decide what to build next, because a large pass@1-to-pass@5 gap says the capability is there and the selection is missing, which points at verifier work rather than at model work.

The consistency measure from topic 10.9 is a fourth question again, and mixing it in here confuses people. pass@5 asks whether any of five attempts works; pass^5 asks whether all five do. They move in opposite directions as an agent gets more variable, and a system can look excellent on one and poor on the other.

When pass@5 is an operational number#

There is a condition under which the 82% becomes real, and stating it precisely also explains why it is rarely met. pass@k is operational when the running system's own checks are at least as strong as the eval's oracle, because then the loop can identify a passing attempt with the same reliability the grader did.

For coding agents that condition is occasionally satisfiable, and the cases share a shape: the specification already exists as executable checks before the agent starts. A failing test in CI that needs fixing is one — the oracle is the test, the loop has the test, and if the agent can make it pass without touching it, the loop's judgement and the grader's judgement are the same judgement. A dependency upgrade graded by the existing suite is another. A migration from one internal API to another, where the acceptance criterion is that the full suite stays green and the output of a codemod — a scripted bulk edit applied mechanically across the tree — compiles, is a third. In all of them the eval oracle and the production verifier are literally the same artefact, so there is no gap for selection to fall into.

Feature work and defect reports where the tests do not yet exist are the case where the condition fails, and they are most of the queue. The agent can write its own tests, and those tests are evidence of something, and they are not the oracle — a test written by the same process that wrote the patch shares the patch's misunderstandings, which is the same problem as the tests-written-alongside-the-fix bias from the mining chapter, arriving from the other direction. When the agent's self-authored tests are what the loop selects on, the 91% figure above is exactly what you get.

The practical consequence is that a merge policy should be scoped by task class rather than set globally. The tasks where the oracle pre-exists can plausibly merge on a green pipeline; everything else needs review, and the evaluation should report the loop-selected rate separately for each class so the policy has something to stand on. On the internal suite, splitting that way gave 79% loop-selected on the 46 tasks with pre-existing failing tests and 51% on the other 141, which is a much more actionable pair of numbers than a single 58%.

Running each task ten times to compute any of this is not free — the 187-task suite at ten attempts is 1,870 agent runs, roughly $310 in tokens and 26 hours of wall clock on eight parallel workers. That expense buys something beyond the k-curve, which is a per-task distribution rather than a per-task verdict, and that distribution turns out to be what makes it possible to tell whether a change to the agent did anything at all.