Evaluating coding agents

Read these first

• Test execution as the verifier
• Agent evaluation

What you will know

• Build a regression suite of tasks on your own repository

  A regression suite of real tasks from your own repository predicts far better than public benchmarks.

• Measure pass@k rather than a single attempt

  pass@k acknowledges that agents are stochastic and one attempt is not the whole story.

• Track human review time as a first-class cost

  Human review load is a first-class cost: an agent that needs heavy review saves little.

• Explain why public benchmarks don't predict your repo

  Public benchmarks measure other people's repositories with other people's conventions.

• Show a change to the agent improving the suite

  Showing a change improved the suite is what makes agent work engineering rather than tinkering.

Chapters

1. 1

   A benchmark task arrives with its own oracle attached

   A model release lifts your agent's public score from 41% to 58% and nothing measurable moves on your own repository. Look first at task shape: benchmark items are resolved issues with a linked test that already fails and then passes, so the agent is graded on problems that come with an oracle. Your tickets do not. This chapter also fixes what a vendor's benchmark number legitimately tells you, without either dismissing it or over-reading it.

   Read chapter
2. 2

   Merged commits hand you oracles somebody already reviewed

   The practical route to hundreds of trustworthy tasks is history: take a merged change's description as the task, revert the code, keep the tests it shipped with, and require the agent to make them pass. Two biases arrive with the method — only well-tested areas produce tasks at all, and tests written alongside the fix hand over a specification the original engineer never had. The harness has to hold those shipped tests read-only and fail any run whose diff touches them, or the score measures oracle editing. You will also learn which merged changes to exclude, and why each exclusion is more than convenience.

   Read chapter
3. 3

   A suite grown around one agent scores fit, not capability

   The tasks came from your history and the harness, retrieval and context policy around them are yours, tuned for the incumbent — so running a vendor's agent through the same pipeline measures how well it fits your plumbing at least as much as what it can do. A fair comparison holds only the tasks and their oracles constant, gives each agent its own harness, and budgets for the tuning the incumbent has already had. When three weeks will not buy that, split the trial into separable questions — does it run in your environment at all, does it produce reviewable patches, does it win on tasks neither harness was tuned for — and decide in advance what a null result settles. Some conditions end the evaluation before it starts: an agent that cannot run inside your sandbox, or that needs source egress your policy forbids, yields no number about capability.

   Read chapter
4. 4

   pass@1 and pass@5 answer questions for different audiences

   An agent at pass@1 of 47% and pass@5 of 82% has produced two numbers, and a product manager reading the second is proposing autonomous merging. The gap is exactly the work of identifying the passing attempt, which a production loop cannot do unless its own checks are as strong as the eval's oracle. Here you learn which figure belongs in a merge-policy deck, and the condition that makes pass@5 an operational number rather than a research one.

   Read chapter
5. 5

   Two tasks flipping is not a five-point improvement

   A context-selection change moves the suite from 61% to 66% across 40 tasks at one run each, and the change is declared a success. Two tasks flipped, and a stochastic agent flips that many between identical runs. This chapter replaces the aggregate comparison with several runs per task and a paired analysis, so the question becomes whether a given task's outcome changed rather than whether a headline number moved.

   Read chapter
6. 6

   An agent that needs heavy review has not saved anybody time

   Median time from patch opened to first approval drops from 22 minutes to 17 while reviewers say the work feels heavier — because the metric stops at first approval and the burden lives after it, in re-review of revisions, in rework the reviewer does themselves, and in patches quietly abandoned. It also rewards patches that merely look convincing. Build a review-load metric with a defensible denominator, and design for reviewers knowing they are measured.

   Read chapter
7. 7

   A suite used to steer every change stops being a sample

   Six months of weekly runs, every harness change tuned against them, a score now at 88%, and engineers who insist the agent is no better than it was in month two. The suite became the target. This chapter covers how that reads from inside, why held-out tasks and periodic re-mining from recent history slow it down, and why a rising score alongside flat perceived quality is the specific pattern to distrust rather than celebrate.

   Read chapter