Labels drawn from merged commits, not from judgement

The dashboard says recall at ten is 0.91 against a labelled set of 180 tasks. The other dashboard says the coding agent's first-attempt compile rate in production is 47%, and that the single most common failure, at 38% of the broken builds, is an unchanged caller of a function whose signature the agent altered. Leadership reads the two numbers together and concludes the retriever is fine and the model is the problem. Both numbers are correct measurements. They are measurements of different things, and the gap between them is entirely explained by how the 180 tasks were labelled.

Two true numbers about the same system#

The labelled set was built the way most of them are. Two engineers took 180 historical tickets, read each one, searched the repository, and wrote down the files they judged to be related. It took nine days and it produced a set that is genuinely useful for some purposes and structurally unable to measure the failure that is breaking production.

Consider what a caller is, from the labeller's point of view. A ticket says "change calculate_proration to take an explicit cycle length instead of deriving it". The engineer labelling that ticket opens billing/proration.py, notes it, opens the test file, notes it, perhaps notes the calendar module. The fourteen call sites across seven modules are not on screen, they are not conceptually interesting, and enumerating them requires running a find-references query that a person labelling 180 tickets in nine days does not run. So the label for that task is three files, and a retriever that returns exactly those three files scores a perfect recall of 1.0 on it while leaving the agent unable to see eleven of the fourteen files that must change for the build to pass.

The metric is not merely noisy about this failure. It is blind to it in a correlated way: the files that judgement-based labelling systematically omits — callers, tests that assert on the old behaviour, the config schema that has to gain a key — are the same files whose omission produces a broken build, because they are the files that must change rather than the files that explain the subject. Chapter five's split between the read set and the edit set is what a human labeller reproduces badly; people label the read set well, since that is what they would open to understand the ticket, and label the edit set only where it coincides with the read set.

A second inflation is hiding in the same number. Recall at ten counts a file as retrieved if any chunk of it appears in the top ten, so a file represented in the results only by its module docstring counts as a hit, which is precisely the case chapter one opened with. Two independent generosities compound: the label set omits the hard files, and the hit criterion is satisfied by the easy chunks.

Ground truth you did not have to invent#

Merged changes are a supply of tasks whose correct answer was determined by something other than an opinion. Somebody described a problem, somebody produced a change, the change was reviewed and merged and did not get reverted, and the diff records exactly which files had to change. The label is not a judgement about relatedness; it is a record of what happened.

# One task per merged PR, on the mainline, over the last two quarters.
gh pr list --state merged --base main --limit 400 \
   --json number,title,body,mergeCommit,files,closingIssuesReferences \
> prs.json

# The edit set: files the merge changed, minus the ones nobody chose.
git show --name-only --pretty=format: "$MERGE_SHA" \
  | grep -Ev '(^|/)(vendor|node_modules)/' \
  | grep -Ev '\.(lock|sum|pb\.go|generated\.ts)$'

# The index must be built from the state the author started from.
git rev-parse "${MERGE_SHA}^1"     # parent on main = the commit to index

Three details in that snippet do most of the work. Generated files and lockfiles are excluded, because they change on every merge and would flood every label with the same twenty paths. The parent commit is what gets indexed, since running the retriever against a repository that already contains the fix is a leak that inflates every number and is the most common error in home-grown code retrieval evaluations. And the task text comes from the linked issue rather than the pull request title where one exists, because a PR titled "fix proration for partner subs in serializers.py" hands the retriever the answer.

Leakage in the task text is worth being aggressive about. Strip file paths, strip symbol names that appear in the diff, and drop the pull request body entirely if it contains a code block from the change. What remains is closer to what an agent actually receives — a description of a symptom, sometimes a stack trace, sometimes nothing but a sentence — which is the condition the retriever has to perform under.

The edit set falls out of the diff. The read set has to be approximated, and a reasonable approximation is derivable from the diff too: for every identifier introduced or modified in the change, resolve it to its definition and include that file. If the patch starts calling rate_limited, then middleware/rate_limit.py was read. If it constructs a Subscription, the model file was read. This misses files that were read and produced no change — the class docstring whose invariant the author respected leaves no trace in the diff — and it is a systematic undercount in a known direction, which is a better property than an undercount in an unknown one. Label the two sets separately and report against them separately, weighting the edit set as the number that matters.

Judgement does not disappear from this process, it moves. Instead of asking two engineers to decide which files are related, which is the question the retriever is supposed to answer, you are asking them to decide which merged changes make reasonable tasks and whether a task's text has been stripped of leaks. Those are questions a person can answer reliably in under a minute per task, and getting one wrong adds noise rather than a systematic bias in the retriever's favour. That is the property that matters: a labelling process can be imperfect as long as its errors are not correlated with the thing being measured, and "the labeller did not run a find-references query" is a correlated error while "the labeller included a drive-by typo fix" is not.

Some pull requests should not become tasks at all. A rename touching 240 files, a dependency bump, a formatting pass, a revert — these produce labels that are either trivially large or meaningless, and a filter of "between one and fifteen non-generated files, not a revert, has a linked issue or a descriptive body" keeps a usable set. On the monorepo in question, 400 merged pull requests reduced to 118 usable tasks, which is enough to see a 5-point difference in a metric and not enough to see a 1-point one.

The metric that tracks the compile rate#

With honest labels, the choice of metric decides whether the number means anything. Mean recall at ten over all edit-set files is the conventional choice and it is the one that produced 0.91, because it averages over files and a task with six files where four were retrieved contributes 0.67 rather than zero. A patch does not work that way: a change that must touch six files and touches four is broken.

The all-files column is the one that reconciles the two dashboards. With an average of 3.4 edit-set files per task and independent misses, a per-file recall of 0.91 would give an all-files rate of about 0.73, and misses are not independent — they cluster on callers, which cluster on exactly the tasks that change signatures — so the real figure lands lower. An all-files recall of 0.51 sitting under a 47% compile rate is not a coincidence needing explanation; it is very nearly the same measurement taken twice.

Whether to replace the retriever is not answerable from the original numbers, and the honest position to take into that meeting is that the retriever has not been measured yet. Replacing it on the current evidence risks spending a quarter to swap in something that scores 0.93 on the same uninformative set. The evidence that settles it is the rebuilt evaluation: run the existing retriever against merged-commit labels at the parent commit, and look at all-files recall on the edit set and at caller recall on the subset of tasks that change a signature.

The week's work is bounded. Sample 60 merged pull requests stratified so that at least 20 of them change a public signature, build the labels with the script above, index each parent commit, and run the retriever. Stating the keep condition before seeing the result is what stops the analysis from being a rationalisation: if all-files edit-set recall comes in at or above 0.80 and caller recall at or above 0.90, the retriever is not what is producing a 47% compile rate and the investigation moves to how context is assembled and how the agent verifies its own patch. Anything below that, and the specific gap is now visible enough to fix — a caller recall of 0.44 is a reverse-edge problem with a known repair, not a reason to start over.

What a merged commit still will not tell you#

The sampling is biased in ways worth holding onto. Merged changes are changes somebody managed to make, which excludes the tasks that were abandoned because nobody could find the right file — the exact population where retrieval fails hardest. They are written by people with repository knowledge an agent does not have, so the diffs are cleaner and more targeted than an agent's would be. They frequently contain drive-by edits, a typo fix in an adjacent file, that enter the label set as noise. And a squashed merge collapses an author's exploratory commits, hiding the three files they opened, misread, and backed out of, which is information about the read set that no longer exists anywhere.

There is also a limit to what any retrieval metric can establish. A perfect edit-set recall means the agent could see every file it needed; it does not mean the patch is right, and a retriever tuned to the point where the correct files are always in the context has done its job while the compile rate can still be poor for reasons that belong to generation and verification. Retrieval quality is a necessary condition rather than a sufficient one, and treating a retrieval metric as a proxy for agent quality is the same category error as treating a topical label as a proxy for what must change.

That distinction — necessary rather than sufficient — has a sharper form when a code retriever is put next to a document retriever, because the same metric at the same value carries very different consequences on the two sides. The last chapter of this topic is about why a number that is perfectly adequate for a documentation assistant is not adequate for a patch, and about what happens when a document RAG stack is pointed at a repository without changing anything.