Equal recall hides unequal usefulness on code

Two teams present in the same review. The documentation assistant reports recall at ten of 0.84 on its evaluation set, a satisfaction score of 4.4 out of 5, and about 900 questions a week. The coding agent reports recall at ten of 0.84 on its evaluation set, a first-attempt compile rate that will not climb past the low fifties, and a steady stream of complaints that its patches miss a call site. Somebody asks the reasonable question: if the retrieval numbers are identical, why is one of these products and the other one a science project?

The same number, two different consequences#

The answer is in what happens to the output when retrieval misses one source. A question like "how do we rotate the signing key for partner webhooks?" has an answer that can be assembled from three of the four relevant documents, and the fourth document, which covers the staging environment's variation, would have made the answer more complete without making the three-document answer wrong. The reader gets a correct answer with a gap, notices the gap or does not, and asks a follow-up. Prose degrades continuously.

A patch does not. A change that must touch six files and touches five does not produce five-sixths of a change; it produces a build failure, or worse, a change that builds and leaves one caller passing the old argument shape into a function that no longer accepts it. The grading function is conjunctive, meaning every required file has to be present for the task to count as passed and any single absence fails the whole of it, and conjunctive grading turns a respectable per-item recall into a poor per-task one. With an average of 3.4 required files per task, a per-file recall of 0.84 gives an all-files rate of about 0.55 if misses were independent, and lower in practice because misses concentrate on callers, which concentrate on the tasks that change interfaces. The documentation system's 0.84 is measured against a requirement of roughly "at least one good source", where 0.84 per source means something close to 0.98 per question.

The rest of the difference follows from the same place. A code task is graded by machinery that runs the result, in the sense chapter 13.1 established, so a retrieval gap surfaces as a red build rather than as a vaguely unsatisfying paragraph. That is the good news about code and the reason the gap is visible at all; it is also why an aggregate retrieval metric borrowed from document search reports comfortable numbers about a system that fails more than half the time.

What breaks when a document stack is pointed at a repository#

The port is usually done in an afternoon and looks like a win, because the pipeline runs and the numbers come out familiar. Extract text from every file, chunk at 1,000 characters, embed with a general text model, cosine search, top ten into the prompt. Two assumptions travel with that pipeline and neither survives contact with a repository.

The first is that a chunk is a self-contained unit of meaning. That holds for prose, where a paragraph cut from an article still says something true, and fails for code, where a fragment is either syntactically incomplete or semantically dependent on a scope that was left behind. The symptom is the one from the first chapter of this topic — a patch written from a docstring, a method patched in violation of an invariant stated on its class — and it shows up in review as a large, confident diff that removes a guard nobody meant to remove.

The second is that relevance is similarity. That holds for prose, where a document about a subject and a question about that subject share vocabulary by construction, and fails for code, where the file that must change is frequently connected to the task by a call edge rather than by a word. The symptom is the missing caller and the untouched test, and it is invisible to the pipeline's own metric because the labels were written by the same topical intuition that the retriever implements.

Both breakages sit outside the encoder. One is upstream: chunks are cut before anything is embedded, and no encoder can restore a function body that was never handed to it. One is downstream: the required file is related to the task by an edge in a graph, and no encoder recovers a relationship that is not expressed in the text it encodes.

What a code encoder is actually good at#

That is not a reason to dismiss code-trained embedding models, which do something real and specific. They tokenize identifiers compositionally, splitting calculateProration and calculate_proration into parts, so a task that says "prorate" matches a symbol that never contains that exact string — a general text model tokenizes calculateProration into fragments that carry little of the meaning, and the match simply does not happen. They are trained on pairs of natural-language descriptions and implementations, which is precisely the retrieval direction a ticket needs, mapping an English statement of intent onto code that has no English in it. And they have learned that a docstring and an implementation are different kinds of text, which reduces the systematic advantage that prose-shaped chunks held in the first chapter, because the encoder is no longer scoring everything by how much it reads like a sentence.

On the rebuilt evaluation from the previous chapter, swapping a general text encoder for a code-trained one moved mean file-level recall at ten from 0.71 to 0.79 and all-files edit-set recall from 0.44 to 0.51. That is a genuine eight-point gain for a two-day change, and it is roughly a fifth of the distance to a system that reliably puts every required file in the context.

Where the gain concentrates is informative. Almost all of it lands on tasks whose text names a concept that appears in the repository under a different surface form — "prorate" against calculateProration, "retry backoff" against exponential_delay_ms, "soft delete" against tombstoned_at. Almost none of it lands on the tasks that were failing for structural reasons, because a task about a missing caller was never a vocabulary problem. Averaged together these produce a modest overall improvement that reads as incremental progress on one problem, when it is decisive progress on one problem and no progress at all on another.

The parts that are not a retrieval model at all#

The largest remaining lever is not an index at all. Give the agent the tools a person uses — find references, go to definition, grep for a literal, list a directory, read a symbol by name — and let it search during the task rather than receiving a fixed context assembled before the task begins.

[
  {"name": "grep", "description": "Regex search across tracked files",
   "input_schema": {"type": "object",
     "properties": {"pattern": {"type": "string"},
                    "path_glob": {"type": "string"}},
     "required": ["pattern"]}},
  {"name": "find_references", "description": "All call sites of a symbol",
   "input_schema": {"type": "object",
     "properties": {"symbol": {"type": "string"}},
     "required": ["symbol"]}},
  {"name": "read_symbol", "description": "Source of one symbol with its enclosing scope",
   "input_schema": {"type": "object",
     "properties": {"symbol": {"type": "string"}},
     "required": ["symbol"]}}
]

A trace on the proration signature task runs about eight calls: grep for calculate_proration, read the symbol, find references, read two of the fourteen call sites, grep for the test file, read it, and only then start editing. That sequence retrieves all fourteen callers exactly, because find_references is not an approximation of the call graph, it is the call graph. The observation behind Jason Liu's account of why grep outperformed embeddings in a SWE-bench agent — SWE-bench being the public benchmark of real GitHub issues where an agent's patch is graded by whether the repository's own test suite passes afterwards — is this one: for a query that is an exact identifier, exact search is both cheaper and more accurate than any vector approximation of it, and much of what a coding agent needs to find is an exact identifier.

Agentic search has its own costs, and they are not small. Eight to fourteen tool calls add 20 to 60 seconds of latency and several tens of thousands of tokens of transcript, and the loop only works when the agent can guess a starting name — on a 4-million-line monorepo where the task says "exports are slow for enterprise tenants" and names no symbol, a grep-first agent flails. The arrangement that works is hybrid: the index supplies seeds and orientation for a task with no obvious symbol, and the tools supply exactness and completeness once a symbol is in hand. Reverse edges in particular are better served by find_references at inference time than by a precomputed graph, because the tool cannot be stale.

Knowing whether any of this helped is the previous chapter's machinery applied to each change: all-files recall on the edit set, caller recall on the signature-change subset, and, in production, the first-attempt compile rate together with the share of failures attributable to an unchanged caller. That share is the number that should move; if it falls from 38% to under 10% while the compile rate rises, the retrieval work is what did it, and if the share holds steady while the compile rate rises, something else did.

What is still open#

Repository-scale retrieval has no clean public benchmark of its own. It is almost always measured through patch success on SWE-bench-style tasks, which folds retrieval quality together with generation quality and makes it hard to attribute an improvement to either. Teams that want to know whether their retriever improved end up building the merged-commit evaluation themselves, on their own repository, which is the right thing to do and is not a substitute for a shared yardstick.

Scale is unresolved past a certain size. The techniques in this topic have been described against a 4,300-file repository, where a symbol index is a few hundred megabytes and a full re-index takes minutes. At ten million lines, incremental indexing becomes mandatory, the import graph has hub nodes with degree in the thousands, and the candidate pool at one hop is already larger than any context window. And the hardest case is the one where the required file is in a different repository entirely — a caller in another service, a schema owned by another team — where no amount of local retrieval reaches it and the only signals available are a published interface and a contract test.

What the reader now has is a retrieval stack with defined parts: chunks cut on syntax with their scope attached, a graph of import and call edges with runtime keys bridging what the parser cannot see, a ranker that lets structure outweigh vocabulary, a budgeted cutoff, and an evaluation drawn from changes that actually happened. The next thing is to put that context in front of an agent that edits files and runs the verifier, and to watch what it does with a context that is correct — because a complete retrieval only removes one of the reasons a patch can be wrong.