Each extra file adds a plausible wrong answer

The patch was clean, well-formed, and applied on the first attempt. It also began with this line:

from reporting.legacy.money import format_currency

billing/invoice.py already imports format_currency, from billing.money, and the two round differently: billing.money uses banker's rounding on Decimal at two places, which sends an exact half to the nearest even digit so that a long series of amounts does not drift upward, and reporting.legacy.money does round(value, 2) on a float, which is where the eleven-cent discrepancy in the Japanese VAT test came from. The agent had not hallucinated the module. It was in context, along with 38 others, because the context builder had been asked for files mentioning currency anywhere in the monorepo and had returned all 40 of them.

Three implementations of that name were visible to the model at once.

Nothing in the retrieved text says which of the three billing/invoice.py is entitled to use. The ownership rule exists — it is in a CODEOWNERS file, which maps each directory to the team whose review a change there requires, and in the head of a design doc — and neither of those was in the window. What the model had was three functions with the same name, each with a plausible call site, and a task that needed one of them. It resolved the ambiguity the only way it could, by preference: the legacy signature takes a float and a symbol, the VAT calculation in the ticket was written with floats in the example, and that made the legacy version the easier fit for the code it was about to write.

Recall improved and the patches got worse#

This was not a one-off. The same builder had been expanded from 6 files per task to 35 a few weeks earlier, on the strength of an offline retrieval measurement, and the two numbers moved in opposite directions across a 240-task evaluation set. The first of them is recall, the share of the files a correct patch had to touch that the builder actually put in the window, which is how retrieval quality is usually scored offline. The second is the first-attempt pass rate in CI — continuous integration, the automated build-and-test run that decides whether a patch may merge — which is the outcome anyone actually cares about.

The natural reading of the top row is that retrieval got better, and it did — the 35-file builder almost always has the file the patch needs to touch, where the 6-file builder missed it three times in ten. The natural next step is to call the pass-rate drop noise or an unrelated regression, and that reading does not survive contact with the diffs. Pulling the 41 tasks that regressed and diffing the patches against the 6-file arm's patches for the same tasks, 27 of them differ by importing, subclassing or copying the conventions of a file that appears only in the 35-file arm's context. The extra 29 files were not inert. They were 29 more answers the model could give.

This is what "relevant" buys you and what it does not. Every one of those 40 currency files is relevant by any reasonable definition; a human asked to mark the irrelevant ones would mark none, exactly as in the earlier work on selection where twenty on-topic passages carried six distinct facts. The difference in a codebase is that a redundant passage is merely wasted tokens, while a redundant implementation is a fork in the road. Prose that restates a fact reinforces it. Code that restates a capability with different semantics creates a choice, and the model has to make that choice with whatever evidence is in the window.

The comparison is worth defending before drawing conclusions from it, because a 17-point difference on a small evaluation set can be an artifact of sampling. Both arms ran the same 240 tasks with the same model and temperature, and the analysis is paired: for each task, did the outcome change and in which direction. Fifty-eight tasks moved from pass to fail and seventeen from fail to pass, which under a null hypothesis of no effect would be a coin flip landing 58–17, and it is not that. Three repeat runs of the 35-file arm scored 41%, 43% and 40%, so the run-to-run spread is about 3 points against a gap of 17.

An engineer facing the same three format_currency implementations does not have this problem, and the reason is instructive. They resolve it with signals that live outside the file contents: which package they are in, what the neighbouring code imports, git blame showing the legacy module untouched since 2019, a deprecation note in a README, and the knowledge that reporting/ belongs to another team. An embedding of the file body encodes none of that, so a retrieval system ranks the deprecated implementation and the live one on the strength of their shared vocabulary, which is high precisely because one was copied from the other. Restoring those signals — package boundaries, import-graph distance, last-modified date, ownership — is possible and is not something similarity search does for you.

The base rate makes this worse than it sounds. A mature monorepo has near-duplicates everywhere, because that is what a decade of teams shipping independently produces: two date helpers, three retry decorators, a deprecated ORM layer that still compiles, a vendored fork of a client library, and the same four utility names in five packages. Widening retrieval widens the set of these the model can see, and the widening is not selective — a query for currency handling pulls the good implementation and the deprecated one with roughly equal enthusiasm, since they use the same vocabulary. That is why they are near-duplicates.

What a larger window fixes#

The proposal that follows this result is usually to move to a model with a much larger window and include even more. It is worth being precise about which of the problems in this topic that solves, because it solves some of them.

A 1M-token window makes the arithmetic of the first chapter comfortable. A first-turn fill of 68k against a loop that grows by 24k per attempt is no longer near any limit, so patches stop truncating mid-function, tool results stop being trimmed at arrival, and the eviction machinery of the second chapter stops firing on ordinary runs. Those are real gains and they remove a real class of failure.

None of that touches the mechanism above. With three format_currency implementations in a 1M window there are still three, and the model still has no rule that says which one billing/invoice.py may use; giving it 200 more files raises the chance that a fourth implementation joins them. Cost and latency move the wrong way at the same time — the 35-file arm already tripled time to first token, and a run that iterates four times pays that on every turn. There is also good reason to expect the selection effect to persist rather than to wash out with scale: retrieval quality degrades with position in long contexts, and a distractor sitting at 400k is still readable, still syntactically valid, and still a candidate answer.

So the honest answer to the proposal is that a bigger window is a fix for the budget problem and not for the selection problem, and the two failures look different in the logs. The budget failure is truncation and eviction, which is loud. The selection failure is a well-formed patch that imports the wrong thing, which is silent until the eleven cents show up in a reconciliation report.

Two explanations, one experiment#

There is a rival explanation for the pass-rate drop that the diffs above do not rule out. The 35-file arm has the same 6 files the small arm had, plus 29 others, but they are not in the same place: the file that actually needs editing might now sit at position 22 of 35 rather than position 2 of 6, buried in the middle of the window where attention is thinnest. That is displacement rather than distraction, and the fix for it is ordering rather than exclusion, so it matters which one is operating.

Two ablations separate them, an ablation being a run in which exactly one ingredient is removed or altered so that whatever changes in the outcome is attributable to that ingredient, and they are cheap because both reuse the existing eval set.

The first holds the file set fixed and permutes the order. Run the 35-file arm three ways — retrieval-score order, the file-to-be-edited first, and the file-to-be-edited buried at the midpoint — with identical content in all three. Any difference in pass rate is positional by construction, because no arm has an option the others lack. In the run that produced the numbers above, moving the target file to the front recovered 6 points, from 41% to 47%, which says position is real and is not the whole story.

The second pads the 6-file arm with inert filler up to the same token count. Inert means it adds no rival implementation and no conflicting convention: vendored license texts, a changelog, docstrings from an unrelated service in another language. If pass rate falls under inert padding, the damage is dilution and position. If it holds, the damage is coming from the content of the extra files. Here the padded 6-file arm scored 55% against the unpadded 58%, so roughly 3 points of the 17-point gap is dilution and the remaining 14 belong to the rival implementations themselves.

The decomposition matters because it prescribes different work. Fourteen points of distraction is an argument for excluding files, which is the inclusion policy at the end of this topic. Three points of dilution is an argument for ordering and for keeping the fill small on general principle, and it is the cheaper of the two to act on.

One cheap intervention falls straight out of the decomposition, and it is worth naming here even though the policy work comes later. Most of the fourteen points come from duplicate names, so deduplicating candidates by exported symbol before filling removes the ambiguity without removing coverage: when three modules export format_currency, keep the one closest to the target file in the import graph and drop the others. On this eval set that single rule recovered 9 of the 17 points on its own, which is more than reordering achieved and considerably less work than rewriting the retriever.

What none of these arms measure is the cost of being wrong in the other direction. The 6-file builder misses the file a correct patch has to touch 29% of the time, and a patch that edits a file it never saw is not a subtle failure. Somewhere between the two arms is a builder that keeps the recall of the large one without importing its rival implementations, and getting there means being more precise about what "including a file" is allowed to mean. Sending all 900 lines of a module to make one 30-line function visible is what put most of the 29 extra implementations in the window in the first place, which is the trade the next chapter is about.