One ordered list out of four disagreeing signals

Take the eight candidates from the rate-limiting ticket and put every available signal next to each one. Similarity to the task text, the number of hops from the nearest seed along resolved import and call edges, whether the chunk defines a symbol or merely mentions one, and how often the file has been committed alongside the seeds in the last hundred commits that touched either. Four columns of numbers, four different files at the top.

Similarity ranks the middleware first and the file that must change seventh. Co-change ranks tests/test_export.py first. Definition-site alone ties three files and separates nothing. Hop count puts four files at distance one and gives no order among them. Every signal is individually weak, at least one of them is right about every candidate, and none of them is right about all of them — which is the ordinary condition of ranking rather than a defect to be engineered away.

Making scores comparable before adding them#

The first instinct is a weighted sum over the raw columns, and it fails for a reason that has nothing to do with the choice of weights. The four quantities live on incompatible scales. Cosine similarity against a task description occupies a compressed band, typically 0.25 to 0.90, with most of the discriminating information in a range of about 0.15. Hop count is a small integer where lower is better. Definition-site is a boolean. Co-change is a ratio out of a denominator that varies per query. Adding them directly means whichever column happens to have the widest spread dominates the result, and the weights end up compensating for measurement scale rather than expressing a belief about importance.

There are two standard repairs and they behave differently.

Reciprocal rank fusion discards the magnitudes and keeps only the orderings, scoring a candidate by the sum of 1 / (k + rank) over the signals that ranked it.

def rrf(rankings: dict[str, list[str]], weights: dict[str, float], k: int = 60):
    scores: dict[str, float] = defaultdict(float)
    for signal, ordered in rankings.items():
        for rank, doc in enumerate(ordered, start=1):
            scores[doc] += weights[signal] / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

It needs no calibration, it cannot be broken by one signal drifting, and it is a reasonable thing to ship on day one. The price is that it throws away margin: two candidates at 0.88 and 0.87 similarity are treated as a full rank apart, and a candidate that one signal is overwhelmingly confident about gets the same first-place credit as one that barely won. The constant k controls how much a top rank is worth relative to the tail, with 60 being the conventional default and smaller values sharpening the top.

Per-query normalisation keeps the magnitudes and puts them on a common footing, usually by converting each column to a z-score across the candidate pool for that query — a z-score being the raw value restated as the number of standard deviations it sits above or below the pool's mean, so that a boolean, a small integer and a cosine all end up in the same units.

def normalize(values: list[float]) -> list[float]:
    mu = mean(values)
    sd = stdev(values) or 1.0
    return [(v - mu) / sd for v in values]

def score(cands, w):
    sim   = normalize([c.similarity for c in cands])
    hops  = normalize([-c.hops if c.hops is not None else -4 for c in cands])
    defn  = normalize([1.0 if c.is_definition else 0.0 for c in cands])
    co    = normalize([c.cochange for c in cands])
    return [
        w["sim"] * s + w["hops"] * h + w["defn"] * d + w["co"] * c
        for s, h, d, c in zip(sim, hops, defn, co)
    ]

Normalising per query rather than globally matters, because the absolute similarity level shifts with the length and specificity of the task text — a two-line ticket produces a pool where everything sits near 0.4, and a pasted stack trace produces one where everything sits near 0.7. What is stable across queries is the relative position within a pool.

Exact matches on distinctive literals deserve separate treatment from either scheme. The string /export appears once in the repository, and a single occurrence of a rare literal is not evidence of the same kind as a cosine of 0.83; treating it as another term in the average dilutes it. The workable arrangement is a hard boost applied after fusion — a candidate containing an exact match on a literal extracted from the task text is promoted above the unboosted pool — with the boost restricted to literals that are rare in the repository, since promoting every candidate that contains the string id helps nobody.

The weights that let a wordless file surface#

Running the normalised sum with weights of 1.0 on similarity, 1.4 on hops, 1.2 on definition-site and 0.6 on co-change reorders the pool so that routes/export.py comes second and both runbook documents fall out of the top five. The arithmetic is worth following: routes/export.py is 1.4 standard deviations below the mean on similarity and roughly 1.2 above it on each of the other three, so the three structural terms have to outweigh the one textual term for it to surface at all. That is the substantive claim buried in the weight vector — for a repository where the files that must change often do not use the task's vocabulary, structure has to outrank vocabulary, and a ranker whose similarity weight dominates will never surface a wordless file no matter how good its encoder is.

Those weights are not defensible as intuitions. They came from a grid search over a labelled set of 140 tasks, optimising recall of the edit set at a cutoff of ten, and on a different repository they come out different — a monolith with strong naming conventions supports a higher similarity weight, and a polyglot repository with a weak import resolver supports a lower hop weight because the graph is less trustworthy. Weights tuned on nothing are weights tuned on the last three tickets somebody happened to look at.

The co-change signal is the one most teams have never built, and it is cheap enough to be worth describing concretely. Walk git log --name-only over the last few thousand commits, and for every commit touching between two and fifteen files, increment a counter for each pair of files in it. That gives a symmetric matrix of co-edit counts, which for a 4,300-file repository is a few million non-zero entries and fits comfortably in memory. Querying it is a lookup: given the seed files, sum each candidate's co-edit count with each seed, divided by the candidate's total commit count so that a file touched in every commit does not win everything.

def cochange_score(candidate: str, seeds: list[str], m: CoEditMatrix) -> float:
    joint = sum(m.pairs[(candidate, s)] for s in seeds)
    return joint / (m.commits[candidate] ** 0.5 + 1)

Two adjustments keep it honest. Commits touching more than about fifteen files are excluded from the counts entirely, since a formatting pass or a dependency bump creates a dense clique of co-edits between files that have nothing to do with each other, and a handful of such commits will dominate the matrix. And the counts should decay with age, because a pair of files that were edited together constantly two years ago and never since is describing a structure that no longer exists; an exponential decay with a half-life of around six months tracks refactors reasonably well. The signal is at its most valuable exactly where the others are weakest, since a config schema and the route that reads it share no imports and no vocabulary, and are edited together eleven times in fourteen commits.

The cutoff is a token budget, not a file count#

"Top ten" is a convenient way to report a metric and a poor way to build a prompt, because the ten results are not the same size. A 22-line route handler and a 900-line service class both count as one, and the difference between them is 12,000 tokens. The cutoff that matters is a budget — say 30,000 tokens reserved for retrieved code out of a larger window, leaving room for the task, the tool definitions, and several turns of the agent's own file reads — and the packing problem is which candidates buy the most relevance per token.

def pack(cands, budget_tokens: int):
    ranked = sorted(cands, key=lambda c: c.score / max(c.tokens, 1), reverse=True)
    out, used, per_dir = [], 0, Counter()
    for c in ranked:
        if used + c.tokens > budget_tokens:
            continue
        if per_dir[c.directory] >= 3:
            continue
        out.append(c)
        used += c.tokens
        per_dir[c.directory] += 1
    return out

Sorting by score per token rather than by score changes what gets in. A 200-token symbol chunk scoring 0.6 displaces a 6,000-token file scoring 0.9, and usually should, because the large file's score was earned by one method inside it. That points at the more useful lever: rather than dropping a large candidate, downgrade it — include the one symbol chunk that scored, plus its header, and record the file path so the agent can read the rest if it decides to. Retrieval sets the starting position; it does not have to be the only thing the agent ever sees.

Three other economies are worth applying in order. Deduplicate to the file level first, since a file retrieved as four separate method chunks is spending four headers to say the same thing. Cap the number of candidates from any one directory, because a pool dominated by tests/ or by one package produces context that is locally dense and globally blind. And reserve a slot for an exemplar even when it scores below the cut, since a peer file that already does the thing correctly is worth more than the fourth-best document about it.

What ordering cannot fix#

A ranker's ceiling is the candidate set it was given. If expansion never generated handlers/payment_succeeded.py, no weight vector puts it first, and the whole of the previous three chapters is upstream of this one for that reason. When a retrieval system underperforms it is worth separating the two failures before touching weights: measure how often the required file appears anywhere in the candidate pool, and how often it appears in the final cut. A pool recall of 0.95 with a cut recall of 0.62 is a ranking problem. A pool recall of 0.60 is a candidate generation problem, and tuning weights against it is rearranging a list that does not contain the answer.

The useful diagnostic when tuning is an ablation rather than a sweep. Turn each signal off in turn and re-measure, and the shape of the result tells you where the system's competence lives. A signal whose removal costs nothing is either redundant with another signal or broken, and the second is more common than the first — a hop term contributes nothing when the import resolver is failing on 11% of statements, and a co-change term contributes nothing when the repository's history is a series of squashed monolithic merges. Ablations also expose the case where two signals are quietly measuring the same thing, which is what happens when the embedding is computed over a chunk that includes its file header, so that path similarity and content similarity are correlated and the weight vector is double-counting.

The state of the art past a weighted sum is a reranker that reads the candidate and the task together rather than comparing two independently computed vectors — a cross-encoder, or an LLM asked to order a list of twenty candidates. Both are materially better at judging whether a chunk is relevant to a task, and both cost something: a cross-encoder over 200 candidates adds roughly 150 to 400 milliseconds and a model to host, and a listwise LLM rerank over twenty candidates costs a few cents and a second or two per task. Neither of them recovers a candidate the graph never produced, and neither of them can tell you whether your ordering is improving, because a reranker is another opinion about relevance rather than a source of ground truth.

Everything in this chapter has a free parameter in it: four weights, a fusion constant, a boost, a budget, a per-directory cap. Setting them requires a set of tasks whose correct answer is known, and the obvious way to build one — sit two engineers down and have them write out the files that look related to each task — produces a set that will confirm whatever the ranker already does. The next chapter is about where the labels come from instead.