Fusing two retrievers without inventing comparable scores

The pull request was eleven lines and it read as obviously correct.

def fuse(image_hits, text_hits, alpha=0.5):
    def norm(hits):
        s = [h.score for h in hits]
        lo, hi = min(s), max(s)
        return {h.id: (h.score - lo) / (hi - lo) for h in hits}

    a, b = norm(image_hits), norm(text_hits)
    combined = {i: alpha * a.get(i, 0) + (1 - alpha) * b.get(i, 0) for i in a | b}
    return sorted(combined.items(), key=lambda kv: -kv[1])

Two retrievers on incompatible scales, both rescaled to [0, 1], averaged. Every part of that sentence is defensible and the function is broken in a way that a unit test with one good query will never catch.

Here is the trace for a query the system handles well and one it cannot handle at all.

query: "red enamel kettle with a chipped spout"
  image   raw 0.312 0.297 0.291 0.284 ...   -> normed 1.00 0.55 0.38 0.00
  bm25    raw 41.2  22.8  19.4  8.1   ...   -> normed 1.00 0.44 0.34 0.00
  fused top-1: IMG-88231  score 1.00   (correct)

query: "an image that conveys quarterly momentum"
  image   raw 0.194 0.191 0.190 0.188 ...   -> normed 1.00 0.50 0.33 0.00
  bm25    raw 5.1   4.9   4.4   4.0   ...   -> normed 1.00 0.55 0.36 0.00
  fused top-1: IMG-41904  score 1.00   (a photograph of a filing cabinet)

Both queries produce a top result scoring exactly 1.00, because min-max over the returned candidates maps the maximum to 1.0 by construction. The transform's parameters are the minimum and maximum of that list, for that query, so it cannot preserve any information about how good the list is — that information lives precisely in the level of the raw scores, which is the thing being normalised away. A retriever whose best hit scored 0.312 and one whose best hit scored 0.194 emerge indistinguishable, and the downstream consumer that was going to threshold at 0.8 to decide whether to show results at all now shows the filing cabinet with full confidence.

Everything the per-query transform destroys#

The floor is as corrupted as the ceiling. The worst candidate in each list is mapped to exactly 0.0, so the tenth image is scored zero whether it is a near-duplicate of the first or entirely unrelated. Fusing on those numbers, an item that appears at rank 10 in one list and rank 3 in the other gets 0.5 × 0 + 0.5 × 0.34, and is beaten by an item that appears only in the first list at rank 2 — the appearance in two lists, which is the strongest signal fusion has to work with, is thrown away by the encoding.

The transform is also a function of k. Fetch 10 candidates and the tenth defines the zero point; fetch 100 and the hundredth does, which compresses the top ten into a narrow band near 1.0 and changes every fused ordering. A parameter that everyone treats as a performance knob silently rewrites the scoring function.

Then there are the candidate populations, which are not the same set of things. The image retriever returns k results for any query, always, because there is always a nearest neighbour — that is what a dense index does, and its scores are a narrow band by nature, 0.18 to 0.31 on this corpus, because chapter 1 showed the cross-modal band is narrow. BM25 does not: it returns only documents containing at least one query term, so a query with no lexical overlap returns an empty list, and max(s) on an empty list raises, while a single-result list makes hi - lo zero and raises again. Both edge cases fire on real traffic — the first on any query in a language the descriptions are not written in, the second on rare-term queries — and both are one-line crashes in the fusion layer of a search service.

BM25's scale itself is not stable across queries in the way the code assumes. Its value is a sum over matched terms, so a three-word query scores roughly three times a one-word query, and a term with high inverse document frequency contributes far more than a common one. A query containing a rare SKU can score 60; a three-word descriptive query tops out near 12. Those are not comparable to each other, let alone to a cosine.

The usual next suggestion — tune a weighting factor α on a held-out set and use α·image + (1−α)·text — does not address any of this. Fitting α on 900 judged queries gave a best value of 0.45 and lifted nDCG@10 from 0.31 to 0.34. It cannot do better, because α is a single global constant applied after the per-query transform has already discarded the level; it corrects the average balance between the two retrievers and has no way to reach back and recover what each list's raw scores said about its own quality. On the momentum query above, α = 0.45 produces a fused top score of 0.55 for the filing cabinet, which sits in the middle of the distribution of scores for correct answers on good queries. No value of α makes a list say "I have nothing".

Before any fusion strategy runs, the two lists have to be talking about the same objects. The image index is keyed by asset id and the description index by product id, and a single product with four photographs appears once in the text list and up to four times in the image list — so a naive union double-counts it, and any strategy that rewards appearing in both lists rewards it four times over. Deduplicate to a canonical entity id before fusing, keeping the best-ranked member of each group, and decide explicitly whether the result set is products or images, because those are different surfaces with different ideal behaviour. The candidate depth is worth setting per retriever rather than globally as well: the image retriever always returns exactly k and its marginal candidates are cheap, while the lexical retriever often returns fewer than k and its marginal candidates are the ones that matched a single common term.

Three fusions that survive a bad query#

Reciprocal rank fusion discards the scores entirely and combines positions: each item scores Σ 1/(k + rank) over the lists it appears in, with k conventionally 60. It is two lines, needs no training data, is immune to scale, cannot crash on an empty list, and rewards appearing in both lists — which is the signal the min-max version destroyed.

def rrf(*lists, k: int = 60, top_n: int = 20):
    scores: dict[str, float] = {}
    for hits in lists:
        for rank, hit in enumerate(hits, start=1):
            scores[hit.id] = scores.get(hit.id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])[:top_n]

What it gives up is margin. A first-place hit at cosine 0.31 and a first-place hit at 0.194 both contribute 1/61, so RRF is exactly as confident about the filing cabinet as about the kettle — it produces a defensible ordering on every query and still cannot abstain. It also flattens genuine gaps: when the top image is a near-exact match and the second is unrelated, RRF treats one rank of separation as one rank of separation. On the appliance evaluation it took nDCG@10 from 0.31 to 0.52, which is most of the available win for an hour of work.

The constant k is the one knob, and it controls how sharply the top of each list is favoured: at k = 60 the gap between rank 1 and rank 2 is 1/61 − 1/62 ≈ 0.00026, so a document at rank 3 in both lists beats one at rank 1 in a single list, while at k = 10 the first rank is worth enough that a single strong list can carry a document alone. Sixty is the published default and is deliberately conservative — it assumes neither retriever's top result deserves much more trust than its third. If one retriever is clearly stronger on the traffic you have, a per-list weight is the honest expression of that (w_i / (k + rank)), fitted on the same held-out queries you would otherwise use for α, with the difference that here the weight is modifying a bounded, well-behaved quantity rather than papering over a broken one.

Calibration is what buys back the level. Fit a monotone map — one that never assigns a lower output to a higher raw score, so the retriever's own ordering is preserved — from each retriever's raw score to the probability that a result at that score is relevant, using held-out judgements, and both retrievers then emit numbers on the one scale that is genuinely shared — probability of relevance.

from sklearn.isotonic import IsotonicRegression

# raw scores and binary relevance labels from the pooled judged set
img_cal  = IsotonicRegression(out_of_bounds="clip").fit(img_scores,  img_labels)
text_cal = IsotonicRegression(out_of_bounds="clip").fit(text_scores, text_labels)

def fuse_calibrated(image_hits, text_hits, floor=0.25):
    p = {}
    for h in image_hits: p[h.id] = max(p.get(h.id, 0), img_cal.predict([h.score])[0])
    for h in text_hits:  p[h.id] = max(p.get(h.id, 0), text_cal.predict([h.score])[0])
    kept = {i: s for i, s in p.items() if s >= floor}      # abstain instead of guessing
    return sorted(kept.items(), key=lambda kv: -kv[1])

The isotonic fit on this corpus mapped image cosine 0.31 to P = 0.78 and 0.194 to P = 0.06, so the momentum query returns an empty list and the interface can say so. That is a capability neither of the other two options has. The price is roughly 1,200 judged query-document pairs to fit both curves — about a day and a half of annotator time — and a commitment to refit when the query mix shifts, because the calibration is a property of the traffic as much as of the retriever. Calibrated fusion reached nDCG@10 of 0.57.

Staged retrieve-then-rerank sidesteps the comparison instead of solving it: use each retriever only to nominate candidates, then score every candidate with one model, so the final ordering comes from a single scale by construction.

The reranker sees the query and each candidate together, which is what lets it answer the intent-shaped questions from chapter 4 that no precomputed vector can. Over 45 candidates a text cross-encoder over the descriptions added 40 ms at p95 and cost nothing per query; a VLM scoring the images directly added 340 ms at p95 and about $0.004 per query, and reached nDCG@10 of 0.61 — the best of the four, and the only one that improved the intent-query class at all, from 0.12 to 0.39.

What fusion does not fix#

None of these recover a document that neither retriever nominated. Everything above operates on the union of two candidate lists, so the ceiling is the recall of that union, and a query where both retrievers miss is a query no fusion strategy touches. Measuring the union's recall separately from the fused nDCG is what tells you whether to spend the next week on ranking or on retrieval.

RRF is a floor rather than an answer. It is the right first implementation because it is unbreakable and untuned, and the right thing to replace once you have judgements, because on this corpus it left 0.09 nDCG on the table relative to calibration for the sake of avoiding a day of annotation.

Calibration does not transfer. The isotonic curve fitted on shopping traffic in one category is wrong for another, and it drifts as the corpus grows — the same cosine means something different when the index holds four million images instead of four hundred thousand, because the density of near neighbours changed. Refit on a schedule and monitor the fraction of queries that abstain; a sudden move in that fraction is the first signal that something upstream changed.

Whichever strategy ships, the fusion layer needs two metrics of its own in production, and neither is a quality metric. The first is the abstain rate, or for rank-based fusion the share of queries whose best raw score falls below where good queries usually sit — a number that should be flat and whose movement is the earliest available signal that the corpus, the model or the traffic changed. The second is attribution: for each result that reaches the final top ten, record which retriever nominated it. A steady 70/30 split that drifts to 95/5 over a month means one retriever has quietly stopped contributing, which shows up in this metric weeks before it shows up in aggregate relevance, and which is exactly the question the next chapter has to answer with numbers.

And the reranker adds latency and per-query cost to a system whose whole appeal was that the expensive work happened offline. Four hundred milliseconds and $0.004 are cheap against a bad result on a product search surface and expensive against an autocomplete dropdown, which makes it a per-surface decision rather than a global one.

At this point the machinery is complete: two retrievers on incomparable scales, a fusion that behaves on good and bad queries alike, and an evaluation honest enough to trust. What remains is the question that has been deferred since the first cost table — whether the cross-modal half of that system earns its place at all, given that the caption baseline it is being combined with already scored higher on its own.