Weighted score fusion buys magnitude and pays in calibration

Weighted score fusion combines the retrievers' scores rather than their positions, multiplying each arm's score by a weight and adding. It exists because the previous chapter's rule throws away magnitude, and magnitude sometimes carries the whole answer. This chapter is about the query class where it does, what the weighted rule recovers there, and the maintenance cost that arrives several months later with no error attached.

The cost is worth naming up front because it is the substantive difference between the two families. A rank is a fact about an ordering and survives anything that preserves the ordering. A score is a number produced by a function whose inputs include the collection's own statistics, so a weight fitted to today's scores is fitted to today's collection.

The margin RRF discards#

A controls engineer types E-4417 into the portal of a robot-documentation system. That fault code appears in exactly four chunks out of 1.4 million: the fault-code reference entry that explains it, a firmware release note that fixed it, and two support tickets. BM25 returns those four and nothing else, with the reference entry at 31.4 and the next document at 6.8 — a margin that exists because the code is essentially a document identifier and the second-place document merely mentions it in passing.

The dense arm returns fifty candidates and the reference entry is not among them. E-4417 tokenizes into subwords shared with E-4407, E-4418 and every other code in the series, so the nearest neighbours are the fault-code index page that lists 300 codes, then the E-44xx family overview, then a troubleshooting walkthrough, with cosines of 0.812, 0.809 and 0.806. Those three numbers are within 0.006 of each other, which on this model is indistinguishable from an arbitrary ordering.

Fuse the two lists with RRF at k=60 and the result is wrong in a way that is entirely predictable from the last chapter. The fault-code index page sits at BM25 rank 3 and dense rank 1, giving it 1/63 + 1/61 = 0.03227. The reference entry sits at BM25 rank 1 and nowhere in the dense list, giving it 1/61 = 0.01639. The generic index page beats the exact answer by roughly two to one, because two lists agreed on it and only one list — emphatically, with a 4.6× score margin — nominated the other.

No value of k rescues this, and the reason needs no arithmetic at all. Write the two scores as expressions in k. The reference entry appears in one list, at rank 1, so it scores one divided by k plus one. The index page appears in that same first list at rank 3 and in the second list at rank 1, so it scores one divided by k plus three, plus one divided by k plus one. The second expression contains the first as a summand and adds a strictly positive term to it. One score is therefore larger than the other for every k, and no choice of the constant can reorder them. At k=2, to make it concrete, the reference entry scores 0.3333 and the index page scores that same 0.3333 plus a further fifth on top.

That is the general shape of the problem rather than a quirk of this query. Whenever one arm's list is a superset of what the other arm found — here the dense arm found the index page and the lexical arm found it too, lower down — rank fusion has no mechanism for saying that the lexical arm's own first place was worth more than everything else on the page. Turning k down does not help either, because a small k has a separate cost: at k=0 every dense rank-1 candidate scores a full 1.0000, so on the paraphrased queries where the dense arm carries the whole result the fusion becomes hostage to whichever document happened to land first in a list whose top five differ by 0.02 of cosine. The parameter that would have to move to fix this query is the parameter that breaks the majority class, and it would not fix this query anyway.

Keeping the margin#

Weighted score fusion combines the retrievers' scores rather than their positions:

def weighted_fuse(
    lexical: dict[str, float],      # doc_id -> normalised lexical score
    dense: dict[str, float],        # doc_id -> normalised dense score
    alpha: float = 0.5,             # 0.0 = pure lexical, 1.0 = pure dense
) -> list[tuple[str, float]]:
    ids = lexical.keys() | dense.keys()
    fused = {
        d: (1 - alpha) * lexical.get(d, 0.0) + alpha * dense.get(d, 0.0)
        for d in ids
    }
    return sorted(fused.items(), key=lambda kv: -kv[1])

The alpha here is the same knob Weaviate exposes on its hybrid queries, where alpha=0 is pure keyword and alpha=1 is pure vector; Elasticsearch, Vespa and Qdrant all offer a per-retriever weight in some form. The word normalised in the type hints is where all the difficulty hides, and the next chapter is about doing it properly. For now, take min-max over each retriever's candidate list for this query, so each arm's best document becomes 1.0 and its worst becomes 0.0.

Work that normalisation on the E-4417 lexical list, because the numbers in the table below are not arbitrary. Min-max maps a score by subtracting the list's minimum and dividing by the list's span. The lexical list runs from 31.4 at the top to 2.1 at the bottom, so the span is a little over twenty-nine. The reference entry is the maximum, so it maps to exactly 1.00 by construction. The index page at 6.8 sits about four and a half points above the bottom, which against a span of twenty-nine is 0.160. The index page takes 1.00 on the dense side and the reference entry takes 0.00, since it is absent.

Both α columns follow from the code above, and reading them as a pair shows what the weight is buying. At α = 0.5 each arm contributes half its normalised score, so the reference entry gets half of 1.000 and the index page gets half of 0.160 plus half of 1.000 — the index page still wins, narrowly. At α = 0.3 the lexical arm carries seven tenths and the dense arm three: the reference entry gets seven tenths of 1.000, which is 0.700, and the index page gets seven tenths of 0.160 plus three tenths of 1.000, which is 0.412. The reference entry wins by a comfortable margin.

The important part is that a weight can express the outcome you want here, because the lexical arm's certainty survived into the fused score as 1.000 against 0.160 rather than being flattened into "first" and "third". Rank fusion had no representation for the fact that 31.4 versus 6.8 is a different kind of first place from 0.812 versus 0.809.

That is the honest answer to what RRF throws away. It discards magnitude, and magnitude carries information whenever one retriever's score distribution for a given query is bimodal — a few documents that genuinely contain the thing, then a long tail that does not. Identifier and rare-term queries are exactly that shape on the lexical side, which is why they are where the loss shows up.

The bill arrives as drift, not as an error#

Weighted fusion has a fitted parameter, and the validity of that parameter depends on two score distributions being roughly where they were when it was fitted. Neither distribution is owned by whoever fitted it.

Trace the dependency one link at a time. The weight was chosen so that a typical lexical score and a typical dense score would trade off sensibly after normalisation. Normalisation maps a raw score into a range using the scores around it. Those raw scores come out of functions whose inputs include the collection's term statistics on the lexical side and the model's geometry on the dense side. Both of those change without anybody touching the fusion code. Therefore the weight goes stale as a consequence of ordinary work elsewhere, and the staleness is invisible, because a weight is a valid number whatever the distributions are doing.

Three ordinary changes are enough to move both distributions at once:

Each of those is a normal, well-intentioned change, shipped by a different person, none of whom was thinking about fusion weights. The first two pull in opposite directions on the lexical side, so a monitor watching only the mean would see them partially cancel while the identifier class moved sharply. The third changes what a cosine of 0.80 means, which under a global normalisation changes what fraction of the dense range a document at 0.80 occupies — and note what the last cell says about the spread, which nearly doubled. A wider spread means the same rank ordering now maps onto a wider band of normalised scores, so the dense arm's contribution to every fused score changes even where its ranking did not.

This is the substantive difference between the two fusion families, and it is worth stating plainly because it is the thing the interview question is aiming at. RRF has no state to go stale. Ranks are ranks whatever the collection does; a document that is first is first after a reindex, after an analyser change, after a model swap. Weighted fusion has a fitted parameter whose validity depends on two distributions that are maintained by other teams on other schedules, and the failure mode is not a crash but a slow degradation confined to a query class nobody is looking at.

The operational answer is to monitor the inputs rather than the output, because the output is a ranking and a ranking without judgements cannot be scored. A weekly job that replays a fixed sample of production queries and records the per-arm score distributions gives you something to alarm on:

def score_profile(queries: list[str]) -> dict[str, float]:
    """Replay a fixed query sample and summarise each arm's score range."""
    lex_top1, den_top1, lex_spread = [], [], []
    for q in queries:
        lex, den = bm25(q, k=50), dense(q, k=50)
        lex_top1.append(lex[0].score)
        den_top1.append(den[0].score)
        lex_spread.append(lex[0].score - lex[-1].score)
    return {
        "lex_top1_mean": mean(lex_top1),
        "lex_top1_p10": quantile(lex_top1, 0.10),
        "den_top1_mean": mean(den_top1),
        "lex_spread_mean": mean(lex_spread),
    }

Alarm when lex_top1_mean or den_top1_mean moves more than about 10% from the value recorded when the weight was last fitted. That threshold is a judgement call rather than a derived number, but any threshold beats the current default of finding out from a click-through chart two quarters later. The lex_top1_p10 is in there for the reason the drift table shows: a mean can sit still while two changes cancel, and a low percentile moves when the hard queries move.

Three rules against the same criteria#

Relative-score fusion is Weaviate's name for the middle column-and-a-half: normalise each arm's candidate scores into [0, 1] per query, then take the weighted sum. Their own comparison of the two fusion algorithms reports it winning on queries where one arm is decisively better, which matches the E-4417 case, and losing where one arm's list is uniformly weak, because min-max hands that list a 1.0 anyway.

That last row is the reason normalising is not the same thing as calibrating, and the two are worth keeping apart. Normalising rescales a list against itself, so it always produces a 1.0 whether the list is excellent or worthless. Calibrating would map a score to something with a fixed external meaning, which is the only thing that could tell the two lists apart. It is the trap chapter 4 opens on.

What calibration would actually mean#

The word calibration gets used loosely in this area, and being precise about it explains why the maintenance burden is unavoidable rather than a symptom of a sloppy implementation.

A score is calibrated when its value has a fixed external meaning — most usefully, when a score of 0.8 means the document is relevant about 80% of the time. Neither retriever produces anything like that. A BM25 score of 24.7 means "this document accumulated 24.7 units of term evidence given the current corpus statistics", which is a statement about the collection as much as about the document, and the same document with the same text will score differently next month. A cosine of 0.847 means "this vector is 0.847 of the way toward parallel with the query vector under this model", which is a statement about the embedding geometry and has no probabilistic reading at all.

Making them mean the same thing is possible in principle. You take a judged set, fit a mapping from raw score to probability of relevance for each retriever — logistic regression on the score, in the Platt scaling style, or isotonic regression if you have enough judgements to support a monotone step function — and then the fused score becomes an average of two probabilities, which is a quantity that genuinely combines. That is real calibration, and it is what would let you set a threshold below which the system returns nothing.

Almost nobody does it, for three reasons that are worth knowing before you decide to be the exception. It needs judged data per collection, since a mapping fitted on one set of documents says nothing about another. It needs refitting on the same cadence as the drift above, so it converts one stale parameter into a stale model with more parameters. And the payoff over per-query min-max normalisation is small, because ranking only needs the fused order to be right and calibration is buying an interpretation of the magnitudes that ranking does not consume. The one case where it pays is when you need an abstention rule — a threshold below which you tell the user there is no answer rather than showing the least bad document — and that is a real product requirement that fusion scores cannot serve. If you have it, a calibrated reranker score is the usual answer, and it comes from the same model that produces the ranking.

Weighted fusion is also not learning to rank. A single scalar per retriever is a very coarse model — it cannot say "trust lexical on short queries containing digits and dense on questions", it cannot condition on the query at all, and it has no features beyond the two scores. If you find yourself wanting a second weight for short queries and a third for queries with an uppercase token, you have started building a ranking model with two features and no training loop, and you should either build the model properly or route the query instead of blending it.

What the last two chapters have established is a choice between a rule that needs nothing and a rule that needs a normalisation and a weight. The first is immune to everything the second is exposed to, and it is deaf to the one query class the second hears.

Both of those descriptions are still hand-waving in the same place: min-max over what, exactly, and fitted on which queries. The next chapter makes them concrete operations, sweeps the weight on a held-out split, and shows how quickly a swept optimum turns out to be fitting the judgement noise in a small evaluation set rather than expressing anything about your documents.