Comparable scores are made per query, and the weight is fitted rather than chosen

Normalisation here means mapping each retriever's raw scores onto a common range so that a weighted sum of them is meaningful. The previous chapter left it as a word in a type hint. It is an operation with choices in it, and the choices decide whether the weighted fusion works at all — a badly scoped normalisation can silence an arm on exactly the queries it was there to answer, without producing any error.

The scoping question is the first one, and it has a clean answer: the operation runs per query, on the candidate list that query produced. Everything below is either the argument for that or the consequences of getting the weight wrong once the scoping is right.

What a range fitted over a corpus does#

The first production implementation of the weighted fusion from the last chapter shipped with four constants baked into a config file, sampled from a thousand queries during the build:

normalisation:
  lexical: { min: 0.0,  max: 38.4 }
  dense:   { min: 0.61, max: 0.94 }
fusion:
  alpha: 0.45

Those four numbers describe a distribution measured across many queries. The scores they will be applied to are produced one query at a time, and the failure follows from that mismatch in two directions.

Take the query torque, a single mid-frequency term appearing in 210,000 chunks of a 1.4-million-chunk robot-documentation collection. The best BM25 score in the whole candidate list is 4.1. Against a range whose maximum is 38.4, that normalises to 0.107 — a little over a tenth of the way up. At α = 0.45 the lexical arm therefore contributes fifty-five hundredths of 0.107, which is 0.059, while the dense arm contributes forty-five hundredths of its own normalised 0.667, which is 0.300. Five times as much. The lexical arm has been switched off.

The step worth pausing on is why 4.1 was small. It was not small because the lexical arm was unsure; on this query its rank-1 document is the correct one and the dense arm's is not. BM25 scores are small in absolute terms whenever the query is short and its terms are common, because the score is a sum over terms weighted by rarity. So the number the global normalisation read as low confidence was in fact a statement about the query's shape. A normalisation that cannot tell those two apart will silence an arm for having been asked an easy-vocabulary question.

The same config produces the opposite pathology on Kirchgessner, a rare surname on three internal escalation notes, where the best BM25 score is 41.6. That normalises to 1.083, above the top of the range, so it gets clipped to 1.0 — and so does the second document at 39.8. The two become indistinguishable at exactly the point where the distinction mattered, which is the sharpest way to see the cost: the clip destroys margin, and margin is the only thing weighted fusion was adopted to preserve. Then the corpus doubled and every one of the four constants was fitted to a distribution that no longer existed.

Normalise inside the query, not across the corpus#

The operation that makes two scores comparable has to run per query, on the candidate list that query produced, because the thing being corrected for is not a fixed property of the retriever but the difficulty of this particular query.

from statistics import mean, pstdev

def minmax(scores: list[float]) -> list[float]:
    lo, hi = min(scores), max(scores)
    span = hi - lo
    if span < 1e-9:
        return [1.0] * len(scores)      # every candidate tied
    return [(s - lo) / span for s in scores]

def zscore(scores: list[float], std_floor: float = 1e-3) -> list[float]:
    mu = mean(scores)
    sd = max(pstdev(scores), std_floor)
    return [(s - mu) / sd for s in scores]

The two behave differently in ways that decide which one you want.

Min-max is bounded, which makes the weighted sum easy to reason about and easy to threshold downstream, and it maps the best candidate to exactly 1.0 in every list on every query — that follows straight from the formula, since the maximum minus the minimum divided by the span is the span over itself. That last property is the cost: a list of fifty irrelevant documents and a list containing the perfect answer both present a 1.0 to the fusion. The information the last chapter went to weighted fusion to recover — how far ahead was the leader — survives inside the list, in the gap between the 1.0 and the second value, but the leader's own confidence relative to other queries is gone. Applied to the torque query, min-max over the candidate list maps BM25's 4.1 to 1.0, which is the correct behaviour for that query and would have prevented the failure above.

Z-scoring keeps shape rather than range. It subtracts the list's mean and divides by the list's standard deviation, so the output says how many standard deviations from typical each candidate is. A list whose top document is three standard deviations clear of the rest produces a 3.0, and a list whose candidates are all within a whisker of each other produces values near zero across the board, which is a faithful representation of "this arm has no opinion".

The risk is the opposite one, and it comes from the divisor. A list with genuinely tiny spread, such as fifty dense candidates spanning 0.812 to 0.806, has a standard deviation around 0.0018. Dividing a difference of a few thousandths by a number that small produces values of order one, so a spread that is numerically noise arrives at the fusion looking like a confident ranking. The std_floor above is the crude guard; a better one is to fall back to min-max when the raw spread of the list is below a threshold you set from the retriever's own resolution.

For a two-arm hybrid where one arm is BM25, z-scoring the lexical side and min-maxing the dense side is a defensible mixture, because the two arms have different pathologies: the lexical side has real spread worth preserving and the dense side has spread small enough to be manufactured. It is also a third and fourth thing to maintain, which is the running theme of this half of the topic.

The candidate depth is part of the normalisation#

A detail that gets missed: min-max depends on the minimum, and the minimum depends on how deep you retrieved. Take the gearbox lubricant change interval HX-2140R query and the document tic/48821#grease-change with a raw BM25 score of 14.9.

Work the first row to see where the movement comes from. Subtracting the minimum gives six points of score above the floor, and the span between 8.9 and 24.7 is fifteen and a bit, so the ratio is 0.380. The maximum never changes down the column, because retrieving deeper adds documents at the bottom. What changes is the minimum, which drops as the list lengthens, which stretches the span, which pushes any fixed raw score toward the bottom of the range — except that here it does the opposite, because the numerator grows faster than the denominator when the document sits well above the floor.

Same query, same document, same raw score, three different inputs to the fusion, spanning 0.18. Compare that against what the weight can do: moving α from 0.45 to 0.65 changes the lexical multiplier by two tenths, which applied to a normalised score of around 0.4 changes that document's fused contribution by under eight hundredths — less than half the swing the depth produced. The knob nobody thinks of as a fusion parameter moves the fusion more than the fusion parameter does. If the candidate depth is a tunable in your config — and it usually is, because someone will raise it to improve recall — then the weight you fitted at depth 50 is not the weight for depth 100, and nothing in the system says so. Pin the depth alongside the weight and treat changing it as a reason to refit.

Sweeping a weight on 412 queries#

With the scales aligned per query, α becomes a one-dimensional fit, and a one-dimensional fit invites a sweep. Split the 412 judged queries into 280 for fitting and 132 held out — the two parts add back to 412, so every judged query is used exactly once and no query contributes to both the fit and its check — evaluate nDCG@10 at each α, and take the argmax. nDCG@10 is normalised discounted cumulative gain at depth ten, which credits a relevant document more the nearer the top of the list it sits and then divides by the score of the best possible ordering, so that 1.0 is perfect and the number is comparable across queries with different numbers of correct answers.

alphas = [i / 20 for i in range(21)]
dev = [q for q in judged if q.split == "dev"]

results = {a: ndcg_at_10(run(dev, alpha=a)) for a in alphas}
best = max(results, key=results.get)

The curve is unimodal and the optimum is at α = 0.6, which looks like a result. Three things say it is not.

The first is the width of the plateau. Everything between α = 0.4 and α = 0.7 spans 0.013 of nDCG, which is the gap between the 0.664 at the peak and the 0.651 at the low end of that stretch. A paired bootstrap over the 280 dev queries — resample queries with replacement, recompute the difference, take the 2.5th and 97.5th percentiles — puts the gap between α = 0.6 and α = 0.4 at +0.013 with a 95% interval of [−0.008, +0.034]. The interval contains zero, which means the sweep has not established that 0.6 is better than 0.4 on this data.

The second is arithmetic you can do before running anything, and it is worth walking because the same calculation settles several later questions. For two systems as similar as α = 0.4 and α = 0.6, the per-query differences in nDCG@10 have a standard deviation around 0.15 on a collection like this one, most queries producing exactly zero difference and a minority swinging hard. The smallest difference a paired comparison can detect at 80% power — meaning that a difference of that size really present in the system would show up as significant in four runs out of five — is roughly 2.8 × σ_d / √n. The square root of 412 is just over twenty, so with σ_d = 0.15 that comes to 0.021. The entire range of the plateau is smaller than the smallest effect the evaluation set can resolve, so every choice inside it is a coin flip dressed as a decision.

The third is the stumble, and it is about what the curve's appearance is evidence of. The sweep is persuasive largely because the curve looks smooth and unimodal, which reads as signal. That smoothness is guaranteed by the design: every point is computed on the same 280 queries, so adjacent values of α share almost all of their per-query outcomes and the curve cannot jump. It would look this smooth on a set of random judgements. A property that holds whether or not the peak is real cannot be used to decide whether the peak is real, so the shape carries no information at all about whether the optimum would survive different queries. The only things that speak to that are the spread of the resampled argmax and the held-out score.

Confirm it on the held-out split and the coin lands the other way: on the 132 held-out queries, α = 0.6 scores 0.651 and α = 0.4 scores 0.655. The swept optimum did not transfer, which is the expected outcome when the thing being optimised is smaller than the noise.

How few judged queries does it take for this to happen? Refitting α on 200 bootstrap resamples of the dev set gives an argmax that lands somewhere in [0.35, 0.85] in 90% of the resamples at n = 280, and somewhere in [0.15, 1.00] at n = 100. A hundred judged queries produces a confident-looking curve with a clear peak whose location is essentially random.

The same formula says what it would take to do better, and the answer is discouraging in a useful way. The detectable effect falls with the square root of n, so halving it costs four times the queries. Going from the current 0.021 down to 0.01 is a little more than halving, so it needs a little more than four times 412 — around 1,800 paired judged queries, which is a different order of investment from the two afternoons most teams spend on a judged set.

That does not make the sweep worthless. It makes its output a range rather than a point. The defensible reading of the table is that α anywhere between 0.4 and 0.7 is fine, that α below 0.2 or above 0.9 is measurably worse, and that the correct action is to set α = 0.5, write down the depth and the normalisation it was fitted with, and stop. Reporting "we tuned α to 0.6" implies a precision the data does not support, and it commits you to maintaining a number that chapter 3 showed goes stale silently.

Better instruments for the same signal#

Step back from the mechanics and the honest conclusion is uncomfortable for the whole weighted-fusion project. The signal being chased is one retriever's certainty on a particular query. Reconciling two score distributions is a very indirect way to use that signal, and there are two direct ways that generally work better on the same documents.

The first is to stop combining scores and score the pair. A cross-encoder reranker reads the query and a candidate document together and emits a relevance score that is on one scale by construction, because there is only one model. The comparability problem does not get solved; it stops existing. On this collection the cross-encoder over the top 100 fused candidates moves nDCG@10 from 0.652 to 0.731, an improvement of 0.079 — comfortably outside the resolution limit calculated above. It costs about 84 ms of p95 latency — the latency 95 queries in 100 come in under, which is the number worth quoting because a mean hides the queries that hurt — and a GPU, which is a real bill, but it is a bill you can price against a number you can actually measure.

The second is to route rather than blend. If the reason you want α = 0.3 is that identifier queries need the lexical arm to dominate, then the honest implementation is a rule that recognises identifier queries and sends them to the lexical arm, rather than a global weight that leans slightly lexical on every query in the system including the 172 paraphrased ones that wanted the opposite. A regex over the query for a token matching [A-Z]{1,3}-?\d{3,}[A-Z]? catches most of that class on this collection at high precision, and routing those queries to lexical-only takes their recall@10 from 0.83 under fusion to 0.93, without touching anything else. Whether that trade is available to you is a question the per-class evaluation in chapter 7 answers.

Both of those change the shape of the system rather than the value of a constant, and the resolution arithmetic is why that matters: a change of 0.079 is measurable on 412 queries and a change of 0.013 is not, so shape changes are the only ones this evaluation set can adjudicate at all.

There is a third way to change the shape, which is to change what the retrievers are. The next chapter looks at a retriever that produces a sparse vector over the vocabulary, served by an ordinary inverted index, whose weights come from a model instead of from term counts — and which therefore sits between the two arms you have been trying to reconcile.