Min-max fusion is decided by your weights, not by the evidence

A hybrid retriever runs several searches over the same material and merges what comes back. A question about when a monthly churn rate first crossed 5% might go to three of them at once: BM25, a lexical scorer that adds up term weights and returns an unbounded number; a dense text encoder, which compares embeddings and returns a bounded similarity; and a page-image index, which compares the query against a picture of each page and returns a similarity of its own. Each hands back its own ranking with its own scale — 14.2 at the top of one, 0.81 at the top of another, 0.62 at the top of the third.

Those three numbers are not comparable, and that is a property of the instruments rather than a defect in any of them: a lexical score of 14.2 and a cosine of 0.81 are measurements in different units of different things, and no setting makes one larger than the other in a way that means anything. Something has to reconcile them before a single ranked list can exist, and whatever that something is, it is a decision somebody made. The commonest decision is to min-max normalise each list to 0–1 and take a weighted sum, here with weights 0.4, 0.4, 0.2.

The winner is a glossary page defining how churn is calculated. It contains no time series, no dates, and nothing that could answer the question. It wins by a factor of four over the page image that does, and in this illustration the lists are three items long — with realistic pools of a hundred candidates each, the image sits below dozens of text chunks rather than three.

Three separate things went wrong in that arithmetic, and they are worth separating because they have different fixes.

The top of every list is 1.0 by construction#

Min-max normalisation maps a list's own maximum to 1.0 and its own minimum to 0.0. Writing the transform out makes it clear why that happens and why nothing about the documents can prevent it. Call the largest raw score in a list its maximum and the smallest its minimum, and call the difference between them the list's spread. Each raw score is sent to the difference between that score and the minimum, divided by the spread.

Two steps make up that map, and each is worth pausing on because each is order-preserving, which is exactly what makes the result look harmless. Subtracting the minimum from every score in the list is allowed to leave the ranking alone, because taking one constant away from every member of a list leaves every difference between members untouched; no candidate can overtake another. After that step the smallest score is zero. Dividing every shifted score by the spread is allowed for the same reason, since dividing every member by one positive constant preserves order too. The spread is positive because the maximum is at least the minimum by definition — and the map is undefined when the two are equal, which is a real case rather than a pedantic one: a list whose scores are all identical has no spread, and the transform divides by zero. A shift followed by a stretch is an affine rescaling, and this one is computed per list, from that list's own two extremes. What it deliberately destroys is the only information that could distinguish a list containing the answer from a list that does not: the absolute magnitude of its scores.

Now evaluate the map at the top of the list. The score there is the maximum, so the numerator — the score minus the minimum — is the maximum minus the minimum, which is the spread, which is the denominator. Numerator and denominator are the same quantity and they cancel exactly. The result is one, whatever the maximum and minimum happened to be. The same substitution at the bottom puts the minimum in the numerator against itself, giving zero.

That exact cancellation is the whole problem stated in one line. The top entry of a list scores 1.0 by an identity in the transform. No observation of any retriever enters that value. Whether the image retriever returned a page at 0.62 because that page is a strong match, or at 0.62 because the corpus contains nothing relevant and 0.62 is the best of a bad set, the top entry is 1.0 either way. There is no residual here — the cancellation is complete — and the completeness is the damage, since a lossy step that left something behind would leave something to read. After normalisation, being first in a list is worth exactly the weight of that list. The fused ordering is a statement about your weights.

The mistake to name is not an arithmetic slip. It is a quantity that stays fixed while the property you care about ranges over everything it can be. The dangerous property is whether a list contains the answer at all, and it varies from "this list holds the sufficient page" to "this list holds nothing relevant". The normalised top score does not move across that range. Anything downstream reading a normalised score as confidence is reading the definition of the transform back to itself.

There is a picture that helps, offered as an analogy and no more: min-max is marking on a curve, with every class's papers rescaled so that the best in each class takes full marks and the worst takes none. It breaks in a place worth naming. With exam marks the classes are at least sitting the same subject, and the size of a class is not a free parameter. Here the three lists are scoring different representations, and the number of papers collected is a configuration setting, which means the marks depend on a choice the examiner made about how many scripts to pick up.

That last consequence is the second thing that went wrong, and it is worse and less noticed. Take any two candidates in one list. After normalisation, the gap between them is the gap between their raw scores divided by the spread of the list they are in. The numerator is what the retriever actually said about those two candidates. The denominator is a property of the rest of the list, and in particular of its last member. So a list whose scores are packed tightly together has its differences stretched out to fill the whole zero-to-one range, and a list whose scores are widely separated has its differences compressed into that same range, and after the transform the two look alike. Two candidates' relative standing has come to depend on a third candidate that is neither of them.

Page 14's OCR text was genuinely retrieved by BM25 — it was in the list — and because it happened to be last in the truncated list, min-max assigned it 0.000. A candidate's fused score depends on where the list was cut off, which is a parameter of the retrieval configuration rather than a property of the evidence.

The table demonstrates that rather than merely asserting it. Look at the second page image, deck_q2_2024#p14, which normalises to 0.714. If the image list's smallest score were the 0.60 shown against it, that candidate would be the list minimum and would have to normalise to zero, by the substitution done two paragraphs above. It does not. So the image list holds a smaller score than any the table displays, belonging to a candidate that never reaches the merged pool at all — and given the maximum, one other raw score and its normalised value, that minimum is determined. A displayed row's fused score is fixed in part by a row nobody can see, which is the truncation dependency written into the chapter's own figures. The same check on the dense list gives the opposite answer, since its lowest displayed entry does normalise to zero, so that list was shown down to its floor.

Then there is the summing. Two of the three retrievers read the same extracted text of the same corpus. When BM25 and a dense text encoder agree on a chunk, that is not two independent pieces of evidence; it is one representation being scored two ways, and the correlation between them is high on exactly the queries where lexical and semantic signals point the same direction. A weighted sum treats the agreement as corroboration and awards 0.8 to a candidate that one modality endorses. The image index has no correlated partner, so a candidate it is certain about can score at most 0.2. Adding a second image retriever would, absurdly, help — not by improving evidence, but by giving the visual modality a vote to double.

Both of those figures fall straight out of the weights. A candidate's fused score is the sum, over the lists it appears in, of that list's weight times its normalised value there; a candidate absent from a list contributes nothing, which is itself a choice and a poor one, since "absent from this list" and "worst in this list" then score identically. The glossary chunk tops both text lists, so both of its normalised values are one and its fused score is the two text weights added together. The page image tops the image list, so its fused score is the image weight. Every row in the Fused column reconstructs this way from the columns beside it with no remainder, which tells you what kind of object that column is: a formula evaluated, not an observation recorded.

The factor of four between the glossary and the page image is then a ratio of weights and nothing else. Both candidates were rescaled to one, so both of their raw scores have cancelled out of the comparison entirely; what is left is the two text weights over the image weight. That number would be identical for any other pair of candidates that happened to top their respective lists, whatever the pair was about. It carries no information about churn, about the deck, or about the glossary. The residual worth naming is that this exactness holds only at the top of each list: further down, a candidate's normalised value is a real fraction and the evidence does re-enter the sum, scaled by that list's spread. The claim is exact for top-of-list comparisons and only approximate below them.

The weights sum to one with nothing left over, and that closure buys one bound. A candidate topping all three lists would score one, so a candidate present in only one list has a hard ceiling equal to that list's weight. The image arm's ceiling of 0.2 is fixed before any document is read and holds however certain the image retriever is, which is what makes a second image retriever helpful: a second term in the sum doubles the ceiling without a page changing.

Reciprocal rank fusion, and what it gives up#

The standard answer is to fuse on ranks rather than scores. Reciprocal rank fusion assigns each candidate 1 / (k + rank) in each list it appears in and sums across lists, with k conventionally 60. It came out of metasearch work presented in 2009 at TREC, the long-running retrieval evaluation programme whose shared tasks produced most of the ranking metrics in common use, as a way to combine runs whose scores were not comparable at all, which is exactly the position three retrievers over three representations are in, and its durability comes from having no parameters to tune per corpus.

def rrf(runs, k=60, parent_of=lambda uid: uid):
    """runs: {retriever_name: [unit_id, ...] in rank order}"""
    scores, members = {}, {}
    for run in runs.values():
        for rank, unit_id in enumerate(run, start=1):
            group = parent_of(unit_id)          # ch.4's parent_unit_id
            scores[group] = scores.get(group, 0.0) + 1.0 / (k + rank)
            members.setdefault(group, []).append(unit_id)
    ordered = sorted(scores, key=scores.get, reverse=True)
    return [(g, scores[g], members[g]) for g in ordered]

Run the same three lists through it and page 14's image moves from 0.200 to 1/61 = 0.0164, third overall, behind the glossary chunk at 0.0328 and the page's own OCR text at 0.0320. Better, and still not the answer.

Every one of those three figures can be read straight off the ranks, and doing so shows where the method's character comes from. The page image stands first in the image list and appears nowhere else, so it collects one term at rank one: one over sixty-one. The glossary chunk stands first in both text lists and collects that same term twice, which is why its score is exactly double. Page 14's OCR text sits third in the lexical list — below the glossary's 14.2 and the board minutes' 11.8 — and second in the dense list, so it collects one over sixty-three plus one over sixty-two.

The near-tie between those last two is the design of the method showing through. The glossary earns its score from two first places and page 14's text from a third and a second, and the two totals differ by well under a tenth of either. That is what a large k is for: adding sixty to every rank before inverting flattens the top of each list until the difference between rank one and rank three is a rounding detail, so appearing in two lists beats appearing high in one. Working only from positions already throws the scores away; the sixty then throws away most of what the positions said, and what is left deciding the order is how many lists a candidate turned up in. The flattening is not total — the glossary does finish above page 14's text, and rank position is what separates them — but position has been demoted from the thing that decides the order to the thing that breaks ties.

The parent_of argument is what closes the remaining gap, and it is the reason chapter 4 insisted on recording parent_unit_id at index time. The OCR text of page 14 and the image of page 14 are the same evidence retrieved through two representations. Grouping them before scoring gives page 14 a combined 0.0484 and rank 1, ahead of the glossary. The deduplication is not a tidiness measure — it is the step that lets agreement across representations of the same page count once and correctly, instead of splitting a page's support across two entries that then lose to a chunk with undivided support.

The same move under min-max rescues nothing, and the contrast is where the two fixes separate. Adding page 14's two normalised contributions together under the weighted sum still leaves it well short of a candidate topping both text lists, since the two text weights together are four times the image weight and grouping does not change a weight. Under reciprocal rank fusion there is no weight to be short of, so page 14's three rank appearances outnumber the glossary's two and it finishes first. Two rules that share no step — one keeps magnitudes and rescales them, the other discards magnitudes and counts positions — agree that the ungrouped glossary beats the ungrouped image, which says the failure is not an artefact of min-max and that swapping the rule is necessary rather than sufficient. They disagree about what grouping is worth, and that disagreement is exactly the weight ceiling. The margin after grouping is thin, one term's worth, and it would disappear if the glossary turned up in a third list.

What RRF costs is the opposite of what min-max costs. Min-max keeps magnitude and destroys comparability; RRF destroys magnitude entirely. A retriever that is extremely confident about its top hit gets the same 1/61 as one that is barely distinguishing its top hit from its second. On queries where one modality is decisively right and the others are guessing, RRF hands the guessers equal weight, and the fix — weighting the runs — reintroduces the global-weight problem in a milder form.

Between the two sits a family of distribution-based normalisations that keep more information than ranks and less arbitrariness than min-max. Z-scoring each list — subtracting the list's mean and dividing by its standard deviation — makes a candidate's score a statement about how far it stands out from its own list's typical result, which is at least a property of the retrieval rather than of the truncation point. It fixes the min-end artefact entirely, since a candidate's score no longer depends on what happened to be last. What it does not fix is the cross-list comparability that started the problem: a z-score of 2.1 from the image retriever and a z-score of 2.1 from BM25 are both "unusually good for this list", and whether one is better evidence than the other depends on scale parameters the normalisation just divided away. Calibrating each retriever's scores against held-out relevance judgements — fitting the mapping from raw score to probability-of-sufficiency per retriever — genuinely does make them comparable, and it costs a labelled set per retriever and a recalibration every time an encoder or corpus changes. It is the right answer for a system that will run unchanged for years and overkill for one still finding its shape.

Calibration is the only one of the three that adds information rather than rearranging what is there. Min-max and z-scoring are computed from a list and nothing else, so neither can know anything the list did not already contain, and the belief that either has "put the scores on the same scale" is the third mistake this chapter has to name. It is tempting because the numbers do share a range afterwards, and a shared range looks like a shared meaning. Matching the endpoints of two lists matches the endpoints and nothing between them: the two largest values now agree and the two smallest now agree, and nothing has been said about whether the middle of one list means what the middle of the other does. Calibration escapes that by fitting against labels, which come from outside the list.

The candidate pool is its own set of decisions#

Fusion presupposes a merged pool, and assembling that pool has choices in it that get made by default when nobody makes them.

The three k values differ for a reason. Text candidates are nearly free to carry through the pipeline, so drawing 100 costs almost nothing. Image candidates get rendered, possibly reranked by a model that reads pixels, and finally consume thousands of tokens each, so a pool of 40 is already generous when only four to six survive into the prompt. Setting all three to the same k because it looks symmetrical either starves the text side or pays for image candidates that were never going to be sent.

The chain under that asymmetry is short and each link is checkable. An image candidate cannot be scored or shown without first being rendered into pixels, and rendering is per candidate, as is any pass by a model that reads pixels. So both costs scale with pool depth. A surviving image then occupies thousands of tokens in the prompt where a text chunk occupies hundreds, so the marginal cost of one more unit of depth is far higher on the image side. And only a handful of image candidates can be sent at all, so depth beyond that handful buys candidates that were never eligible.

Deduplication happens before reranking rather than after, because a reranker asked to score a page's text chunk and the same page's image independently will spend two of its slots on one piece of evidence and will produce two different scores for it, leaving downstream code to guess which one to believe.

The reranker itself is the strongest available fix for the calibration problem and the most expensive. A cross-modal reranker takes the query and one candidate — a paragraph or a crop — and returns a relevance score on one scale, which is precisely what the three retrievers cannot do. Because it looks at both together rather than embedding them separately, it can tell that a page showing a churn line chart with a visible 5% gridline crossing is better evidence for this query than a glossary entry defining churn, which no per-modality similarity can express.

That is a different kind of instrument rather than a better-tuned one. Every score in the merged table was computed by embedding the query once and each candidate once and comparing the two. A similarity computed that way can only report how close a candidate sits to the query in some space; it has no way to represent "this page states the fact being asked for", which is a relation between one question and one page's contents, and both had been compressed before they ever met. Reading the two together is what lets that relation survive to a point where it can be scored.

Turning up the image weight trades one population for another#

The instinct when chart questions fail is to raise the image weight until they stop failing, treating it as a tuning problem with a validation set to find the optimum. Here is what the sweep actually looks like on a query set where 78% of questions are answerable from text and 22% need a figure:

The fourth column is not a fourth measurement. Take any row, multiply the text figure by the text share of traffic and the figure-slice figure by the figure share, add the two, and the result is the overall figure printed beside them, to the two places it is given. Every row closes that way with no remainder, which means the overall column is the first two columns projected onto one particular traffic mix. A different mix of questions therefore moves the optimum without anything in the system changing, and the mix is a property of who is asking rather than of what was built.

Reading the first two columns as gaps rather than levels shows where the maximum comes from. Between the second and third rows the figure slice gains twenty-seven points while the text slice loses five; discounted by their shares of traffic, that is a gain worth about six points of overall against a loss worth about four, so overall rises by about two — which is the rise the fourth column shows. Between the third and fourth rows the figure slice gains fourteen and the text slice loses seven; discounted, about three against about five, so overall falls by about two, which is again what the column shows. The optimum sits where those two discounted marginals cross, and it is broad and shallow precisely because they cross at a shallow angle.

The overall column has a broad, shallow maximum around 0.35, and every point on it is a compromise between two populations that want different settings. At 0.45 the chart questions are working and the text questions have lost 13 points, which at this traffic mix is roughly three times as many affected queries. The comparison in that last clause is between populations rather than between the gain and the loss: the text-answerable slice is more than three times the size of the figure-answerable one, so a fall of a given size on the text side reaches roughly three times as many queries as the same-sized rise on the figure side would. Reporting 0.76 at the optimum conceals that the system got worse for most users in order to get better for some.

There is a second belief to name here, and it is the one that keeps teams sweeping. Running several retrievers and fusing them feels like a defence: if one arm is wrong, the others outvote it. That is true when the arms are independent, which is where the intuition comes from — averaging independent errors does cancel some of them. Two of these three arms read the same extracted characters of the same corpus, so their agreement is one measurement counted twice, and the sum treats it as two. Fusion in this configuration does not defend the weak arm against the strong ones. It gives the correlated pair a double vote and leaves the arm that holds the answer with whatever weight it was assigned.

The reason a sweep cannot escape this is structural. A weight is a constant applied to every query, and which modality holds the answer is a property that varies per query. No constant can express "this question is about a trend line, so trust the image retriever" and "this question is about an error code, so trust BM25" at the same time. The search for the best weight is a search for the least-bad way of being wrong about half the traffic.

What can express it is something computed per query. A cross-modal reranker does it implicitly by scoring candidates against the query rather than against a prior about modalities. A lightweight query classifier does it explicitly, predicting whether the answer is likely visual and adjusting the slot budget accordingly — cheap, around 8 ms, and it introduces its own error mode, since a misrouted query loses the modality that had the answer entirely rather than merely down-weighting it. Running the classifier to set slot budgets while letting the reranker set the ordering keeps the failure recoverable: a misrouted query still has the right candidate in the pool, just fewer slots for it.

One thing in all of this is forced and the rest is chosen. That three retrievers over three representations return numbers on incomparable scales follows from what each one computes, and no setting changes it. Min-max as the reconciliation, the weight vector, treating an absent candidate as a zero rather than a missing value, how deep each list is drawn before truncation, the sixty in the reciprocal-rank denominator, the three pool sizes, reranking twenty-five candidates, and the split of prompt slots between images and text are all decisions, and the merged ordering is their output. When a chart question fails, the first question is which of them produced the ordering, because the evidence had already been found.

Any of these changes can be shown to help the chart questions. None of them, on its own, tells you whether the visual index is worth having. A hybrid system that scores six points above text-only might be getting those points from evidence in the images, or from the ordinary ensembling gain of running two retrievers over the same text and averaging their mistakes away. Those two explanations imply completely different next projects — one says invest in the visual encoder and the resolution budget, the other says the visual arm is an expensive way to buy an ensemble — and the comparison that distinguishes them needs an arm nobody would ever ship.