A retriever prefers a document in the query's own language, right or wrong

A company selling into fourteen markets does not hold one copy of its documentation. It holds an English cancellation policy, a German translation of it produced eighteen months ago by a localisation vendor, and a Spanish one produced by a different vendor at a different time, and all three sit in the same index as separate rows. The question a retriever is answering has therefore changed shape. It is no longer whether a German query can reach an English passage; it is which of several candidates — some of them renderings of the very same content, disagreeing with each other — comes back first.

This chapter is about a preference the retriever exercises in that situation without being asked to. A German query systematically favours German documents, by an amount that is large enough to reorder the top ten and small enough to leave the results looking sensible. It is a separate failure from the alignment gap measured in the previous chapter, it survives a model with excellent alignment, and it presents to everyone downstream as a content problem rather than a retrieval one.

Why the same-language candidate wins#

A German query and a German document share more than meaning. They share script, so their byte sequences overlap; they share subword tokens, so their token sequences overlap; they share morphology, function words and orthographic conventions, so the surface statistics the encoder learned to compress overlap too. On top of all that sits the language-identity direction described in chapter 1 — the direction a masked-language-modelling objective is directly rewarded for encoding — which is present even in models aligned on bitext, because alignment pulls translations closer without ever collapsing the languages onto each other.

The result is a similarity floor that has nothing to do with relevance. Take a question about a cancellation policy, in German, and score four candidates with an aligned multilingual encoder:

Start with the ordering rather than the numbers, because the ordering is the finding and needs no arithmetic. The correct answer is third. It is beaten by the stale German translation of itself, which is expected — that document says the same thing in the query's language, and only the notice period differs. It is also beaten by a German article on a different subject, and that is the part that should worry you, because it means the wrong topic in the right language outranked the right topic in the wrong language.

Now size the effect, which takes two subtractions and one decision about which pair to read. The clean measurement of the language bonus is the KB-1180 pair, because those two rows are renderings of one piece of content on one subject: German at 0.703 against English at 0.612 is a gap of about nine hundredths, attributable to language and nothing else. The KB-2291 pair gives a larger gap of about thirteen hundredths, and that one is contaminated — those two rows differ in language and in content, since one says fourteen days and the other says thirty. Reading the bonus off the contaminated pair would overstate it, so roughly 0.09 is the number to carry.

The topical difference is the other subtraction, and it is worth being honest that the table gives two answers. Within English, cancellation at 0.664 against pausing at 0.612 is about five hundredths. Within German, 0.791 against 0.703 is about nine. Call it about 0.06 as the middle of that bracket. Either way the comparison lands in the same place: language identity is at least as strong a ranking signal as subject matter for this query, which means the ordering of your top ten is partly a sort by language with relevance as a tiebreaker.

The bonus is not uniform. It is largest between languages sharing a script and a lot of borrowed vocabulary, and it interacts with document length, since a short German document accumulates less topical signal to offset it. Empirically, on a mixed corpus with an aligned encoder, the same-language preference tends to be worth somewhere between 0.05 and 0.12 of cosine, which is enough to reorder the top ten and rarely enough to pull an unrelated document into it. That is precisely the range that produces plausible-but-wrong results rather than obvious garbage, and plausible-but-wrong is what gets filed as a content bug.

A hybrid retriever amplifies all of this rather than damping it, and the mechanism is arithmetic rather than subtle. The lexical branch — BM25, the standard keyword-scoring function, or whatever your search engine calls it — scores documents by the query's terms appearing in them. A German query contains German terms. An English document contains English terms. So the lexical branch has no cross-lingual capability whatsoever: for a German query it returns German documents almost exclusively, with the occasional English hit carried by a shared product name or a numeral. Fusing that branch with the dense, embedding-based one — merging the two ranked lists into a single ordering — at equal weight therefore hands half your candidate budget to a component that is structurally incapable of returning the English original, and the fused list ends up more same-language-biased than either input. Measuring the two branches separately before fusion is worth doing once: on these German queries the lexical branch returned German documents in 96% of its top-ten slots, and reweighting fusion from 0.5/0.5 to 0.25/0.75 in favour of the dense branch moved de→en recall@10 by 0.04 with no measurable cost to de→de.

Filling in the grid#

Chapter 2 measured one column: several query languages against a purely English corpus. A mixed corpus turns that column into a grid, and the grid has to be filled in cell by cell, with the same fixed judgments, because the interesting behaviour lives in the comparison between cells rather than in any of them alone.

The judgments need one extension to make this work, and it is the extension that makes the whole chapter measurable. A relevance judgment in a mixed corpus has to name the logical document rather than one language's copy of it. The reason is that "the article that answers this" is a fact about content, and KB-2291 and KB-2291-de are two renderings of one piece of content that happen to disagree — so a judgment naming a doc_id is silently also a decision about which rendering the user should get, taken by whoever wrote the judgment, usually without noticing. Splitting those apart means giving every translation group a stable canonical_id, judging at the canonical level, and then deciding separately — as a policy question — which rendering the user should see.

{
  "doc_id": "KB-2291-de",
  "canonical_id": "KB-2291",
  "lang": "de",
  "is_translation_of": "KB-2291",
  "source_updated_at": "2026-03-04",
  "translation_updated_at": "2023-11-19",
  "authority": "localised",
  "script": "Latn"
}

With that metadata in place, a sweep produces a table with two distinct populations in it. Here is the German row and the English row from a five-language corpus, with recall@10 computed at the canonical level:

Two of those cells should look familiar, and the fact that they do is a check worth performing. The de→en cell reads 0.766 and the en→en cell reads 0.863, which are exactly the German and English recall figures from the previous chapter's drop table. They agree because they are the same measurement: chapter 2's table is the "Corpus en" column of this grid, computed before there was anything else in the corpus to compare against. Two independently constructed tables landing on the same two numbers licenses treating the grid as an extension of the earlier method rather than as a new one, so the earlier conclusions carry over unchanged.

Reporting the diagonal separately from the off-diagonal cells is what makes this readable. The diagonal — de→de at 0.881, en→en at 0.863 — is monolingual retrieval, and it is the number that looks fine on every dashboard and that MIRACL-style benchmarks report. The off-diagonal cells are cross-lingual retrieval, and they are systematically lower everywhere. Pooling them together gives a per-language mean that rises or falls with how much content happens to exist in that language, so a locale's number can improve because the localisation team shipped forty translations, with no change in retrieval quality at all.

There is a third quantity the grid does not contain and that belongs beside it: the language distribution of the returned results. For each query language, print what fraction of the top ten came from each corpus language, next to the corpus's own composition. That comparison is what names the bias directly rather than leaving it to be inferred from a spread of scores, and the arithmetic is a single expectation. If ranking ignored language entirely, German documents would take a share of the top ten equal to their share of the corpus, since nothing would distinguish them. In one such system the corpus is 22% German by document count, and for German queries a German document was ranked first 84% of the time — close to four times the language-blind expectation, which is the size of the effect stated without any cosines at all.

Making the policy explicit#

Once you have those numbers, the question stops being about the encoder. Some component is deciding which language's rendering of a canonical document a user sees, and by default that component is the cosine ranking, which has no information about freshness, authority or the user's actual language preference and is applying a same-language bonus it was never asked to apply.

The shape of that pipeline is the point: retrieval decides what the answer is, at the canonical level, and a separate policy step decides which rendering of it to serve. Collapsing renderings before ranking also removes a second problem the grid exposes, which is that three translations of one article can occupy three of your ten slots and squeeze out genuinely different content.

def collapse(hits, docs, locale, max_translation_age_days=90):
    best = {}
    for h in hits:                      # hits sorted by score desc
        c = docs[h.doc_id]["canonical_id"]
        best.setdefault(c, {"score": h.score, "renderings": []})
        best[c]["score"] = max(best[c]["score"], h.score)
        best[c]["renderings"].append(h.doc_id)

    for c, entry in best.items():
        entry["serve"] = choose_rendering(c, locale, docs,
                                          max_translation_age_days)
    return sorted(best.items(), key=lambda kv: -kv[1]["score"])

Taking the maximum over renderings, rather than the mean or the first, is the one line worth defending. A canonical document's score should answer "did any rendering of this content match the query", because the content either answers the need or it does not; averaging would penalise a canonical for having a badly-translated rendering, which is a fact about the translation and not about whether the content is the answer.

choose_rendering is where the organisation's actual policy has to be written down, and there are only a few defensible ones.

The last two are the ones worth arguing about, and both need translation_updated_at to be real rather than a field that gets bumped by a CMS re-save. The 90-day staleness window in the code above is a stand-in for a decision that belongs to whoever owns the content: legal and pricing pages might need seven days, and a troubleshooting article can be a year old without harming anyone.

Better alignment does not remove the bonus#

This is not the alignment problem from chapters 1 and 2, and swapping in a better-aligned encoder does not fix it. A stronger model raises cos(de query, en document) from 0.664 towards 0.72, which narrows the same-language bonus and does not remove it, because the bonus comes from genuine shared surface structure that the model is right to encode. A model that treated German and English text as indistinguishable would be worse at everything else you use it for — it could not tell you what language a document is in, and it would lose the morphological signal that makes monolingual German retrieval work at 0.881.

It is also not a language-detection problem, although language detection is a component of the fix. Knowing that the query is German is easy and nearly free; it tells you nothing about which corpus language should win, which is a question about content authority and freshness that lives in your metadata. Teams that install a language detector and filter the index to the detected language have chosen the second row of the policy table without noticing they made a choice, and they usually discover it when coverage for a small locale turns out to be 6% of the corpus.

Nor is canonical collapsing the same as near-duplicate detection. Near-duplicate filtering removes redundancy by similarity threshold and would happily merge two genuinely different articles that share boilerplate, while leaving a translation pair separate if the cross-lingual similarity fell below the threshold. Translation groups are a fact recorded by the content system, and treating them as something to be inferred from vectors is how you end up merging the German cancellation article with the Austrian one.

Finally, the grid extends further than language. The same query language and corpus language can behave very differently across scripts within one locale, and navigational queries behave differently from descriptive ones, so query language crossed with corpus language is the first two dimensions of the stratification rather than all of it.

What a mixed corpus adds, then, is a competition the single-corpus case never had. Cosine ranking resolves that competition using a signal — shared script, shared subwords, shared morphology — that is real, that the model is correct to encode, and that carries no information whatever about which rendering is current or authoritative. The repair is not a better encoder. It is to judge at the level of the content, collapse renderings before ranking, and write down which rendering gets served, so that a decision somebody would defend in a meeting is being made by a rule rather than by a tiebreak.

The five-language grid above has 25 cells, and a real system with fourteen locales has 196. Filling all of them from a 480-query set means some cells are computed from nine queries, and a cell computed from nine queries can move 0.2 because one judgment was wrong. Knowing which cells carry evidence and which are decoration is a counting problem, and it is the reason an aggregate that has stayed green all quarter can be entirely uninformative about the locale that is on fire.