Cross-lingual retrieval

Cross-lingual retrieval is matching a query written in one language against documents written in another, and it is the default situation for any assistant sold into more than one market on top of a knowledge base written in English. It is also the stage where a multilingual failure most often originates and the stage that least often gets blamed, because the observable symptom is a sentence the model produced, sentences are what prompts control, and so the prompt is where attention goes.

Nothing about the symptom points at the retriever. A retriever that returns the wrong ten documents does not raise an error; it returns ten documents with respectable similarity scores, and the model does the only honest thing available to it, which is to say that the answer is not in what it was given. Separating the two possibilities takes an afternoon and a set of labels, and everything else in this chapter depends on having done it.

The silence has a cause, and stating it is what the monitor at the end of this chapter is built on. A similarity score is relative. It ranks documents against each other for one query; it does not judge whether any of them is relevant, and it has no representation of an empty result. Ask for ten and ten come back, every time, because the operation is sort by score and take the first ten, and a sorted list is never empty. The scores attached to them look respectable for the same reason: a score means something only against the distribution of scores that query produces, and nothing in the pipeline makes that comparison. So there is no stage at which the system could have noticed, and the first thing capable of noticing is a person reading the answer.

Ask retrieval the question on its own#

Take the non-English questions, attach to each one the identifier of the English document that answers it, and run retrieval alone.

def recall_at_k(cases, retrieve, k=10):
    hits = 0
    for c in cases:                      # c = {"query": str, "gold_doc_ids": set[str]}
        got = {d.id for d in retrieve(c["query"], k=k)}
        hits += bool(got & c["gold_doc_ids"])
    return hits / len(cases)

for lang, cases in fixed_sets.items():
    r10 = recall_at_k(cases, retrieve)
    r10_pivot = recall_at_k(cases, lambda q, k: retrieve(translate(q, "en"), k=k))
    print(f"{lang}: native={r10:.2f}  query-translated={r10_pivot:.2f}")

Run against a corpus of English support articles, that produces the grid that decides where the work goes:

Retrieval finds the right document for German just under six times in ten. There is no prompt on earth that makes a model answer from a document it was never shown, so the whole of that 12.6% is a retrieval question, and iteration on the prompt would move a number that is not the bottleneck.

The gold document identifiers are the part that costs something. Somebody has to sit with each question and record which article answers it, which is roughly ninety seconds per case with a good search interface, so a 150-case set is about four hours of a person who knows the corpus. That is the price of admission for every claim in this chapter, and it is why teams skip the step and go straight to the prompt, which costs nothing to start and produces a diff by lunchtime.

That is one multiplication: ninety seconds is a minute and a half, and a hundred and fifty of those is a little over two hundred minutes, which is under four hours. The figure is worth carrying because it is the number the decision to skip this step is implicitly made against, and it is almost never stated out loud. An afternoon of one person who knows the corpus, once per language, buys the ability to say which stage is broken. The alternative costs nothing to begin and then costs a week of prompt iteration, at the end of which it has answered a different question.

Why the prompt patch appears to work#

Suppose the instruction is softened instead, telling the model to answer from its general knowledge when the retrieved passages seem thin. The dashboard reacts immediately and in the desired direction:

The complaint that started the investigation goes away, because the model stops saying it could not find anything and starts producing plausible German answers about refund windows and cancellation terms drawn from whatever it absorbed in pretraining. Those answers are frequently right, which is what makes this stick, and the ones that are wrong no longer announce themselves. A visible failure has been converted into an invisible one, and the invisible version reaches customers.

The three rows also constrain one another, which is what makes that table an argument rather than three readings. Conversations ending in "not found" fell by a little over nine points. Answers the corpus contradicts rose by a little over seven. Those recovered conversations had to become something, and nothing else in the pipeline changed, so nearly four in five of them became answers the corpus disagrees with. The trade is not a little risk for a lot of coverage. It is most of the coverage arriving as error.

The citation row says something the other two cannot, and it is the reason a change like this can never be scoped to the cases it was written for. Valid citations fell by twenty-seven points, which is roughly three times the number of conversations that stopped saying "not found". A change touching only the recovered cases could not move that row nearly so far. The softened instruction altered behaviour on conversations that had been answering successfully all along — it lowered the bar for grounding everywhere, including everywhere it had been working.

This is the general hazard with patching the stage that produced the symptom. "I could not find that information" is a well-behaved failure: it is honest, it is countable, and it triggers a handoff. Trading it for a silent wrong answer is a strictly worse position that looks better on every dashboard the team currently has.

What breaks when the query and the corpus disagree#

Three mechanisms are usually in play and they are worth separating, because they respond to different fixes.

Lexical retrieval contributes essentially nothing across a language boundary. A BM25 arm — the standard keyword scorer, which ranks a document by how many of the query's own words it contains and how rare those words are — scoring a German query against English documents matches only on numbers, product names and loanwords, so the term overlap the scoring function is built on is close to empty. If your retriever is hybrid, meaning BM25 fused with dense vector search through reciprocal rank fusion, which merges two ranked lists on each document's position rather than its raw score and is the default in most stacks, then half of the ensemble has quietly stopped contributing for non-English queries while the fusion code carries on combining two ranked lists as though both were informative. The effective retriever is the dense arm alone, at whatever weight the fusion gives it, and nothing logs that.

Why nothing logs it is in how the fusion works. BM25 scores a document by the query terms it contains, weighted by how rare those terms are, so across a language boundary almost every document scores near nothing — and a ranking over near-nothing scores is still a ranking. What comes out is a confident-looking ordered list of ten documents. Reciprocal rank fusion then combines the two lists using each document's position rather than its score, which is normally the right design, because it makes the fusion robust to two scorers on incomparable scales. Here that robustness is the defect. Positions exist in a meaningless list exactly as they do in a meaningful one, so the fusion has discarded the one quantity that would have revealed the lexical arm had stopped contributing.

Dense retrieval works, unevenly. An embedding model turns a passage or a query into a vector of a few hundred numbers positioned so that texts with similar meaning sit close together, and multilingual embedding models are trained to align languages into one shared space so that a sentence and its translation land near each other. The alignment is much tighter for high-resource pairs with abundant parallel data than for the rest, which is why German comes in at 0.58 and Thai at 0.34 on the same corpus with the same model. Alignment also degrades faster on domain-specific language than on general language, since the parallel corpora these models learn from are news, subtitles and web text rather than your product's support articles — the model knows that Erstattung and refund are close, and it has never seen Rückerstattungsfrist für Jahrespläne next to "refund window for annual plans".

Morphology compounds the problem in specific languages. German nominal compounding produces single tokens that correspond to English phrases, Finnish and Turkish agglutination, which glues several affixes onto a stem to make one long word, does something similar, and the chunking and embedding pipeline — the job that splits each article into passages and stores a vector per passage — handles those less well than it handles the space-separated equivalents. It is the same tokenizer pressure that squeezes generation budgets in these languages, showing up on the retrieval side as vocabulary the embedding model resolves imprecisely.

Four fixes, and what each one costs#

Query translation is the cheapest thing that works and it is available whether or not the rest of the pipeline pivots. Retrieval-time query translation is a retrieval decision, not an architecture decision, and a natively-prompted assistant can translate its query for the retriever while keeping the Thai system prompt and the Thai answer. That combination — native generation, with an inbound pivot confined to the retriever — is where a lot of mature multilingual RAG (retrieval-augmented generation, meaning an assistant that answers from documents fetched at request time rather than from what the model happens to remember) ends up, and it is invisible if you treat the pivot as an all-or-nothing choice about the whole pipeline.

Corpus translation buys the best recall and the maintenance line is the reason it usually fails in year two. Every edit to an English article invalidates eleven translations, and the process that keeps them in sync is a content operations process rather than an engineering one; the failure mode is a German index that is nine months stale, answering confidently from a refund policy that changed in March.

The index-cost cell and the maintenance cell are the same multiplication applied to two different things. 4,100 articles across twelve languages is close to fifty thousand documents to store and embed, which is a storage number and therefore the easy half. The maintenance cell applies the same twelvefold factor to a rate: one change to an English article creates eleven translation tasks, and the rate those tasks arrive at is the rate the English corpus changes at — which nobody controls, and which nobody measured before choosing this row.

Two cheaper ways to get the same warning#

The four hours of gold labelling is a fine price once. Paying it for every language on every corpus revision is not, and there are two approximations that cost nothing and fire on the same condition.

The first is to mine the labels from traffic you already have. English conversations that ended with the customer confirming the issue was resolved, and where the model cited a document, give you a query and a gold document identifier for free. Filter to the conversations with a single citation, take a few hundred, and you have an English retrieval set built without an annotator. The German set can then be seeded by having a speaker rewrite those queries as a German user would ask them, which is faster than authoring from scratch because the gold answer is already fixed — the same seed-and-adapt shape that works for per-language safety sets, for the same reason.

The second needs no labels at all. Retrieval scores are already being computed, and their distribution per language is a signal on its own. Mean top-1 similarity, the score of the single best-matching chunk for a query, ran 0.81 in this corpus for English queries and 0.63 for German, with the German distribution not merely shifted but flatter — the gap between the first and tenth hit was 0.09 for German against 0.19 for English, which is what "the retriever cannot tell these documents apart" looks like numerically. Neither number tells you whether the right document came back, and both move sharply when cross-lingual retrieval degrades, so they work as a monitor: alert on the per-language top-1 score distribution and on the first-to-tenth spread, and investigate with gold labels only when one of them moves. That turns a four-hour annotation task from a prerequisite into a response.

The spread is the more informative of the two, and it is worth being exact about what it implies. It measures how much better the best match is than the tenth. A wide gap says the retriever found something distinctive. A narrow one says the ten are close to interchangeable to it, so which of them lands first is close to arbitrary. That degrades recall at one far faster than recall at ten, which matters because a model handed ten passages reads all ten while a reranker handed ten reorders them. Isolating the two is worth the trouble: a language whose recall at ten holds up while its top-1 spread collapses has a reranking problem rather than a retrieval one, and the two take different fixes.

Recall is not answer quality#

Everything above measures whether the right document came back, and none of it measures whether the answer is any good. Instruction compliance thins in the same languages, structure gets dropped, the answer runs short — each stage in a multilingual pipeline degrades in the same direction at once, so the compounding is worse than in a monolingual pipeline where only one stage is under stress. Component metrics tell you where a stage stands, and topic 3.14 already covered why they do not compose into an end-to-end number.

Isolation also assumes you know what the stages are and can run them individually with real inputs. That holds for retrieval, because a retriever is a function you can call. It holds much less well for a boundary like "the model's interpretation of an ambiguous term", which is not a stage with an interface. And it requires that the fixed set carry a label for the intermediate stage — the gold document identifiers here — which is a per-language annotation cost that grows with every language and every corpus revision.

The discipline that survives all of that is narrow: when a multilingual assistant reports that it could not find something, the first measurement is always retrieval on its own, against labels, per language. It is cheap relative to the alternatives, it is decisive in a way a prompt experiment is not, and the patch it prevents is one that would have made the dashboard better and the product worse.

Retrieval is the easy case because the gold label is a document identifier and the check is a set intersection. The harder situation is a failure that has already passed through five stages, where the answer is fluent, confident, wrong, and every stage individually did what it was told.