The cross-lingual drop is a table of language pairs, not one score

Most teams that have shipped English retrieval already own the expensive half of a cross-lingual evaluation without knowing it. A support assistant serving several locales out of one English knowledge base has a golden set: 480 English questions drawn from real tickets, each with the article that answers it marked relevant, plus a few dozen partial-credit judgments added during a relevance review. Against a 41,000-article index it scores nDCG@10 of 0.71 and recall@10 of 0.86 — recall@10 being the fraction of questions whose answering article turns up anywhere in the top ten results, and nDCG@10 the same idea with extra credit for placing it near the top rather than at position nine — and those two numbers are what the retrieval team reports.

The reason that set is most of a cross-lingual evaluation is that everything in it except the queries is language-neutral. Its corpus is English and stays English. The relevance judgments say "for this information need, article KB-2291 is the answer", and that is a statement about what the article contains, so it does not change when the need is expressed in Turkish. So you can hold the corpus, the judgments, the index, the retriever configuration and the cutoff completely still, translate the 480 questions, and run them again. Whatever the score does is caused by the query language, because nothing else moved.

Holding everything still except one variable#

The discipline is the whole value of the method, and it is easy to break by accident. Anything that differs between the runs other than the surface language of the query becomes an alternative explanation for the drop, and the usual culprits are mundane: someone re-indexes between runs, someone changes the chunker, someone uses the vendor's German analyzer for the German run because it seemed like the right thing to do, someone lowers k for the second run because the first was slow. Each of those makes the resulting number uninterpretable, and none of them announces itself.

import json, pytrec_eval
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")
qrels = json.load(open("qrels_en.json"))          # {qid: {doc_id: grade}}
index = load_index("kb_en_41k")                    # built once, never rebuilt

def run(queries, k=10):
    vecs = model.encode([f"query: {q}" for q in queries.values()],
                        normalize_embeddings=True)
    return {qid: index.search(v, k=k)
            for qid, v in zip(queries.keys(), vecs)}

ev = pytrec_eval.RelevanceEvaluator(qrels, {"ndcg_cut_10", "recall_10"})

for lang in ["en", "de", "tr", "es", "ja"]:
    queries = json.load(open(f"queries_{lang}.json"))   # same 480 qids
    scores = ev.evaluate(run(queries))
    n = len(scores)
    ndcg = sum(s["ndcg_cut_10"] for s in scores.values()) / n
    rec = sum(s["recall_10"] for s in scores.values()) / n
    print(f"{lang}->en  n={n}  nDCG@10={ndcg:.3f}  recall@10={rec:.3f}")

The qrels object — the relevance judgments, under the name the TREC evaluation format gave them — is loaded once and never touched, the index is built once, and the loop varies exactly one thing. Two details in that arrangement do more work than they look like they do. index is constructed outside the loop, so no arm can accidentally get a freshly built graph with different neighbour lists. And every query set carries the same 480 query IDs, which is what lets you compute per-query deltas afterwards and find the queries that broke rather than only the average that moved. Query sets with different IDs can only ever be compared as aggregates.

What the run produces#

The fourth column is the second column subtracted from the en→en row, and it closes exactly in all four rows, which is worth checking rather than assuming. That closure is what makes the deltas comparable with one another: every one of them is measured against the same baseline by the same operation, so the statement that Turkish costs four times what Spanish costs is arithmetic rather than impression.

Five rows, and the useful information is in the spread rather than in any single cell. Spanish costs seven points of nDCG, which is the kind of degradation you can absorb with a slightly larger k and a reranker, meaning a second-stage model that re-scores the top handful of candidates before anyone sees them. Turkish loses thirty points and 148 of 480 gold articles fall out of the top ten entirely, which is not a degraded experience but a different product. If you had pooled all five runs into one cross-lingual number you would have reported 0.567 and described no user at all: no human being issues queries in a uniform mixture of Spanish, German, Japanese and Turkish, so the average corresponds to nobody's session. A pooled figure also depends on which queries survived each run, which is a decision described below, so it is not even reproducible from the table.

The per-pair framing matters for a second reason that shows up as soon as the corpus stops being purely English. What you are measuring is a cell in a grid indexed by query language crossed with corpus language, and the table above is one column of it. Reporting "the German score" rather than "German query against English corpus" makes the column implicit, which works exactly until someone adds German documents.

The rightmost column is worth printing next to the metrics because it is the number you can act on, and it also carries information the average does not. German's recall falls by about ten points, and ten points over 480 queries is a net loss of roughly fifty gold articles from the top ten — while the column records sixty-one departures. The two do not have to agree, and the gap between them is the count of queries the language change helped: articles that were outside the top ten in English and arrived inside it in German. Every row shows the same pattern, with the gross departures exceeding the net loss by a dozen or so. So the drop is a net figure over two-way movement rather than a uniform slide, and a per-query diff will always find queries that improved. The arithmetic here only closes approximately, because a handful of queries have more than one relevant article and contribute a fraction rather than a whole.

An average that moved 0.13 is an argument; a list of 61 query IDs whose gold article dropped out of the top ten is a debugging session. Reading twenty of those 61 German queries takes half an hour, and on this set they sorted into three recognisable groups: 9 where the gold article was displaced by another billing article with heavy term overlap, 7 where the query contained a compound noun the encoder appeared to treat as a single opaque token, and 4 where the English gold article was genuinely ambiguous and the English run had been getting credit for a coin flip. The last group is a judgment problem rather than a language one, and finding a few of those in every sweep is normal.

Reusing the English judgments is what makes this affordable, and it is valid only where the information need survives the change of language. Most support needs do: a returned direct debit is the same event whether the customer describes it in Turkish or English, and KB-2291 answers it either way. Some do not. Roughly 30 of the 480 queries in this set were about tax invoices, data-retention rights and payment methods, where the correct article differs by country, so the English judgment points at the article for the US and the German user needs the DACH one. Those queries have to be re-judged per locale or excluded from the cross-lingual runs, and excluding them silently is better than leaving them in with the wrong gold, because a query whose judgment is wrong contributes noise in the direction of "the locale is broken" and adds a phantom point or two to every drop you report.

The instrument flatters the result#

The queries for that table were produced by machine-translating the English set with a commercial machine translation (MT) service, which is the only reason it was cheap enough to run on a Tuesday. That choice puts a specific, directional bias into every number in the table, and the bias has to be stated wherever the number is quoted.

Work out what a translated query actually is and the direction of the bias follows without any measurement. It is a near-parallel of an English query that was itself written by an English speaker who had the English corpus in mind. It therefore inherits English sentence structure, English length, English politeness conventions, and a term-by-term correspondence with the vocabulary of the documents being searched. Now recall from the previous chapter how cross-lingual alignment was trained: on bitext, meaning sentence-level parallel pairs. A machine-translated query is a sentence-level parallel of an English sentence. It is, almost by definition, the input distribution the alignment objectives described in the previous chapter were trained on, which makes it close to the easiest possible input for a cross-lingual retriever.

Real queries from the locale look different in ways that are measurable. On a German support queue, the mean production query is 6.2 words against 11.4 for the translated set. Native queries use the local term where the translation used a literal rendering: users write Lastschrift and SEPA-Mandat, while the MT output of "direct debit mandate" came back as Einzugsermächtigung, a word that is legally obsolete and appears nowhere in the corpus. They code-switch, embedding untranslated English product nouns inside German sentences, which produces mixed-script-free but mixed-vocabulary inputs no bitext ever contained. And they skew towards keyword fragments rather than full questions, because that is what people type into a search box.

translated:  "Warum wurde meine Lastschrift von meiner Bank zurückgegeben?"
native:      "lastschrift zurück rücklastschrift gebühr"

translated:  "Wie kann ich das Abrechnungsintervall meines Kontos ändern?"
native:      "billing cycle umstellen monatlich"

Both of those native examples score worse than their translated counterparts on the same qrels, and the second one is a query the translated set could not have produced at any temperature. The practical consequence is that the drop table above is an optimistic bound: the true de→en gap is at least 0.129 and the true tr→en gap is at least 0.303.

Optimism is the wrong direction here, and it is wrong in a way that compounds. The lift a query gets from still being shaped like the English it was translated out of — call it the translationese advantage — is largest exactly where native query conventions diverge most from English. The agglutinative languages, where the native keyword form is a single compound word, get a large lift from being handed a multi-word English-shaped phrase instead. Languages whose domain vocabulary is a borrowed English term get a large lift because MT dutifully translates that term away, leaving behind a rendering the corpus does contain. Both descriptions pick out the locales furthest from English, which are the locales whose numbers you were most worried about, so the measurement is most flattering exactly where it most needs to be honest.

There is a way to get most of the way to the right-hand column without writing queries from scratch: take the 480 English information needs, and have a native speaker rewrite the query as they would type it, keeping the need — and therefore the judgment — intact. That preserves qrels reuse, which is the expensive asset, while replacing the surface form, which is the biased part.

A property of a pipeline rather than of a model#

The drop is a property of a pair and a system, and not of the embedding model on its own. Everything between the query string and the ranked list contributes: the tokeniser, the normalisation applied before encoding, the chunk boundaries, any lexical branch in a hybrid retriever — one that runs keyword matching beside the vector search and merges the two result lists — the reranker if there is one, and the index's own configuration. Attributing the whole 0.303 to "the model is bad at Turkish" is the most common misreading, and it sends people shopping for a new encoder when the fault may sit two components upstream. Chapter 5 takes a broken locale apart layer by layer for exactly this reason.

The drop is also not an end-to-end quality measure. It says the answering article was or was not in the top ten, and it says nothing about whether the generation step then produced a usable German answer from an English source, which is a separate evaluation with separate judgments. A locale can have healthy retrieval and unusable output, and the reverse is rarer but happens when a long-context generator recovers from a mediocre candidate set.

Nor does a drop table settle the translate-versus-native architecture question, and the reason is the bias derived above turned round. If you build the German queries with MT and then embed them with a multilingual model, you have measured neither production path cleanly: you measured a multilingual encoder eating translationese, which is close to the translate-then-search path's input distribution and unlike the native path's. Comparing the two architectures properly means running them over the same native queries, which is chapter 6.

Finally, one clean sweep of the grid is a snapshot. Query distributions shift when a locale grows, corpora get new documents in new languages, and a model upgrade can move one pair while leaving the others still. The table is something you re-run on a schedule and diff, and per-query IDs are what make the diff readable.

A cross-lingual drop is therefore two things at once: a controlled measurement, which is what makes it worth having, and a measurement of a query population you chose, which is what makes a single figure misleading. Fix everything but the query language and the difference has one cause. Report it per pair, since the spread from seven points to thirty is the finding and the pooled mean is not. Quote the bias in the same sentence as the result, because a translated query set is a best case and its optimism is largest for the locales furthest from English. And keep the query IDs stable, because the list of queries that broke is worth more than the average that moved.

Everything so far has assumed one corpus language. That assumption holds for a company whose documentation is English-only and is false for most companies with fourteen locales, because the same cancellation policy exists in English, German and Spanish, written at different times by different teams and not saying quite the same thing. Once the corpus is mixed, a German query is choosing between candidates in several languages at once, and the retriever has a strong preference between them that has nothing to do with which one is correct.