A leaderboard number is one average over dozens of datasets

Picking an embedding model off the top of a public ranking is the cheapest decision available in a retrieval project, and it fails in a repeatable way: the model sitting several places higher retrieves worse on the material you actually have. That failure is rarely a mistake in the ranking. The published score and your own measurements are usually both correct, and they are correct about different things, because a public ranking is built to describe many bodies of text at once while you have exactly one. The cost of not reading it that way is paid in machine time, since changing which model you use means producing every stored vector again before anyone can tell whether the change helped.

Reading such a number properly means knowing what got averaged into it and what got averaged out. One score per model is a single figure standing in for several independent properties — how the queries are phrased, what vocabulary the documents use, which languages appear, how long a document is, what somebody decided counts as relevant, which measure was taken and how the search was run. Any of those can differ between a benchmark and your own system, and any of them on its own can flip which of two models is better.

What a leaderboard is scoring#

The thing being ranked is worth naming plainly before any argument leans on it. An embedding model is a program you hand a piece of text and that hands back a fixed-length list of numbers — a vector of 768 or 1024 coordinates, the representation 8.4 introduced — arranged so that texts with related meaning come out close together and unrelated ones far apart. bge-base-en-v1.5 is one of them, a few hundred megabytes of downloadable weights; hosted providers sell others behind an endpoint that takes a string and returns an array of floats. Such a model exists because keyword matching cannot connect a support engineer asking about a container stuck at a port to the contract clause that only ever uses the word demurrage, whereas a model that has read enough English to place those two texts near each other in its space can.

It is used at two moments. Offline, every document is cut into chunks — passage-sized pieces of a few hundred tokens, each small enough to stand on its own — and every chunk goes through the model once, with the resulting vector stored in a database built to find near neighbours quickly. At serving time the user's query goes through the same model and the database returns the chunks whose vectors lie nearest to it. That pair of steps is what an internal search box runs on, and it is the retrieval half of a retrieval-augmented generation pipeline, where the chunks that come back are pasted into a language model's prompt so it can answer from them. Because the stored vectors were produced by one particular model, a different model makes all of them meaningless at once, which is why swapping one is an offline job measured in machine-days rather than a configuration change.

The evidence for such a swap usually comes from a leaderboard, a public table that runs many embedding models over the same benchmark datasets and sorts them by their mean score. Both that table and any local regression run are quoted in the standard retrieval measures. Recall@10 is the share of the documents known to answer a query that appear anywhere in the top ten results; nDCG@10, normalised discounted cumulative gain at ten, is one number that also rewards placing them nearer the top; MRR, mean reciprocal rank, follows how far down the list the first relevant document sits. The three can move in different directions on the same change, since recall only asks whether a document is inside the cutoff at all, while the other two also ask where inside it the document landed. A set of your own queries with recorded judgments about which chunks answer them is a golden set, the local counterpart of a benchmark dataset.

What one number is averaging over#

The Massive Text Embedding Benchmark, or MTEB, in the form most leaderboards present it, spans eight task categories across 58 datasets and 112 languages. Classification, clustering, pair classification, reranking, retrieval, semantic textual similarity, summarisation and bitext mining are all in there, and the headline column that orders the table is a mean over those categories. A model can climb that column on the strength of clustering and classification scores that have no bearing on whether it puts the right contract clause in your top ten. Filtering to the retrieval column is the first correction, and every serious comparison starts there rather than at the overall average.

That correction is not sufficient, because the retrieval column is itself an average over fifteen datasets drawn from BEIR, a suite assembled to test retrieval models on domains and query styles they were not trained for, and those fifteen disagree with each other about what retrieval even looks like. MS MARCO carries short web search queries against passage-length documents. Natural Questions carries real Google questions against Wikipedia. HotpotQA needs two documents joined to answer one question. FiQA is finance-forum questions against forum answers. SciFact matches a one-sentence scientific claim to an abstract. ArguAna is the outlier that most people never notice: its queries are entire argument paragraphs, hundreds of tokens long, and a model that scores well there is being rewarded for long-query behaviour that will never occur in a search box.

The step that makes those disagreements matter is worth taking slowly, because it is the whole argument of this chapter compressed into one move. A dataset is not a neutral sample of retrieval. It is a set of choices somebody made — who wrote the queries and in what register, what a document was cut down to, which vocabulary appears, which language, and what a judge was told to count as an answer. A model's score on that dataset is therefore a measurement taken under those choices. Averaging fifteen such scores averages fifteen sets of choices, each entering with the same weight regardless of how many of them your own system shares. A model that is excellent under one set of assumptions and mediocre under fourteen others can still finish behind a model that is uniformly adequate, and if the one set is the one you run on, the ranking has told you the opposite of what you needed.

Read that table as a list of independent axes rather than a list of complaints. Each row is one dimension along which a benchmark and a production system can differ, and the rows are close to independent of each other: how your users phrase a query says nothing about which languages they write in, and neither says anything about how long your chunks are or whether a filter runs before the search. Independence is what makes them dangerous, because differences on separate axes do not cancel each other out and can all point the same way at once. Any one of those rows can move a comparison by more than the 5.8 points that typically separate two models sitting several places apart on a retrieval leaderboard, and several rows moving together is how a published ordering ends up reversed on somebody's own material.

The weights involved are small enough to be worth stating as arithmetic. A single BEIR dataset contributes one-fifteenth of the retrieval column, since that column is the mean of fifteen per-dataset scores and a mean divides each contribution by the count. The retrieval column then contributes as one of the eight task categories the headline averages, so a single dataset's influence on the headline is one-eighth of that already small share. If exactly one of the fifteen resembled your material, a model twice as good as its rival on that one dataset would move the retrieval column by a fifteenth of the difference and the headline by an eighth of that again, so a large fact about the one dataset that matches you arrives at the top of the table shrunk twice over. Dilution is not a defect in the benchmark. It is the entire design goal: a benchmark averages many bodies of text so that no single one dominates the result, and you are the single one it was built to dilute.

The categories also close. Eight task categories account for all 58 datasets, with retrieval's fifteen among them and nothing left over, which is what makes the arithmetic above binding rather than indicative. There is no further column outside those eight where the dataset that matches your material might be hiding at full weight, so a reader cannot rescue the headline by supposing that some unaccounted-for part of it speaks to their case. Every dataset that could speak to it is already inside, at the fraction just computed.

One belief survives all of this and should not. Having learned that the headline mixes in categories you do not run, it is tempting to conclude that the fix is a filter — restrict to the retrieval column, and the number is now about retrieval, therefore about your retrieval. The first half is true and the filter is a genuine correction. The second half does not follow, and the reason is that the retrieval column is constant with respect to everything that would distinguish your case. It changes when the model changes. It does not change when your query style, your vocabulary or your language mix changes, because none of those are inputs to it. A check that returns the same answer whether or not the fault is present cannot detect the fault, and the retrieval column reads identically whether your corpus resembles all fifteen of those datasets or none of them.

Why query style reverses orderings#

Of those rows, query style is the one that reverses orderings on real corpora more than the rest combined, and the mechanism is worth being precise about. Retrieval encoders, meaning embedding models trained for search rather than for the classification or clustering the same architecture can serve, are trained contrastively on pairs, which means the model is shown a query alongside the passage that answers it and a batch of passages that do not, and is pushed to score the correct one higher than the rest. The overwhelming majority of the public pairs available for that training are question-and-answer shaped: a question paired with the passage that answers it, mined from the web, from StackExchange, from question-answering datasets.

Follow that through one link at a time. The training objective pushes a query towards the passage that answers it, so whatever the queries look like during training is the shape the query side of the model gets good at. Given a supply of pairs that is overwhelmingly interrogative, what the query encoder ends up holding is a very good mapping from interrogative English into the region of the space where answering passages live. Nothing in that process arranges anything for a query written in a register the training pairs did not contain. A user who types X-2210 no ship EU is handing that encoder something it saw comparatively little of, and the vector it produces is correspondingly less reliable — not wrong in any way an error would show, simply placed with less confidence. Models differ enormously in how much non-question data went into their mix: some were trained heavily on titles paired with bodies, which behaves much more like keyword search, and some were tuned hard on question-answering pairs because that is what the benchmark rewards. Two models whose averages differ by a couple of points can therefore differ by far more than that on keyword queries alone, in either direction, and the average has no way to report which.

Language mix produces the same kind of reversal on a different axis, and it is easier to miss because it hides inside an English-looking aggregate. A support operation whose tickets are a quarter Spanish, sometimes with both languages inside one ticket, is served by an English retrieval leaderboard that has nothing to say about the situation. The multilingual boards exist, and they average over 112 languages, which means a model can rank highly on the strength of languages you will never serve while being mediocre in the two you do. A model that handles Spanish well and code-switched Spanish-English badly will score identically on any benchmark that keeps its languages in separate datasets, because no public dataset contains a ticket that starts in Spanish and quotes an English error message.

The second most common reversal is domain vocabulary, and it fails in a way that looks like the model being stupid. Terms such as demurrage, FCL, Incoterms 2020 DAP and AWB are either rare or absent in general web pretraining, so the encoder has no strong representation for them and the vectors for two chunks that a domain expert would call unrelated end up close together because they share the same surrounding style of contract prose. Benchmarks with domain datasets exist — SciFact and FiQA are exactly that — but they cover science and finance, and a strong FiQA score tells you the model handles the vocabulary of financial forums rather than the vocabulary of your industry.

Tracing a number back to what it measured#

The leaderboard is backed by per-dataset results that are published alongside it, which means the drill-down is mechanical. Pull the per-task scores for the two models you care about, restrict to retrieval, and look at the datasets whose query distribution resembles yours rather than at the mean.

import json

# results/<model>/<revision>/<Task>.json as published in the MTEB results repo
RETRIEVAL = [
    "MSMARCO", "NQ", "HotpotQA", "FiQA2018", "SciFact", "TRECCOVID",
    "ArguAna", "QuoraRetrieval", "NFCorpus", "SCIDOCS", "Touche2020",
    "DBPedia", "FEVER", "ClimateFEVER", "CQADupstackRetrieval",
]
# datasets whose queries are short and keyword-ish rather than sentences
SHORT_QUERY = {"MSMARCO", "TRECCOVID", "DBPedia", "QuoraRetrieval", "NFCorpus"}


def per_task(model_dir: str) -> dict[str, float]:
    scores = {}
    for task in RETRIEVAL:
        with open(f"{model_dir}/{task}.json") as f:
            payload = json.load(f)
        scores[task] = payload["scores"]["test"][0]["ndcg_at_10"]
    return scores


a, b = per_task("results/bge-base-en-v1.5"), per_task("results/candidate")
for task in RETRIEVAL:
    marker = "*" if task in SHORT_QUERY else " "
    print(f"{marker} {task:24s} {a[task]:.3f}  {b[task]:.3f}  {b[task] - a[task]:+.3f}")

short = [t for t in RETRIEVAL if t in SHORT_QUERY]
print("short-query mean delta:",
      round(sum(b[t] - a[t] for t in short) / len(short), 3))

Running that on the logistics team's two candidates showed the candidate ahead on eleven of fifteen datasets and behind on three of the five short-query ones, with the overall lead carried mostly by ArguAna and HotpotQA — the two datasets least like a support search box. Those per-dataset numbers were public before the reindex started, and they point the other way from the aggregate on exactly the group that resembled the company's traffic: an overall lead built on long argument queries and multi-hop questions, alongside losses on most of the datasets whose queries are short. That drill-down took twenty minutes and would have been worth more than the two weeks of reindexing that followed.

Four questions get you from a published number to something you can reason about. Which task was it — retrieval, or an average that includes six tasks you do not run? Which corpus, and what do its queries look like next to yours? Which metric at which cutoff, since nDCG@10 rewards ordering within the top ten while your pipeline may only pass five chunks to the model and care about nothing below that? And which protocol, because BEIR scores come from exact search over the full corpus with no filters, while production applies a tenant filter and then searches an approximate index — a structure such as the HNSW graph named in the table above, which finds near neighbours by examining a small fraction of the corpus rather than all of it, at a setting such as ef_search that buys speed with a point or two of recall.

The metric question deserves more attention than it usually gets, because nDCG@10 and recall@5 reward different behaviour and a model can be ahead on one and behind on the other. nDCG applies a logarithmic discount by position, so moving a relevant document from rank 3 to rank 1 improves it materially even though both ranks are inside any reasonable cutoff. A pipeline that retrieves five chunks and concatenates them into a prompt does not care about the order at all, because the generator reads all five, which makes recall@5 the honest measure of whether the retriever did its job. In the logistics case the two metrics disagreed in sign on the question-shaped stratum: the new model ordered its results better and surfaced slightly fewer relevant documents inside the cutoff, which is a win on nDCG@10 and a loss on recall@5.

That last one is a persistent source of confusion when people compare a published score against a production dashboard and conclude the model underperforms its own paper. It does, by construction, and the construction is worth spelling out. A benchmark score is measured with exact search over an untouched corpus and with relevance judged by the people who built the dataset; production adds an approximate index, a metadata filter, chunks somebody else cut, and a definition of relevance written by whoever wrote the golden set. Every one of those subtracts. The published number is therefore an upper bound measured under conditions your serving stack does not reproduce, so the only fair use of it is comparative — model against model under identical conditions — and even that only holds if the conditions resemble yours.

What the ranking is still for#

None of this makes the leaderboard worthless, and the failure mode of over-correcting is teams who evaluate eleven models by hand because they no longer trust any published number. The ranking is a shortlisting instrument: it is very good at telling you which models are in the conversation at all, and the columns beside the score — embedding dimension, maximum sequence length, model size, licence — rule out more candidates faster than any experiment you could run. Those columns are worth more than the score for a specific reason: they are facts about the model rather than measurements taken on somebody else's text, so none of the averaging in this chapter applies to them. A model whose context window is 512 tokens is disqualified if your chunks run to 800. A model under a non-commercial licence is disqualified regardless of its score. A 4096-dimension model may be disqualified by the storage arithmetic before anyone measures its quality, which is a calculation taken up later in this topic.

What the ranking cannot do is decide, and the reason is structural rather than a matter of the benchmark being poorly built. The MTEB authors are explicit in the paper about the limits of aggregation and about the benchmark's skew towards English and towards the domains where public data exists. A benchmark measures the average of many corpora precisely so that no single corpus dominates, and you have exactly one corpus, which the average was designed to dilute. There is also the ordinary pressure of a public leaderboard: training sets increasingly include data adjacent to BEIR, and the gap between the top ten models is often smaller than the noise on any one dataset, so the ordering at the top is partly measuring who tuned closest to the test distribution.

Used that way, the leaderboard did its job for the logistics team on the second attempt. Filtering to models with a 512-token limit or better, a permissive licence, at most 1024 dimensions, and documented Spanish coverage took the field from several hundred entries to eleven, and sorting those eleven by retrieval score rather than by the overall average put four of them within two points of each other. Four models separated by less than the noise on a single BEIR dataset is the correct output of a leaderboard consultation, because it says the published evidence cannot distinguish them and the distinguishing has to happen somewhere else.

The number at the top of a public table is one average over dozens of datasets, and every property that would tell you whether a model suits your material — how your users phrase things, which words your documents use, which languages arrive, how you cut your chunks, what your judges counted as an answer, which measure you take and how your index is searched — was averaged out on the way to producing it. That is the benchmark working as designed rather than failing. Used as a filter it removes models that cannot meet your context length, your licence or your storage budget, and it narrows hundreds of candidates to a handful in an afternoon. Used as a decision it substitutes fifteen other people's retrieval problems for yours, weighted equally, and it will occasionally recommend the model that is worst on the axis you care about most.

The shortlist that comes out of this is two or three models, not one, and settling between them means running them against each other on your own material. Before that comparison can mean anything, there is an input most people do not know they are supposed to supply — a short string glued to the front of every query and every passage, different on each side, which several major model families require and which silently costs a model several points when it is missing. A bake-off that gives one candidate its prefixes and forgets the other's has already picked its winner.