A locale at 2% of traffic can fail totally and move nothing

A retrieval dashboard with one headline number on it — recall@10, traffic-weighted across all fourteen locales, currently 0.841, with a threshold line at 0.80 that has not been crossed in five quarters — can sit above a locale whose retrieval, measured properly, is at 0.15. Both statements are true at the same time. They are not in tension, and the arithmetic that reconciles them takes one line.

That reconciliation is the subject of this chapter, and it matters because the conclusion generalises past the one locale. An average is an instrument with a known and computable sensitivity, and once you can compute it you can say in advance which failures it is capable of reporting and which it is not. Nothing about the dashboard needs to be broken for it to be silent, and no amount of tuning its threshold will make it speak.

The sensitivity of a weighted mean#

A traffic-weighted aggregate is the sum, over locales, of each locale's score multiplied by its share of traffic. Written that way, the question "how much does the aggregate move when one locale's score changes" has an immediate answer: every other term in the sum is untouched, so the change in the total is the change in that locale's score multiplied by that locale's weight, and nothing else. The sensitivity of a weighted mean to any one stratum — one slice of the traffic defined by a property you care about, here a single locale — is exactly that stratum's weight. That is a property of the arithmetic and not of the system being measured, which is what makes the table below mechanical rather than surprising, and worth putting in front of anyone who proposes a single-number quality gate.

Put Turkish through it. Turkish is 2% of query volume. If Turkish retrieval were as good as the median locale — call it 0.87 — the traffic-weighted aggregate would be 0.841 + 0.02 × (0.87 − 0.15) = 0.855. Complete, total, unusable failure of an entire locale is worth 1.4 points on the number the team is accountable for, which is inside the range the aggregate moves anyway when the query mix shifts between a quiet week and a release week. The dashboard is not broken and it is not lying. It is answering a question about the average session, and no one ever asked it a question about a locale.

Every entry in the middle column is the same shortfall — 0.87 down to 0.15 — multiplied by the weight on its left, and all six of them close. That is worth checking rather than reading past, because it means the column is not six measurements of six locales. It is one quantity multiplied by six numbers, so it describes the metric rather than the system, and it would read identically on a completely different product with the same traffic mix. Nothing about Estonian is being reported there except how little of the traffic it is.

Eight of the fourteen locales in this system are under 2%. Together they are 9% of traffic, and the same multiplication says that all eight could fail simultaneously and completely while the aggregate fell only by nine hundredths of the shortfall, landing at 0.776 — which would finally cross the 0.80 line, four months after the first of them broke, with no indication of which ones were involved. That is the strongest thing that can be said for the aggregate as an alarm.

A macro-average, where every locale counts once regardless of volume, is the reflex fix and it is a weak one. Thirteen locales at 0.87 and Turkish at 0.15, divided by fourteen, give a macro-average of 0.819, which is 0.05 below the healthy value and still comfortably above a 0.80 threshold. The sensitivity argument explains why in one step: under macro-averaging every locale's weight is one over the number of locales, so the failure is diluted by 14. That is better than dilution by fifty for a 2% locale, and it has its own defect — the weight shrinks as locales are added, so the metric loses sensitivity exactly as the number of ways to break it grows. Macro-averaging is genuinely better than traffic weighting for this purpose, and it is not enough, because any average over strata is designed to hide variance between them.

What actually detects a broken locale is a per-cell report where the worst cell is a first-class number. A minimum-over-locales metric with a floor — no locale below 0.60 — fails loudly the moment one does, and it is trivially gameable only in the sense that you have to fix the worst locale to make it pass, which is the intended behaviour.

The strata that matter#

Stratifying by locale is the first cut and not the last. Four dimensions carry most of the signal, and they interact, which is why they have to be crossed rather than reported one at a time.

Query language crossed with corpus language is the grid from chapter 3, and it is the primary stratification because those two dimensions produce genuinely different systems rather than variations of one. Collapsing a language's diagonal cell together with its off-diagonal cells produces a number that moves with corpus composition, which chapter 3 covered; the counting point here is that crossing 14 query languages with 5 corpus languages produces 70 cells from one query set, and most of the cells will be thin.

Script matters as a separate dimension because one language can arrive in more than one, and the retriever's behaviour differs sharply between them. Turkish users routinely type without diacritics, so görüşme kaydı arrives as gorusme kaydi, which tokenises differently and matches differently. Serbian arrives in Cyrillic and Latin from the same locale. Japanese queries arrive in kanji, in kana, and occasionally in romaji, and the three are not interchangeable for a subword tokeniser. Reporting one Turkish number over a mixture of diacritic and diacritic-free queries averages two populations that need different fixes.

Query type is the dimension people leave out, and it is the one most likely to inflate a locale's score. A support corpus receives navigational queries ("invoice download"), natural-language questions, and pasted error strings or codes. Error strings are language-neutral, and it is worth saying exactly why: ERR_MANDATE_REVOKED is the same byte sequence in every locale, so it tokenises identically, embeds identically and matches identically whether the surrounding sentence is English or Turkish. A query of that kind is not testing cross-lingual retrieval at all — it is testing exact-string retrieval, which works. If 35% of a locale's evaluation queries are error-string lookups, that locale's aggregate carries 35% of a subpopulation where the property under investigation is not being exercised. Splitting Turkish by query type on this system gave 0.62 for error strings and 0.09 for natural-language questions, and the pooled 0.15 was a description of the mix rather than of the system.

The fourth thing beside every cell is the count. A cell without an n is not a measurement, and the reason is the width of the confidence interval around a proportion — the range of true values that would plausibly produce what you observed, computed here with the Wilson method, which stays honest when the sample is small and the proportion sits near zero or one.

Read the third column as widths rather than as endpoints and it has a shape. The interval at thirty queries is a little under three tenths wide, at a hundred about a sixth, at four hundred under a tenth. Quadrupling the sample from a hundred to four hundred roughly halves the width, which is the inverse-square-root behaviour of a standard error showing through: the uncertainty falls with the square root of the count, so buying half the interval costs four times the queries and buying a quarter of it costs sixteen times. That exchange rate is what makes the last row expensive and the fourth row the practical ceiling for most teams.

A cell with nine queries can read 0.78 while the truth is 0.50, and it can read 0.44 while the truth is 0.80. Both directions cause damage: the optimistic reading closes the investigation, and the pessimistic one sends a team after a locale that is fine. The practical rule is a floor of 30 for "is this locale catastrophically broken", 100 for "does this locale meet the bar", and 400 before anyone compares two models within a cell — and cells below the floor render as their count rather than as a score, so nobody quotes them.

Producing the report#

from statsmodels.stats.proportion import proportion_confint

MIN_N = {"screen": 30, "gate": 100, "compare": 400}

def stratify(results, docs, queries, use="gate"):
    cells = {}
    for qid, hits in results.items():
        q = queries[qid]
        gold_lang = docs[q["gold_canonical"]]["lang"]
        key = (q["lang"], gold_lang, q["script"], q["type"])
        cells.setdefault(key, [0, 0])
        cells[key][1] += 1
        cells[key][0] += int(q["gold_canonical"] in
                             {docs[h]["canonical_id"] for h in hits[:10]})

    rows = []
    for key, (hit, n) in sorted(cells.items()):
        lo, hi = proportion_confint(hit, n, alpha=0.05, method="wilson")
        rows.append({
            "cell": key, "n": n,
            "recall_at_10": None if n < MIN_N[use] else round(hit / n, 3),
            "ci": (round(lo, 3), round(hi, 3)),
            "status": "insufficient" if n < MIN_N[use] else "reported",
        })
    return rows

Two details in that function do most of the work. The score is None rather than a number when the cell is thin, because a number in a cell will be read as evidence no matter what caveat sits next to it, and a None next to n=9 forces the reader to either collect more queries or say nothing. And membership is tested against the canonical ID of each hit rather than the document ID, which is the convention chapter 3 introduced; without it, serving the German rendering of the right article counts as a miss, and the report would blame retrieval for a rendering policy.

Where stratification does not save you#

You cannot stratify data you did not collect. This is the failure that turns a stratification initiative into three weeks of dashboard work with nothing at the end: the team slices the existing 480-query evaluation set fourteen ways, finds that eleven locales have fewer than 20 queries each, and concludes the report is inconclusive. Sampling has to be designed by stratum, which means deliberately oversampling small locales relative to their traffic — 120 Turkish queries when Turkish is 2% of volume is a 6× oversample, and it is the only way the Turkish cell ever becomes readable. The weights come back at the end if you want a traffic-weighted headline; they do not belong in the sampling.

The cost of that oversampling is the number to bring to the planning conversation, because it is smaller than people assume when they hear "build a per-language evaluation set". Reusing the English information needs and having a native speaker rewrite the query, as chapter 2 described, produces 120 queries for a locale in about six hours, and judging the pooled top-10 from two retrievers over those 120 queries is another day. Call it two person-days per locale, once, plus a half-day per quarter to refresh. Fourteen locales is under six person-weeks for the initial build, spread across contractors who already exist because someone is paying for translation anyway. Compare that against the four months a failure of this kind runs undetected.

Stratified evaluation is also not production telemetry, and the two answer different questions. The evaluation set tells you whether the system can find the right document for a need you have judged; telemetry tells you what users are actually asking, which drifts, and which is where next quarter's evaluation queries come from. A locale can pass a stratified eval built from last year's tickets and fail in production because the query distribution changed after a UI redesign. Running both, and periodically refreshing the eval set from live logs for each locale separately, is what keeps the two in contact.

And the aggregate is not worthless. The sensitivity argument that condemns it for one locale is exactly what recommends it for a global regression — a bad index build, a model swap gone wrong, an embedding-dimension mismatch — where every stratum moves together, so every weight contributes in the same direction and the pooled number moves fastest because it has the most data behind it. Keeping it as a smoke alarm while gating releases on the per-cell report costs nothing and gives each number the job it can do.

From a defensible number to a cause#

The remaining limitation is one that no amount of stratification removes: the cells are computed from judgments you wrote, and a stratum whose judgments are systematically wrong — chapter 2's country-specific articles, for instance — will report a confident, well-bounded, incorrect number. The confidence interval quantifies sampling error and says nothing about judgment error.

An aggregate over locales, then, has a sensitivity you can compute before you trust it, and for a small locale that sensitivity is small enough that no alert threshold will ever be crossed by anything short of a global outage. The repair is not a better threshold or a cleverer average, because every average over strata is built to suppress the variance between them. It is a report where each cell stands alone, the worst cell is the one the release gates on, and every cell carries the count that says whether it is evidence or decoration.

Turkish natural-language questions against the English corpus retrieve the right article 9% of the time, with 118 queries behind that and an interval tight enough to act on. That is a defensible finding and it is not yet a cause, and the next step is unusual in that the cause turns out to sit upstream of anything discussed so far — before the encoder, before the index, in the two lines of code that lower-case the query and split it into tokens.