Aggregating quality across locales

A locale is a language paired with a region — vi-VN is Vietnamese as written and regulated in Vietnam, pt-BR is Portuguese in Brazil — and a product sold into nine markets typically serves all nine from one assistant, one set of prompts, and one large language model (LLM) underneath. Task success is the verdict on a single conversation, pass or fail against a rubric, averaged into the percentage that appears on a quality slide. Combining those verdicts across locales assumes that a request is a request: that a Vietnamese conversation and an American English conversation are two draws from the same process, so averaging over them loses nothing anybody would have acted on.

That assumption is false, and the name for why it is false is multilingual behaviour — how a model's quality moves with the language of the text in front of it. Same weights, same prompt, same task, different success rate depending on whether the user typed English or Vietnamese. It exists because a model does not learn languages in equal measure: pretraining text is scraped from a web that is overwhelmingly English, and the instruction-following and safety data layered on top of it is more English-dominated still, so English and German arrive with orders of magnitude more coverage than Vietnamese or Swahili. The languages at the thin end are called lower-resource, which is a claim about how much of that language the model was trained on rather than about how many people speak it — Bengali has roughly 240 million speakers and is lower-resource, Norwegian has five million and is not. What follows from that is a measurement problem, and per-language evaluation — computing, publishing and gating on a separate quality number for every language you serve — is the subject of this topic. You meet it the first time one assistant is sold into more than one market: a language picker in the settings, a store listing in twelve countries, a support queue arriving in a language nobody on the team reads.

What a weighted mean can and cannot move#

The averaging is worth working out with actual numbers rather than describing, because the size of the effect is what makes it dangerous. Traffic is not evenly split across locales, and it never is in a real product.

Weight each rate by its share and the headline comes out at 88.1%. Now break Vietnamese completely — not degrade it, break it, so that four answers in five are wrong, unusable, or in the wrong language entirely — and recompute.

locales = [
    ("en-US", 0.38, 0.91), ("de-DE", 0.12, 0.88), ("fr-FR", 0.11, 0.87),
    ("en-GB", 0.09, 0.90), ("es-ES", 0.08, 0.86), ("pt-BR", 0.07, 0.85),
    ("ja-JP", 0.06, 0.83), ("pl-PL", 0.05, 0.82), ("vi-VN", 0.04, 0.85),
]

def headline(rows):
    return sum(share * rate for _, share, rate in rows)

broken = [(c, s, 0.20 if c == "vi-VN" else r) for c, s, r in locales]

print(f"{headline(locales):.3f}")   # 0.881
print(f"{headline(broken):.3f}")    # 0.855

The headline moves from 88.1% to 85.5%. Vietnamese users went from mostly served to mostly failed, and the number that governs the release moved 2.6 points — a move the same size as ones a weekly aggregate makes several times a quarter for reasons nobody investigates, because a 2- or 3-point wobble is what a weekly aggregate does.

Both of those figures are one operation repeated. A traffic-weighted mean multiplies each locale's success rate by its share of traffic and adds the products up: American English contributes 91% of thirty-eight hundredths, German 88% of twelve hundredths, and so on down to Vietnamese contributing 85% of four hundredths. Nine products, added, give 88.1%. Multiplying by the share is the right operation because a share is the probability that a request drawn at random came from that locale, so the sum is the probability that a request drawn at random succeeded.

The move to 85.5% is simpler still, and it is the arithmetic worth carrying out of this chapter. Only one term in that sum changed, so only that term's change can reach the total. Vietnamese fell from 85% to 20%, a drop of sixty-five points, and sixty-five points arriving through a weight of four hundredths is sixty-five multiplied by four hundredths — a little over two and a half points. That is the whole of the 2.6. The headline's movement is the weight times the change in the rate, and it can never be anything larger.

Proportional sampling puts the power where the risk is not#

The 80 Vietnamese cases in that sample are not a budget accident. They are what you get when the eval set is drawn from production traffic in proportion to volume, which is the natural default because it makes the set an unbiased picture of the experience you are actually serving. It also allocates statistical power in almost exact inverse proportion to where the uncertainty lives: English, the locale everyone has the most evidence about and the most confidence in, receives 760 cases, and Vietnamese, the locale nobody can say anything about, receives 80.

Equal allocation is the obvious alternative and it has a cost people underestimate. Two thousand cases split nine ways gives 222 per locale, which is enough to put a usable bound on every one of them. The aggregate then has to be reweighted by traffic share before it means anything, since the raw pass rate over an equally-allocated set describes a product where Vietnamese and American English have identical volume, and that product does not exist. The reweighting is two lines of code and a permanent reporting hazard, because two numbers now exist for the same run and someone will paste the wrong one into a summary.

Square-root allocation is the compromise that survives contact with a real team. Give each locale a share of the sample proportional to the square root of its traffic share, and the tail gets a workable sample while the head keeps enough cases to stay precise:

That last column is built in three steps. Take the square root of each locale's traffic share. Add the nine roots together. Give each locale its own root divided by that total, as its share of the 2,000-case budget. What the square root is doing is compressing the range: American English and Vietnamese differ by nearly tenfold in traffic and their square roots differ by about threefold, so the head keeps far more than an equal split would leave it while the tail gets far more than proportional would. The reason a square root rather than some other compression is that the precision of a measured rate improves with the square root of the number of cases behind it, so spending the budget on roots spreads precision more evenly than spending it on shares does.

Vietnamese goes from 80 cases to 142 for the same annotation spend, English drops from 760 to 439 and loses almost nothing that matters, since the width of an interval at 439 cases is already well inside the precision anyone acts on. The reweighting hazard remains, so whichever allocation you choose, the discipline is to compute the traffic-weighted headline from the per-locale rates rather than from the raw set, and to keep the per-locale sample sizes visible next to every rate you publish.

The threshold that would have caught this does not exist#

The reflex, once someone sees this arithmetic, is to keep the metric and put an alert on it. Set the gate at 85% and the broken week clears it at 85.5%. Set it at 86% and the broken week fails, along with a fair number of healthy weeks, because 86 is inside the band the metric already occupies. Set a relative rule instead — fire if the headline drops more than two points week over week — and you catch this particular collapse only if it arrives all at once. Vietnamese quality rarely arrives all at once; a prompt change ships, one locale degrades over three weeks as it rolls out by percentage, and each week's move is under a point.

The reflex is not careless, and the reason it is so strong is worth naming. In nearly every other kind of monitoring a threshold is the right instrument, because the metric being watched actually contains the failure. And the arithmetic above did move the headline, so the failure looks like something the metric registered. It did register it — as two and a half points, in a series that makes two-and-a-half-point moves for unrelated reasons that were never investigated. A threshold cannot separate a signal from noise of its own size, and no sophistication in the detector changes that, because the separation was lost before the detector ran.

The deeper problem is that the aggregate moves for reasons that have nothing to do with quality. Suppose nothing about the model, the prompt, or the corpus changes, and a marketing campaign in Brazil pushes pt-BR from 7% to 12% of traffic while en-US falls from 38% to 33%. Every per-locale rate in the table stays exactly where it was.

shifted = [
    ("en-US", 0.33, 0.91), ("de-DE", 0.12, 0.88), ("fr-FR", 0.11, 0.87),
    ("en-GB", 0.09, 0.90), ("es-ES", 0.08, 0.86), ("pt-BR", 0.12, 0.85),
    ("ja-JP", 0.06, 0.83), ("pl-PL", 0.05, 0.82), ("vi-VN", 0.04, 0.85),
]
print(f"{headline(shifted):.3f}")   # 0.878

The headline falls 0.3 points because more of the sample now comes from a locale that was always slightly worse.

That is one subtraction and one multiplication. Five points of traffic share moved from American English, which succeeds at 91%, to Brazilian Portuguese, which succeeds at 85%. The two rates differ by six points. Five hundredths of the traffic changed which of those two rates it would be scored under, so the headline changes by five hundredths of six points, which is three tenths. Notice what is not in that calculation: nothing about either locale's behaviour, since neither rate moved. The only inputs are how much traffic moved and how far apart the two rates it moved between were.

A team watching only the aggregate sees a small unexplained decline and goes looking for a regression that does not exist, and the same mechanism runs in reverse: growth in a strong locale can mask a real decline in a weak one, so the metric that is supposed to be your alarm is also a source of false alarms and false all-clears in roughly equal measure. Any threshold you set is being applied to a quantity that is part quality and part traffic mix, and the two are not separable after the weighting.

There is a general form here worth stating plainly, because it applies well beyond language. A weighted mean bounds the influence of any component by its weight. A locale at share w can move the aggregate by at most w × (its current rate), so the ceiling on Vietnamese's influence at 4% traffic and 85% success is 3.4 points, and that is the number you get only in the case where Vietnamese goes to zero — every answer wrong, every time. Any smaller catastrophe moves it less. The 3.4 is the same multiplication as before with the largest change that can be put into it: if a rate can fall by at most its whole current value, from 85% to nothing, then the biggest product available at a weight of four hundredths is four hundredths of eighty-five points, which is three and two fifths. No arrangement of a weighted mean lets a locale at that weight move the total further, which is what makes this a ceiling rather than an estimate. It is not what usually happens. It is the worst that can.

Meanwhile the noise floor of the aggregate is set by the total sample size and by mix drift, and for a 2,000-case weekly sample it sits comfortably above two points. When a component's maximum possible influence is below the metric's noise floor, that component is invisible to the metric, and no threshold, alert, anomaly detector, or change-point model applied to the aggregate can recover it. The information was destroyed at the point of averaging. That comparison — a locale's traffic share against the aggregate's noise floor — is the arithmetic worth doing before you trust any cross-language number. In this product, five of the nine locales sit below the floor. In a product with thirty locales, twenty-five will.

Saying the information was destroyed by the averaging is precise rather than rhetorical, and the precision is what closes off the search for a cleverer detector. A weighted mean maps nine numbers onto one, and the map is many-to-one: an enormous set of different per-locale vectors all produce a headline of 88.1%, the healthy one and a great many unhealthy ones among them. Given the output alone, no operation recovers which input produced it. Anomaly detectors, change-point models and alert thresholds are all functions of that output, and no function of an output can distinguish inputs the output does not distinguish.

Nine numbers instead of one#

The change that follows is to stop computing one number and start computing nine, which in most stacks is a GROUP BY and an afternoon:

SELECT
  language,
  count(*)                                        AS n,
  avg(succeeded::int)                             AS success_rate,
  count(*) FILTER (WHERE NOT succeeded)           AS failures
FROM eval_results
WHERE run_id = :run_id
GROUP BY language
ORDER BY n DESC;

The afternoon goes on the language column, which has to exist on every row, populated at request time and carried through the eval pipeline. In most systems it does not exist yet, or exists as something that only resembles a language. Whatever value you put there decides which bucket every failure lands in, so a wrong value does not produce a missing number, it produces a confident wrong one. That is the whole of the next chapter, on where a language label actually comes from.

The unlabelled bucket is the interesting row#

When that query first runs against real data, there is almost always an unknown bucket holding something like 6% of traffic, and it is the most interesting row in the result. Unlabelled traffic is not randomly distributed — it concentrates in exactly the clients, regions, and integrations that the rest of your instrumentation also misses. Sorting by n DESC and reading the top three rows is how that bucket stays unread for a year.

That concentration has a mechanism and is not bad luck. A row is unlabelled because the code path that sets the label did not run for it. The paths that skip it are old client versions shipped before the field existed, third-party integrations that call the API directly and never send the header, and server-side jobs that build a request with no user session attached. Which markets are served through partner integrations and older clients is not independent of which markets are small and recently added. So the unknown bucket is enriched in precisely the locales you have least evidence about, and a habit of reading the top rows by count guarantees the enrichment is never seen.

Nine numbers on a dashboard also change nothing on their own. A dashboard is a thing someone has to look at, and the failure mode of per-language dashboards is not that people disagree with them, it is that nobody opens the tab. The number for pl-PL sat at 82% for a quarter in the table above and no one acted on it, which is a plausible outcome even after the reporting exists. Reporting makes a collapse visible to someone who goes looking. A gate makes a collapse block the release whether or not anyone goes looking, and those are different pieces of machinery with different failure modes.

The gate is where the obvious objection lands, and the objection is not stupid. Vietnamese was 80 cases in that sample. A rate computed on 80 cases moves by more than a point when four cases flip, so a naive per-language gate at 85% on a small locale will fail releases for reasons that have nothing to do with the release. Anyone who has watched a flaky test block a deploy knows exactly how that ends, which is with the gate switched off and a message explaining that it was too noisy.

The granularity is worth making concrete, because it sets the whole design of a per-language gate. With 80 cases, one case is one eightieth of the rate, which is a point and a quarter, and the rate can only ever land on multiples of that step. Four cases flipping is therefore five points. Five points is larger than the distance the gate is being asked to police, so the question actually being put to it is not whether it can detect a five-point regression. It is whether it can tell a five-point regression apart from four coins landing the same way, on evidence that reports in steps of a point and a quarter.

A weighted mean is a fine number for the question it answers, which is roughly "what fraction of the requests we served last week went well" — a capacity and cost question, useful for forecasting support load, useless as a release gate for any individual locale. The failure is not the metric, it is the substitution of one question for another, and the substitution is invisible because both questions are about quality and both produce a percentage.

So the reporting change on its own leaves two things unfinished. It needs labels that describe the language of the text rather than something adjacent to it, and it needs a gating design that can tell a real 10-point regression in a small locale apart from 80 coin flips landing badly. The labels come first, because a gate on a mislabelled locale fails in the one way that is worse than not gating at all: it produces a green number that someone will cite.