Machine-translated eval sets

A machine-translated eval set is the English case set pushed through a translation API into every launch locale, run once, and scored with the English rubric. It costs two days of engineering and effectively no annotation money, which is why the plan is on a slide in front of you rather than being argued about, and the reason to think carefully before signing it is not that it produces bad data. It produces useful data. It also produces a number in the shape of a quality score, and that number will outlive every caveat attached to it.

The distinction that has to survive is between a smoke test — a check that the thing runs at all, rather than a measure of how well it runs — and a quality measurement. Everything below is about which failures fall on each side of that line, what the real thing costs per language per year, and how to allocate a budget that will not stretch to twelve.

What the translated set genuinely catches#

Dismissing the plan wholesale is both wrong and unpersuasive. A machine-translated set run once is a competent smoke test, and it will catch every failure mode in the following list:

• responses that come back empty, truncated at max_tokens, or in the wrong language entirely
• encoding and script breakage — mojibake, which is text decoded with the wrong character encoding so that accented letters arrive as runs of symbols, dropped diacritics, Thai rendered without word boundaries, right-to-left text mangled by a downstream template
• systematic refusals, where a locale declines a large fraction of ordinary requests because a guardrail interacts badly with the language
• structural non-compliance, since a deterministic check for named sections works on translated questions as well as on real ones
• entity loss across a translation boundary, since an assertion that product SKUs survive the inbound MT stage does not care where the question came from
• gross latency and cost regressions caused by the tokenizer charging some languages two or three times per character what it charges English

Those are the failures that make a locale unusable on day one, they are exactly what a launch smoke gate should be for, and they are cheap. As the lowest tier of a tiered per-language gate — the tier that reports and never blocks — the plan is a real endorsement rather than a grudging one.

Topics, phrasing and rubric all drift toward English#

The problem is the distance between a translated English question and a question a user in that market asks. Three separate effects push in the same direction.

The topics are wrong. An English support set is full of the questions English-market users ask about English-market circumstances. Brazilian users ask about failures in Pix, the country's instant bank-transfer scheme, and in boleto, its printable payment slip; German users ask about the statutory Widerrufsrecht, the fourteen-day right of withdrawal, and how it interacts with your refund policy; Japanese users ask questions shaped by 特定商取引法 disclosure requirements. Translating "how do I update my credit card" into twelve languages produces twelve versions of an English question, and the highest-volume, highest-risk questions in each market are simply absent from the set. You cannot fail a question you never asked.

That sentence carries the whole coverage argument, and it is worth stating as a mechanism. Translation is a function on the surface of a set, not on the sample the set represents. It changes which language each question is written in and changes nothing about which questions were chosen, so pushing an English set through twelve engines produces twelve sets sharing one coverage — the coverage of the English market. Every gap in the original is reproduced twelve times, and the gaps that matter are exactly the ones a Brazilian, German or Japanese user would have filled. No amount of translation quality repairs that, which is why dismissing the approach on translation-quality grounds misses the more serious objection to it.

The phrasing is wrong in a direction that inflates the score. Machine-translated text is cleaner than user text: it is grammatical, it is more explicit than a native speaker would be, it inherits English word order and English discourse structure, and it contains none of the abbreviation, code-switching, transliteration or regional slang that real traffic contains. The model finds it easier. On 150 topic-matched cases in one product, machine-translated questions scored 88% while native-authored questions on the same topics scored 71% — a seventeen-point gap that is entirely an artifact of who wrote the questions.

Seventeen points is 88 minus 71, and the direction it runs in has a mechanism rather than being luck. A model's advantage in English is partly an advantage on English-shaped text: English word order, English discourse structure, the question stated explicitly and up front. A translation changes the surface language and preserves that shape, so a translated question sits closer to the model's strongest distribution than a real question in the same language does. That also gives a reason to expect the bias to be worst where honesty matters most — the further a language sits from English, the further real traffic sits from a translated version of it, and the more the set is measuring a population the product does not serve.

The rubric is wrong. "Professional tone", "concise", "uses direct language" are English-calibrated instructions that describe a good English answer. Directness scores well in a German answer and badly in a Japanese one, and a rubric that rewards brevity will penalise the honorific framing a Japanese support answer requires. Grading with a translated rubric means measuring how English-like the answer is, and the more a locale needs to differ from the English original to be correct, the more the rubric punishes it for being right.

The row about the paired gate is the structural failure and it is independent of everything above. A set that runs once before launch produces no baseline, and without a baseline there is no before-and-after on fixed cases, which means a paired per-case regression gate cannot exist. The plan buys a launch check and buys nothing at all for the following two years of releases, which is where the actual risk lives, since the locale that works at launch is not the locale that works after the next prompt change.

The good news inside that is that the two objections are independent and only one of them is expensive. The bias belongs to the set and cannot be argued away. The missing baseline belongs to the word "once", and keeping the translated set and re-running it every release costs nothing beyond compute. What that buys is worth being precise about. A paired comparison scores the same fixed cases before and after, so whatever makes those cases optimistic makes them optimistic in both runs, and the bias cancels out of the difference. A set that is a bad absolute measure can be a perfectly good relative one. Run once, it is a smoke test and nothing more. Kept and re-run, it is a smoke test plus a real regression detector for the failure list above, which is most of what a small locale's gate was ever going to catch.

What per-language evaluation actually costs#

The honest answer to "what will evaluation cost" is a recurring annotation budget per language. Priced with contractors at roughly $45 per hour, for a 150-case set:

Every row is the same two multiplications. Authoring is 150 cases at twelve minutes each, which is eighteen hundred minutes, which is thirty hours; thirty hours at $45 is $1,350. Gold answers are the same 150 at eight minutes, which is twenty hours and $900. Adjudicating a fifth of the set is six hours and $270. Adding those three gives the $2,520, and multiplying that by the four funded languages gives the $10,080 in the budget further down. The per-release line multiplies along a different axis: $45 per release per language, across four languages and two releases a month for a year, is the $4,320 in the same budget.

The first three lines are the $2,520 that gets called "the eval set". The bottom four decide whether the set is still true in a year, and they are the lines cut in the budget review, because they have no launch date attached to them.

Two design choices make the recurring cost survivable. Human scoring of all 150 cases every release is 10 hours per language per release, which at two releases a month across four languages is 80 hours a month and will not be funded. Both figures are the adjudication rate applied to a larger denominator: at four minutes a case, 150 cases is six hundred minutes, which is ten hours, and ten hours across two releases and four languages is eighty hours a month, or half a person. Instead the full set is model-graded each release, and human eyes go only to the cases whose outcome changed, which is the same discordant set a paired gate already computes:

def needs_human_review(prev, curr, calibration_ids):
    changed = {cid for cid in curr
               if curr[cid]["pass"] != prev.get(cid, {}).get("pass")}
    low_conf = {cid for cid, r in curr.items() if r["judge_conf"] < 0.7}
    return sorted(changed | low_conf | set(calibration_ids))

A typical release produces 10 to 20 changed cases per language, so the per-release human cost is about an hour rather than ten — fifteen cases at four minutes apiece. The saving is not a discount on the work. It is the observation that a case scoring the same way in both runs carries no information about the release, which is the same reason a paired test discards it. The calibration identifiers are the second choice: a rotating sample of 40 cases per quarter gets native-speaker labels regardless of whether they changed, and the agreement between those labels and the judge's is what tells you the judge is still usable in that language.

Choosing the four#

Traffic share sets how much aggregate damage a failure does and risk sets how much damage a single failure does, and applied to a twelve-locale launch, three questions decide the allocation. Which locales carry enough projected volume that a failure is a business number rather than a support ticket. Which locales sit in regulated or legally distinctive markets, where a wrong answer about statutory rights is a different category of wrong. And which locales are furthest from English in pretraining coverage, since instruction following degrades steepest there and the model's own behaviour is least predictable — a locale can be small, low-risk, and still deserve a real set because you have no prior on what it will do. The per-language tables in the MMLU-ProX project page are a reasonable free input to that last question before you have data of your own.

The eight that do not get funded still get something: the machine-translated set as a smoke gate, the deterministic structural and entity checks, per-language dashboards, and a named date when their tier is reviewed against real traffic rather than projections.

Where the judge stops being usable#

The calibration sample exists because an LLM judge scores lower-resource languages generously, which means a green per-language number can be an artifact of the grader rather than of the product. Cohen's kappa on the calibration sample is the check — kappa being agreement between two sets of labels after discounting the agreement chance alone would produce — and below about 0.6 the judge's numbers for that language are decoration.

Discounting chance agreement is the part worth understanding, because raw agreement is reassuring for the wrong reason. If most cases pass, two labellers who both mark almost everything a pass will agree almost all the time while knowing nothing whatever about the cases — the agreement is a property of the pass rate, not of either labeller. Kappa subtracts the agreement that arrangement produces on its own and reports what is left over. That is why a judge can post high raw agreement in a language where it is useless, and why the calibration sample is scored on kappa rather than on the number that looks better.

Where kappa falls below that line, the gate falls back to human scoring or is downgraded to a smoke gate, and either answer is better than a gate that reports a number nobody can defend.

The eight unfunded locales carry a related risk that has nothing to do with statistics. A tier is a decision about where evidence is worth buying, and a decision with no review date is a permanent state. The review is what turns "we could not afford it at launch" into "we checked against real traffic and Polish is still tier 3", and without it the tiering slowly becomes the reason those locales never get looked at again.

A machine-translated set is therefore worth having and worth labelling. The labelling is the part that is genuinely hard, because it means shipping a launch artifact that says "no quality evidence" next to eight locales the company is about to sell into, and that sentence is more useful to everybody downstream than a percentage which cannot be defended when it is quoted back.

What this leaves is an operating loop rather than a launch checklist. Every request carries a language label derived from the text the user wrote. Every release runs the fixed set per language, model-graded, paired against the previous release, with a human hour on the cases that changed. Every locale has a tier, a gate and a named owner, and the tier is reviewed when traffic moves. Every incident starts by asking which stage produced the output, with the trace to answer it. None of that is a project that finishes; it is the shape of running a product in more than one language, and the first release where a gate stops a launch because one locale broke is the point at which the whole arrangement pays for itself.