Tracing a pivoted answer

A pivoted pipeline — detect the language, translate inbound, retrieve, generate, translate outbound — produces a specific kind of incident: an answer that is fluent, correctly formatted, carries a citation, contains a factual error, and raised no error anywhere in the stack. Every stage ran. Every stage returned something plausible. The customer acted on it.

Diagnosing that in a day depends on one property of that pipeline and one habit of investigation. The property is a trace: each stage logging its own input as well as its output, under a shared identifier, with the version of whatever produced it. The habit is to stop asking whether the answer was good and start asking, stage by stage, whether each one was correct for the input it was handed. Those two questions have different answers, and the second one localises the fault.

Why it localises is the mechanism the whole chapter rests on. A pipeline is a composition of functions, and end-to-end correctness is a property of the composition rather than of any member of it. Ask "did the customer get a good answer" of each stage in turn and every stage from the fault onward says no, because every one of them contributed to a bad final answer — the question cannot separate the stage that broke a contract from the stages that faithfully carried a broken value forward. Ask "was this correct for the input it was handed" and exactly one stage says no: the one whose own contract failed. The stages downstream did precisely what they promised, on values that were already wrong. One question returns a suffix of the pipeline. The other returns a point in it.

Replay every boundary with the input it really got#

Pull the actual payload that crossed each boundary for one conversation, not a reconstruction:

Rendered from the trace, that is a whole incident on one screen, and the shape of the diagnosis is already visible in the third line. What makes it usable is that each stage's input and output were logged separately:

{"trace":"c_44190","stage":"detect","out":{"lang":"ja","conf":0.99}}
{"trace":"c_44190","stage":"mt_in","engine":"mt-v4.2",
 "in":"ビジネス年間プランの返金期限は？",
 "out":"What is the refund deadline for an annual business plan?"}
{"trace":"c_44190","stage":"retrieve","embed":"emb-3-large","k":3,
 "in":"What is the refund deadline for an annual business plan?",
 "out":[{"id":"kb-0412","score":0.81,"title":"Refunds for annual plans"},
        {"id":"kb-0388","score":0.74,"title":"Cancelling a subscription"},
        {"id":"kb-0155","score":0.71,"title":"Billing FAQ"}]}
{"trace":"c_44190","stage":"generate","prompt_sha":"a91f0c","temp":0.2,
 "out":"You have 30 days from purchase to request a refund on an annual plan…"}
{"trace":"c_44190","stage":"mt_out","engine":"mt-v4.2",
 "out":"ご購入から30日以内に返金をご請求いただけます…"}

Two properties of that log are what make a one-day answer possible. Every stage records its input as well as its output, so you never have to assume that stage n's output was what stage n+1 received — in a pipeline with truncation, normalisation, or a cache in between, that assumption is exactly the one that costs you an afternoon. And each stage records the version of the thing that produced it: the MT engine build, the embedding model, the prompt hash, and the sampling temperature, which is the setting governing how much randomness the model is allowed as it picks each token. The MT engine is a hosted endpoint the vendor updates on their own schedule, so a replay run two days later against mt-v4.3 is not a replay, and without mt-v4.2 in the record you would not know that.

Judge each stage against its own input#

The discipline that finds the answer quickly is to refuse, for one pass, to ask whether the customer got a good answer.

Exactly one row says no. The Business tier's fourteen-day window lives in kb-0731, "Business tier billing terms", and kb-0731 was never a candidate, because after the inbound translation there was nothing left in the query that distinguished a Business plan from any other annual plan. ビジネス became "business" as an ordinary English adjective, which is a defensible translation of those characters and a destruction of the only token that identified the product.

That combination is what lets this class of failure survive every check anyone would think to apply. It is not a mistranslation. A translation-quality metric compares an output against a reference and asks whether the meaning was preserved, and the meaning was preserved — a Japanese speaker reading the English would recognise their own question in it. What was not preserved is a string a retriever needed to match exactly, and no translation metric has ever been asked about that. The engine optimised for the property it was built to optimise, and destroyed a property nobody had told it existed.

The confirming step is a counterfactual replay and it takes ten minutes. Substitute the corrected intermediate — "What is the refund deadline for the Business tier annual plan?" — and rerun everything downstream with the same pinned versions on a shadow pipeline, meaning a copy of the stack standing beside production, serving no users, with every component fixed at the build the trace recorded. kb-0731 comes back at rank 1 with a score of 0.86, the model answers fourteen days, and the outbound translation renders it correctly. Changing one stage's output fixes the outcome with every other stage untouched, which is as close to a controlled experiment as an incident gets, and it is the evidence that turns "the translation looks suspicious" into a finding you can put in a postmortem.

Replay has preconditions worth stating, since a replay that quietly diverges from the original run produces a confident wrong conclusion of its own. Sampling has to be pinned to the recorded temperature, and at temperature 0.2 a single rerun of the generation stage is not sufficient evidence on its own — five runs that all produce thirty days is. Caches are the subtler hazard: if retrieval results are cached on the English query string, the counterfactual query is a cache miss while the original was a hit, so you may be comparing a fresh index against a stale entry. The retrieval index version belongs in the trace alongside the embedding model for the same reason the MT build does, because a reindex between the incident and the investigation changes what "the same query" returns. Where a stage's dependency has moved and cannot be pinned, the stored output is the source of truth and the rerun is corroboration, never the other way round.

Why five runs settle it and one does not is worth a line. A single draw from a distribution is consistent with almost any distribution: the model might produce thirty days nine times in ten or one time in three, and one observation cannot separate those two worlds. Five identical draws are unlikely under the second and expected under the first, so they license the claim that thirty days is what this stage does with this input rather than what it happened to do once. It is the same argument as running an evaluation more than once, applied to a sample of one conversation.

No phrasing downstream of a deletion recovers it#

The instinct at this point is to add a line to the English system prompt: be careful about refund windows, and always state which tier the answer applies to. It is appealing because it is one line, it ships today, and it names the thing that went wrong.

It patches the stage the table above cleared. The generation stage received a question about a generic annual plan, was handed kb-0412 as grounding, and produced the correct refund window for a generic annual plan with a citation to the document it was given. Instructing it to be more careful gives it nothing extra to be careful with; instructing it to state the tier makes it state "annual plan", because that is the only tier information in its context. The word ビジネス stopped existing three stages earlier.

The general statement is worth having, because the same instinct arrives at the next incident wearing different terminology. A prompt is an input to a stage. It can change what that stage does with what it has; it cannot change what it has. Where an earlier stage removed information, every stage after it is a function of an input that no longer contains it, and no function of an input can produce something the input does not determine. A prompt helps upstream of the deletion, or it does not help.

There is a grain of truth in the instinct and it belongs in a different column. Making the answer state the entity it is about does not prevent this failure and it does make it visible — the customer reads "for annual plans" and can notice the mismatch, and an automated check can compare the entity named in the answer against the entities present in the user's original text. That is detection, and it is worth having, filed under monitoring rather than under fix.

Three layers that catch this class earlier#

The repair has three layers and they are worth building in this order, because the cheapest one covers most of the volume.

The do-not-translate glossary on the inbound MT stage gets the tier names in every locale spelling, including the katakana forms — katakana being the Japanese script used to write foreign words, so a product name has a spelling there as well as in Latin letters — and the mixed forms users actually type. That is a data change, not a code change.

A deterministic assertion on the translation boundary catches the ones the glossary has not learned yet. Product entities are a closed set, you know their surface forms per locale, and you can check that anything present on the way in is still present on the way out:

SKU_ALIASES = {
    "business_tier": {"ja": ["ビジネスプラン", "ビジネス年間プラン", "ビジネス"],
                      "en": ["Business tier", "Business plan"]},
    "pro_tier":      {"ja": ["プロプラン", "プロ"],
                      "en": ["Pro tier", "Pro plan"]},
}

def entities_in(text: str, lang: str) -> set[str]:
    return {sku for sku, forms in SKU_ALIASES.items()
            if any(f in text for f in forms.get(lang, []))}

def check_translation(src: str, src_lang: str, mt_out: str) -> list[str]:
    lost = entities_in(src, src_lang) - entities_in(mt_out, "en")
    return sorted(lost)

lost = check_translation("ビジネス年間プランの返金期限は？", "ja", mt_out)
if lost:
    mt_out = repair_with_glossary(mt_out, lost)   # or: route to native pipeline
    log.warn("mt_entity_loss", trace=trace_id, lost=lost, engine=MT_VERSION)

The assertion runs in about 40 microseconds, it fires on the exact case above, and the warning it emits is a metric — entity-loss rate per language per MT version — that will move the day the vendor ships a new engine build, which is the class of change that arrives without a diff of your own.

What makes that assertion possible is a property of the thing being checked rather than any cleverness in the check. Product entities are a closed set with known surface forms, so "was this entity present" is decidable: a membership test over strings, with a yes and a no and nothing in between. Set that against the invariant you would want for the register problem earlier in this topic, or for how a model interprets an ambiguous term. Neither has a closed set, neither has surface forms, and neither has a decidable test, so there is no assertion to write and the only instrument left is a person. The stages you can guard cheaply are the ones whose failure has a shape, and it is worth knowing which of yours those are before an incident rather than during one.

The third layer is a CI job that round-trips every SKU string through the translation boundary for every supported locale and asserts the canonical English form comes back. Twelve locales times fourteen product entities is 168 assertions, it runs in under a minute against the live MT endpoint, and it fails on vendor changes before those changes reach traffic. Regression cases go into the per-language fixed set at the same time — one case per tier per policy-bearing article — so a paired release gate blocks any change that reintroduces the flattening.

The multiplication matters more than the total. Every entity has to be checked in every locale separately, because an engine's behaviour on ビジネス says nothing about its behaviour on the Thai or Portuguese form of the same product name — those are independent facts about independent language pairs. That is also why the count grows along both axes: a thirteenth locale is fourteen new assertions and a fifteenth product tier is twelve. A suite that scales that way stays cheap while both counts are small, and it is worth generating from the alias table rather than writing out by hand.

None of this works without the logs#

Everything above depends on records that already existed at the moment of the incident. A pipeline that stores only the final response and the user's message supports none of it, and the fallback — rebuilding a shadow pipeline and hoping the vendor endpoint has not moved — is a day on its own, which is the whole budget. The instrumentation is the deliverable that makes the next incident an afternoon, and it has to be in place before the incident, which is why it never wins a prioritisation argument on its own merits and always wins one afterwards.

Stage isolation also stops at boundaries that are functions. It found this failure because translation is a function with a legible input, a legible output, and a checkable invariant: entities present on one side should be present on the other. It does not find failures that live inside a stage's judgement — an answer grounded in the right document and interpreted badly, or a refusal that was wrong for cultural reasons — and no amount of trace granularity turns those into a table with one row saying no.

What the method buys, where it applies, is the ability to name a stage instead of naming a language. "Japanese is worse" is not actionable and "inbound translation deletes product entities in 4% of tier questions" is, and the distance between those two sentences is one afternoon of reading logs that recorded each stage's own input.

The three-hour version of this investigation existed because there was a 150-case Japanese fixed set to generalise against. Without it the finding stops at one anecdote, and one anecdote does not tell a support policy team whether to email a class of customers. Which raises what those sets cost, and what happens when someone proposes producing twelve of them by running the English set through a translation API.