An aggregate cannot name the tenant it is hiding

The escalation arrives through the account team rather than through any alert. An enterprise customer on a six-figure annual contract has been submitting German-language policy documents to the summarisation route since early May, and the answers have been confidently wrong for six weeks — right register, right structure, facts that do not appear anywhere in the documents they uploaded. Their internal champion stopped using it in week two and their team quietly went back to doing the work by hand.

Everything in the objective was working. The evaluator was pinned to the snapshot it was supposed to be pinned to, the template hash never changed, the sampler drew its 2% deterministically, the scored-sample volume was steady at about 3,100 a day, and the SLI read between 95.6% and 96.4% for the entire period. The number was correct. It was also, for this customer, completely uninformative, and no amount of care about how it was measured would have changed that.

The arithmetic of a small slice inside a large mean#

Eligible traffic runs about 420,000 requests a week. The customer in question sends about 3,400, which is 0.81% of the total. Before May their answers passed at roughly the fleet rate; from the second week of May their pass rate was 14%.

The aggregate moved 0.66 points. Week-to-week variation on that series had been running around ±0.4 points from ordinary mix shifts, so a 0.66-point step is at the edge of what anyone would notice on a chart and well inside what anyone would explain away. Total failure of a slice that is under one percent of traffic is arithmetically incapable of announcing itself in a fleet-wide number, and this is not a precision problem: estimating the aggregate perfectly, by scoring every single request, still yields 95.3%. Sampling error was never the issue.

The sample did contain the evidence. At 2% sampling, 3,400 weekly requests yield about 68 scored samples from that tenant per week, of which roughly 58 were failures. Across six weeks that is around 350 failed samples with rationales attached, sitting in the same table as everything else, describing the same problem over and over. Nobody read them because there was no view that put them together.

Dimensions are what make a number interrogable#

The repair is to attach to every scored sample the dimensions along which quality can plausibly differ, and then to look along them. In practice the set is small and mostly stable:

{
  "sample_id": "0f4c...",
  "trace_id": "b91a...",
  "verdict": "fail",
  "failure_reason": "unsupported_claim",
  "dims": {
    "tenant_id": "acct_7741",
    "tenant_tier": "enterprise",
    "language": "de-DE",
    "request_class": "document_summary",
    "route": "chat.summarize",
    "model": "claude-sonnet-4-5-20250929",
    "prompt_version": "summarize@v14",
    "retrieval_index": "docs-2025-04-28",
    "region": "eu-central-1",
    "input_tokens_bucket": "8k-16k"
  }
}

Each of those is a thing that can be true of some requests and not others, and each has caused a real quality failure somewhere. With them in place, the six weeks are one query away.

SELECT dims->>'language'      AS language,
       dims->>'request_class' AS class,
       count(*)               AS n,
       avg((verdict='pass')::int)::numeric(4,3) AS pass_rate
FROM quality_samples
WHERE scored_at > now() - interval '7 days'
GROUP BY 1, 2
HAVING count(*) >= 30
ORDER BY pass_rate ASC
LIMIT 10;

The result names the problem without anyone having formed a hypothesis first.

German summarisation is the outlier, German lookups are mildly affected, and everything else is normal, which is already a diagnosis in outline. The cause turned out to be in retrieval: the document index was created with an English analyser, so German compound nouns were never split — a query about Kündigungsfrist did not match a passage discussing Frist zur Kündigung — and recall, the share of the passages that should have been retrieved which actually were, fell on that tenant's corpus from about 0.81 to 0.22 after their May bulk upload rebuilt the index. The model, handed almost no relevant context, answered fluently from what it knew about insurance policies in general. The groundedness judge caught this correctly on 58 samples a week and filed each one in an aggregate that could not care.

Which slices get an objective and which get a scan#

The obvious next move is to put an SLO on every slice, and it does not survive contact with cardinality, meaning the number of distinct series the dimensions multiply out into. Twelve tenant tiers is not the problem; 340 individual tenants crossed with 6 languages, 5 request classes and 3 live prompt versions is 30,600 potential series, most of which will have fewer than ten samples in a week and none of which supports a threshold. Alerting on all of them guarantees that some are always breaching.

Two mechanisms cover the space between one aggregate and thirty thousand alerts, and they do different jobs.

The scan is the cheaper half and would have caught this in week one. Ranking by volume times delta rather than by raw pass rate matters, because raw pass rate puts a 31-sample slice with two failures at the top of the list every week and buries a 71-sample slice that fell from 0.95 to 0.18. Comparing each slice against its own trailing baseline rather than against the global target matters for the same reason: some slices are legitimately harder and sit at 0.89 permanently, and a list of things that are below target is a list of things everyone has already learned to ignore.

The tenant failure, in other words, was never an alerting problem. No threshold on the aggregate, however carefully tuned, could have fired, because the aggregate did not move enough to be distinguishable from a mix shift. What was missing was a dimension on the sample and a routine that looks along it.

The dimensions that pay for themselves#

Not all ten dimensions earn their place equally, and the ones that repay the effort first are the ones along which the system is genuinely heterogeneous. Request class is usually the highest-yield of them, because different classes exercise different code paths and have different intrinsic difficulty; splitting by class is what stops a mix shift from being mistaken for a regression, which is the defect chapter 2 opened with, and it costs nothing beyond a field that most routers already know.

Model and prompt version are the two that make a change safe to ship. A canary release, which sends a small share of live traffic to a new version while everything else stays on the old one, produces at 10% of traffic and a 2% sample rate about 840 scored samples a week under the new version and 7,560 under the old, which is enough to compare pass rates with a standard error — the typical gap between a figure computed from a sample and the true one — of roughly 0.7 points on the smaller arm. Without the dimension, the canary's contribution is a 10% dilution of the aggregate: a new model failing at twice the rate moves the fleet number by 0.4 points and is indistinguishable from a quiet week. With it, the canary has its own reading and the rollout decision has evidence behind it rather than an absence of complaints.

The input-size bucket is the one teams add last and wish they had added first. Quality degrades non-linearly with context length on most systems — the retrieval budget fills, the instruction sits further from the end of the prompt, the model's attention over a long middle section is weaker — and a slice view over input_tokens_bucket will typically show a clean staircase, with the 32k-plus bucket ten to twenty points below the 4k bucket. That staircase is not a bug to fix so much as a shape to know, because it tells you what happens to your quality number when a customer starts uploading larger documents, which is a thing customers do without asking.

Guaranteeing the sample contains the slice, and paying for it honestly#

Uniform sampling gives a slice the same share of samples as it has of traffic, which is fine at 0.81% and useless at 0.05%. A tenant sending 400 requests a week produces 8 samples at a 2% rate, and 8 samples cannot distinguish 100% from 50%. If a slice has a commitment attached to it, the sample has to be constructed rather than inherited.

That reallocation costs slightly less in judge calls than the uniform scheme it replaces and yields a slice view that can actually resolve a failure. It also breaks the aggregate, which is the part that gets forgotten: once strata are sampled at different rates, the unweighted mean of all scored samples over-represents small enterprise tenants by a factor of twenty and is no longer an estimate of anything. The fleet SLI has to be computed with inverse-probability weights, where each sample counts for the number of eligible requests it stands in for.

def fleet_sli(samples) -> float:
    num = den = 0.0
    for s in samples:
        w = s.stratum_eligible / s.stratum_sampled   # requests per sample
        den += w
        num += w * (s.verdict == "pass")
    return num / den

Getting this wrong produces a very specific kind of confusion in which the aggregate and the slice view disagree and nobody can say which is right. Publishing the weighted aggregate and the unweighted per-slice rates side by side, with the weighting stated on the dashboard, is the version that survives being questioned in a review.

What dimensions cost and what they are not#

Dimensions are cheap on a sampled table and expensive on a metrics backend. Two percent of 420,000 requests is 8,400 rows a week, and a relational store will group them by anything you like; the same ten dimensions pushed into a Prometheus histogram as labels is a combinatorial explosion that will take the scrape target down. The pattern that works is to keep the rich dimensionality in the sample store where queries are ad hoc, and to export only the handful of named slices as time series.

Some dimensions should not be recorded at all. tenant_id is an account identifier and belongs in the row; an end-user identifier does not, since the sample table also contains the question text and the answer, and joining a person to their queries turns a quality store into a surveillance store. Detected language is a guess with an error rate — a German document with an English cover page classifies as en often enough to matter — so the dimension is worth carrying as language_detected with its confidence rather than as a fact, and worth preferring the tenant's configured locale where one exists.

The dimensions also do not tell you why. The query above localised the failure to German summarisation in one afternoon and said nothing about analysers, compound nouns or recall; that came from opening the traces behind a dozen of those failing samples and looking at what retrieval returned. A slice view narrows the search from the whole system to one corner of it, and the trace is still where the mechanism lives.

There is a cost to having built all this, and it arrives in the next chapter. A dashboard with one number produces one thing that can cross a threshold; a dashboard with forty slices produces forty, each estimated from a smaller sample than the aggregate, and each therefore noisier. An alerting rule that was merely annoying on the aggregate becomes unworkable on slices, which is a problem worth understanding in its own right before adding a single alert.