Repair rate is a quality signal, and an average

Repair incidence — the share of requests that needed at least one repair attempt — is cheap to emit and it turns a cost that was hidden inside latency and token spend into a number somebody can watch. It is the right first metric for the loop this topic has been building, and it comes with an arithmetic property that decides how much it can ever tell you.

That property is that a rate over a mixed population is a weighted average, and the weights are volume shares. A cohort that is small in volume cannot move the aggregate no matter how badly it is doing, so a threshold placed on the aggregate is not a threshold on quality; it is a threshold on quality-weighted-by-size. This chapter is about the segmentation that makes the number usable, the instrumentation that makes segmentation possible after the fact, and the two other things people mean when they say "repair rate".

What a rate arithmetically cannot do#

Here is one month of an extraction service, broken down by tenant — one customer account among the many sharing the same pipeline, the same models and the same dashboard:

aggregate = Σ (docsᵢ / docs_total) × rateᵢ

Apply that once to a row and the last column stops being decoration. Acme's share of the documents is 1,620,000 out of 2,112,000, a little over three quarters. Multiply that share by Acme's own 3.1% and the result is the 2.39 points the table gives. Do the same for each row and the five contributions add up to 4.22, which is the aggregate — not a separate measurement taken of the fleet, but exactly this sum and nothing else. That is the entire content of the claim that a rate over a heterogeneous population is a weighted average, and it is worth having in the concrete form, because the abstract form is easy to agree with and hard to act on.

Nordwind's weight is 12,000 / 2,112,000, which is 0.57%. Their rate can swing across its entire possible range, from 0% to 100%, and the most it can move the aggregate is 0.57 percentage points. Turn their failure rate up to 100% — every single document unusable, the worst outcome the system can produce for a customer — and the aggregate moves from 4.22% to 4.50%, which on a graph with one-point gridlines is not a visible change. No threshold placed on the aggregate can be sensitive to them, because the signal they are capable of producing is smaller than the noise of Acme having a slow week. This is not a tuning problem that a better threshold solves; it is a property of the arithmetic, and it applies to every aggregate rate in every system, which is why the useful question about any percentage on a dashboard is what population it is averaging over.

The belief that keeps aggregate alerting in place is that a threshold is a dial whose sensitivity you tune — too many pages, raise it; too few, lower it. That is true of a threshold on a single well-behaved series and it is false here, for a reason no amount of tuning reaches. Nordwind's largest possible signal is their weight, 0.57 points. A threshold sensitive enough to fire on 0.57 points would also fire on an ordinary wobble in Acme's own rate, since Acme carries three quarters of the weight and a swing of that size in a tenant that large moves the aggregate by nearly as much. So the sensitive setting pages constantly and gets muted within a month, and a muted alert is worse than no alert, because it still reads to everyone else as a box that has been ticked. The insensitive setting is the one they had. There is no value between the two that works, and that is the tell: when both ends of a dial are wrong, the problem is the series being measured rather than the number on the dial.

The practical consequence is that alerting on an aggregate rate detects exactly one thing: a problem large enough to affect your biggest tenant. That is worth detecting and it is not what anyone believes the panel is for. Detecting a cohort-level failure requires evaluating the rate per cohort and alerting on the worst one, or on the count of cohorts above a threshold, which is a different query against different data.

What to slice, and what each slice is for#

Five dimensions carry nearly all of the diagnostic value, and each answers a question the aggregate cannot. Cardinality, in the second column, is the number of distinct values a dimension can take, and it is the property that decides whether the dimension can live on a metric at all.

Tenant is the dimension that catches a Nordwind, and it is also the one most often left out, because it is the one that breaks a metrics backend. Multiplying it out — 400 tenants by 4 model versions by 12 prompt versions by 6 schema versions by 6 input classes — gives 691,200 possible label combinations, and a Prometheus-style time-series database, which keeps a separate stored series for every distinct combination of label values it has ever seen, will fall over long before that. The instinct to drop tenant is a real operational constraint rather than carelessness.

That multiplication deserves a pause, because the number it produces surprises people who have added labels one at a time and seen nothing happen. Labels combine independently — any tenant can appear alongside any model version, any prompt version, any schema version and any input class — so the count of distinct series is the product of the five figures and not their sum. Adding a fifth dimension does not add a fifth to the cost; it multiplies the cost by that dimension's size. What makes the product the governing quantity, rather than traffic, is what the database stores: one series per distinct label combination it has ever seen, each with its own index entry and its own retained samples. Ten times the traffic through the same labels costs roughly the same. The same traffic through one extra high-cardinality label can cost orders of magnitude more. Tenant is that label, which is why dropping it is a correct local decision that produces a system unable to see its worst-served customer.

The way through is to stop treating this as one metric. Emit a low-cardinality counter for the dashboard and the page, and a structured event per response for the analysis:

# Dashboard + alerting: bounded label set, safe as a time series.
REPAIRS = Counter(
    "extraction_repairs_total",
    "Responses that required at least one repair attempt",
    ["model", "prompt_version", "outcome"],   # ~200 series
)

# Analysis: one row per response, tenant and input class included,
# aggregated at query time rather than at write time.
log.info(
    "extraction_result",
    extra={
        "tenant_id": ctx.tenant_id,
        "model": ctx.model,
        "prompt_version": ctx.prompt_version,
        "schema_version": ctx.schema_version,
        "input_class": classify(doc),      # scanned_ocr | digital_pdf | email_body
        "page_count": doc.pages,
        "attempts": attempts,
        "outcome": outcome,                # accepted | repaired | exhausted | escalated
        "error_types": [e["type"] for e in errors],
        "input_tokens": tokens_in,
    },
)

At 2.1 million responses a month that is about 70,000 events a day, a few hundred megabytes with compression, and a query that groups by tenant and runs in seconds. The daily job worth building on top of it takes one line of logic — for every tenant with more than 200 documents in the window, compute the rate, and page if any of them exceeds 25%.

The error_types array is the field that repays itself most, because it turns the next chapter's diagnostic work from an investigation into a query. Keeping it means keeping the validator's own classification, which chapter 1 already established is the thing worth propagating rather than flattening.

Small denominators bite from the other side#

Splitting the rate finely enough to see a small tenant creates the opposite problem, which is that small cohorts produce noisy rates and a per-cohort alert that fires on noise gets muted within a fortnight. The arithmetic is worth doing once so that the threshold is chosen rather than guessed. For a cohort of n documents at a true rate of 5%, the standard error — how far a measured rate is expected to wander from the true one by chance alone — is sqrt(0.05 × 0.95 / n), so a tenant sending 400 documents in the window has a 95% interval of roughly 3% to 7%, a tenant sending 40 has an interval of roughly 0% to 12%, and a tenant sending 12 has essentially no measurable rate at all — one failure puts them at 8.3% and two put them at 16.7%.

Two steps sit under that formula and neither is optional if the threshold is going to be argued about. The first is why a rate has a standard error at all: a cohort's documents in a window are a sample, each one either needing a repair or not, and even with the underlying rate perfectly fixed, next month's documents would produce a different count purely by which documents turned up. The formula measures how far that count is expected to wander. The second step is the move from a standard error to an interval, which is a doubling: about nineteen samples in twenty land within two standard errors of the truth, so two of them either side of the measured rate gives the band you should be unsurprised by. For four hundred documents at a true rate of 5% the standard error works out near a point, two of them either side is about two points either side, and that is where "roughly 3% to 7%" comes from. Shrinking the cohort by a factor of ten widens the band by a bit over three rather than by ten, because the count sits underneath a square root. That relationship is the one to carry away: halving the width of a band costs four times the observations, so no choice of threshold rescues a cohort that has simply not sent enough documents to have a measurable rate.

That is where the "more than 200 documents in the window" filter in the daily job comes from. Below a couple of hundred observations, a per-tenant rate is mostly a statement about which side of a coin came up, and alerting on it generates pages that resolve themselves. The tenants below that floor are not unwatchable; they need a longer window, a weekly rather than daily evaluation, or grouping into a cohort by some shared property such as document language or industry, so that the group has enough volume to say something. A tenant sending 12,000 documents a month is 400 a day and comfortably above the floor, which is why a per-tenant job would have caught the case below on its first bad day rather than in a support ticket, and why the aggregate did not.

Choosing the alert threshold follows from the same arithmetic. A fixed 25% cutoff is crude, and it has the advantage of being obviously interpretable in a support conversation. The more sensitive version compares each cohort against the aggregate rather than against a constant — page when a cohort's lower confidence bound, the pessimistic end of that interval, sits above three times the fleet-wide rate — which catches a tenant sitting at 14% when everyone else is at 4%, a case that a 25% cutoff sleeps through.

One operational detail decides whether any of this works when you need it: the events have to exist before the incident. A per-response log with prompt_version and schema_version stamped at request time is trivial to add and impossible to backfill, because a version inferred afterwards from deploy timestamps will be wrong for every request in flight during a rollout, and those are frequently the requests you are trying to explain. Ninety days of retention on these events costs a few tens of gigabytes and is what makes it possible to answer questions about a change that shipped six weeks ago.

Three numbers that all get called "repair rate"#

Ambiguity in the definition causes more confusion than the segmentation problem does, because two people can look at the same panel and mean different things by 4%.

They can move in opposite directions, which is what makes conflating them expensive. Tightening a schema in that service pushed repair incidence from 4.2% to 5.1% and pushed exhaustion from 0.31% to 0.09%, because the constraint that was added made the model's mistakes more legible to the validator and therefore more repairable. Reported as "repair rate went up", that change looks like a regression and gets reverted. Reported as three numbers, it is obviously an improvement: slightly more requests needed a second attempt, and under a third as many users ended up with nothing.

That pairing repays slowing down on, because "add a constraint, get fewer bad outcomes" runs against most people's expectation and the mechanism is not self-evident. Before the change, a mistake in the tightened field produced a value the schema accepted. It was not a failure, so it was not repaired and not counted; it went downstream as a wrong value. After the change the same mistake produces a validation error, which is a first-attempt failure and pushes incidence up. It is also an error carrying a field path and a rule, which is precisely the feedback a repair loop acts on well, so most of the newly-created failures recover on the second attempt and never reach the ceiling. The constraint converted a silent wrong value into a loud repairable one, and the two metrics record the two halves of that conversion in opposite directions: incidence up by under a point, exhaustion down from 0.31% to 0.09%, which is under a third of what it was. Only the first of those movements appears on a panel labelled "repair rate", and only the second corresponds to something a user noticed.

Exhaustion rate is the one to put in front of anyone outside engineering, because it is the only one of the three that maps to something a customer experiences. Repair incidence and attempt ratio are internal quality and cost measures, and a product manager reading a 5% repair rate as "5% of results are wrong" will make decisions on it that nobody intended.

A low rate is not automatically the good outcome#

The rate measures how often responses fail your checks, so it moves when either the responses or the checks change, and the checks are the half that is under your control. A team on a neighbouring service reported a repair rate of 0.1% and were pleased with it. Their schema had every field optional, additionalProperties left open, and no enum on the classification field, so almost nothing the model produced could fail — the responses were not better, the question being asked of them was easier. An audit against labelled documents put their field-level accuracy at 71%, sitting underneath a metric that read 99.9%.

Follow what that schema asserts and the 0.1% stops being surprising. A validator decides whether a document belongs to the set the schema describes, so the rate at which it rejects things is a property of how small that set is. Making every field optional deletes every missing failure. Leaving additionalProperties open deletes every unexpected-key failure. Dropping the enum on the classification field deletes every literal_error. What remains in the reject set is documents that are not JSON at all, or that put a list where an object belongs, and a model asked for a schema-shaped response almost never produces either. The rate did not fall because the responses improved. It fell because the population of documents capable of failing had been emptied, and a measurement whose numerator is produced by your own strictness moves whenever that strictness does, in the direction that flatters it.

That direction of error is harder to see than a rising rate, because nothing about it looks like a problem. The defence is to keep the repair rate next to a measurement that comes from outside the loop — a labelled regression set, a periodic human audit, a downstream correction rate — so that a suspiciously clean validation metric has something to be inconsistent with. The two together are informative in a way neither is alone: a low repair rate with a low audited accuracy means the schema is not asking anything, and a low repair rate with high audited accuracy means the pipeline is working.

A repair rate is worth having and it is worth being precise about. It is one number per cohort rather than one number, it is three metrics rather than one, and it is only a quality signal in the company of something the loop did not generate. Instrument it that way and the panel stops being a flat line that proves nothing and becomes the thing that names which customer, which prompt version and which class of document is failing.

With the rate defined precisely and segmented by the dimensions above, it becomes possible to read movement in it. That is a different skill from tracking it, and the case that exercises it is the one where the number climbs steadily while nothing on your side has changed at all.