Conformance rate and accuracy

Two numbers describe a generate-then-verify service and they are routinely confused. A conformance rate is the share of requests that produced an output the checker accepted; it comes free from the logs, it is exactly what an on-call dashboard should alarm on, and it answers the question "how often does this service produce output my downstream parser can consume". A labelled accuracy is the share of accepted outputs that are actually right, and it comes only from hand-labelling a sample blind to the verdict.

The gap between them is where a loop stops improving without anybody noticing. Take a service that reads purchase orders — supplier, PO number, currency, line items, totals, delivery date — emits JSON, validates against a schema, and retries up to three times when validation fails. Its conformance rate is 99.4%, computed correctly. A sample of 200 accepted outputs, judged against the source PDFs under the labelling protocol from earlier in this topic, produced 11 that satisfied the schema and carried a wrong value in at least one field. Field-level accuracy of accepted output is therefore about 94.5%, with a 95% interval running from 91.4% to 97.6%.

Two numbers, and what each one is called#

The 99.4% cannot describe whether the values are right, because the schema has no opinion about values. It also moves in a direction that makes it dangerous as a headline: relax a field from an enum to a free string and conformance rises, which is the only metric in the building that improves when you make the checker weaker.

That last property is worth deriving rather than stating, because it is what makes the number unsafe rather than merely incomplete. A conformance rate is the share of requests whose output landed inside the accept set. It therefore depends on two things: where the generator's outputs fall, and where the boundary of the accept set is drawn. Improving the generator moves outputs toward the boundary from outside, which raises the rate. Widening the accept set moves the boundary out past outputs that are already there, which also raises the rate. The number cannot distinguish the two, because it never observes anything but the relation between them. So a team that reports conformance and is asked to raise it has two levers, one of which is real work and one of which is a schema edit, and only one of them shows up in an afternoon.

The 94.5% is a different kind of quantity and the difference is worth naming precisely. It is measured against labels produced outside the system, so its denominator is fixed by the sample rather than by the checker, and nothing the team does to the checker can move it except by changing what actually ships. That is the property that makes it slow to obtain, expensive, and the only one of the two that can be optimised without being gamed.

The number to report against the word quality is the labelled one, and it needs three qualifiers attached or it will be misread the same way. Field-level accuracy of accepted output is 94.5% ± 3.1, measured on 200 hand-labelled samples drawn from March traffic. Coverage — the fraction of requests that produced any accepted output — is 99.4%. End-to-end, counting the 0.6% of requests that exhausted their retries and fell to manual keying as failures, the service answers correctly 93.9% of the time. Reporting the pair is what stops the accuracy figure being gamed by rejecting more, and reporting the interval is what stops a quarter-over-quarter move of two points being treated as a trend when the sample cannot resolve it.

What raising the ceiling actually does#

The proposal that follows a conformance rate is always to raise max_attempts, here from three to five, on the reasoning that if retries produce valid output then more retries produce more of it. The arithmetic is worth doing because it is short and because it goes somewhere unexpected.

The service's measured per-attempt behaviour is that 91.9% of requests are accepted on the first attempt; of the 8.1% that retry, 26% fail again; of those, 29% fail a third time, which leaves 0.61% of requests exhausting the ceiling. Extending to five attempts, with fourth and fifth attempts failing at the higher rates that requests this far into the tail exhibit — 35% and 40% — brings the exhausted share to about 0.085%. Conformance rises from 99.39% to 99.91%. Expected generations per request rise by 0.008, which is a rounding error on the bill, and the affected requests take up to seven seconds instead of up to four.

Those figures come from multiplying along one chain and they are worth walking, because the chain is where the diminishing return is visible. Of every hundred requests, 91.9 finish on the first attempt and 8.1 go round again. Twenty-six in a hundred of those 8.1 fail a second time, which leaves a bit over two in a hundred reaching a third attempt. Twenty-nine in a hundred of those fail again, which is the 0.61 in a hundred that exhaust the ceiling. Each link multiplies a share that is already small by a fraction, so the population reaching each successive attempt shrinks geometrically. Continue the chain with the two higher failure rates and the exhausted share falls to a bit under nine in ten thousand.

Conformance is one minus that exhausted share, which is where 99.39% and 99.91% come from, and the extra generations are the shares reaching the two new attempts added together — six in a thousand plus two in a thousand, the 0.008. Notice what those two derived numbers have in common: both are dominated by the 0.61 in a hundred that reach a fourth attempt at all. A lever applied to a population that small can only move things by amounts that small, whatever it does to that population, and no amount of engineering on the loop changes the size of the population the loop can see.

None of that touches the 11. Joining the labelled sample back to the attempt counts, 10 of the 11 wrong-value outputs had been accepted on the first attempt and the eleventh on the second, where the retry had repaired a missing currency field and left an incorrect delivery date exactly as it was. The loop never saw these outputs as candidates for repair, because from the checker's point of view there was nothing to repair. A retry ceiling is a parameter on the rejected population, and the errors that matter here live entirely in the accepted one.

Pushed further, the change makes served accuracy slightly worse. Outputs accepted on the third attempt or later were labelled correct 71% of the time, against 95% for first-attempt outputs, because a document that defeats the extractor twice is usually a genuinely hard document — a scanned fax, a supplier whose template puts the total in the header, a multi-currency order. Converting those from unserved requests into served ones moves a visible failure that fell to a human into an invisible failure that ships. The expected effect on the labelled accuracy of served output is around a tenth of a point downward, which is unmeasurable at any sample size the team can afford, and that is the honest description: the change is undetectable in the metric that matters and worth half a point in the metric on the slide.

Both of those small numbers fall out of the same weighted average, and running it shows why they are small for different reasons. The requests converted from unserved to served are the difference between the two exhaustion rates: 0.61 in a hundred down to under a tenth of that, so a bit over half a request in every hundred joins the served population. Those arrivals are labelled correct 71% of the time against a served population running at 94.5%, so they drag the average down — but they are half a per cent of it, and dragging half a per cent of a population from 94.5 to 71 moves the mean by about a tenth of a point. The conformance gain is the same half a per cent read as a rise in coverage, which is half a point on a number that starts at 99.4. One quantity, two presentations, and the presentation that looks like progress is the one on the slide because a tenth of a point of accuracy is inside the interval and half a point of conformance is not.

Only two rows in that table act on the population the 11 came from, and both of them make the conformance number go down.

Narrowing the accept set instead#

The 11 errors sorted into four recurring classes: a delivery date read from the order date line, a supplier resolved to the parent company rather than the invoicing subsidiary, line item quantities transposed with unit prices on documents where the columns are unlabelled, and totals that did not equal the sum of the line items because a discount row had been dropped. Three of those four are decidable from the document plus arithmetic, without any model involved.

def check_purchase_order(doc: SourceDoc, po: PurchaseOrder) -> Verdict:
    subtotal = sum(li.quantity * li.unit_price for li in po.line_items)
    if abs(subtotal + po.tax - po.discount - po.total) > Decimal("0.01"):
        return Verdict(False, f"line items sum to {subtotal}, stated total {po.total}")

    if po.delivery_date <= po.order_date:
        return Verdict(False, f"delivery {po.delivery_date} not after order {po.order_date}")

    for field in ("po_number", "supplier_name", "total"):
        value = normalise(getattr(po, field))
        if value not in normalise(doc.ocr_text):       # every value must be copied, not composed
            return Verdict(False, f"{field}={value!r} does not appear in the document text")

    if po.supplier_vat and not vat_registry.matches(po.supplier_vat, po.supplier_name):
        return Verdict(False, f"VAT {po.supplier_vat} is not registered to {po.supplier_name}")

    return Verdict(True, None)

Four checks are in that function and each one is aimed at a named error class, which is worth tracing because it is the difference between invariants and plausibility rules. The first reconstructs the total from the parts: line items multiplied out and summed, plus tax, minus discount, compared against the stated total with a cent of tolerance for per-line rounding in the document itself. A dropped discount row makes the two sides disagree by exactly the discount, so that error cannot survive this check. The second exploits a fact about the world rather than the document — a delivery cannot precede its order — which catches a delivery date lifted from the order date line, because a value copied from that line is the order date and fails the comparison outright.

The third is the one doing the most work. Every value in a named set of fields has to appear somewhere in the document's own text, which converts "did the model read this correctly" into "is this string present", a question with a mechanical answer. And the fourth reaches outside the document entirely, to a VAT registry, which is the only check here that can know something neither the model nor the page contains. That is worth noting against the definition of independence from the start of this topic: three of these checks are independent of the generator because they consult the source, and the fourth because it consults a third party.

The grounding check is the one that pays for itself most often, because it converts "did the model read this correctly" into "does this string exist in the document", which is decidable in about 4 ms per field against the OCR text the pipeline already produces. It catches the parent-versus-subsidiary error whenever the subsidiary's name is on the page, and it catches transposed quantities whenever the transposition produces a number that appears nowhere. It does not catch a value copied accurately from the wrong row, which is the fourth class and the one that stayed unsolved.

Whether this was worth building is a question the two-arm measurement answers directly, run over the same 400-case evaluation set with the same model and prompts and only the checker changed.

Three and a half points of accuracy for eight-tenths of a point of coverage and 14% more model spend, which is a trade nobody can settle without knowing what each kind of failure costs downstream.

The direction of every row in that table is worth reading as a set of consequences of one change rather than as five independent results. Narrowing the accept set means more candidates fall outside it, which is the rejection rate going up. More rejections means more regenerations, which is the generation count and the spend going up. More rejections also means more requests running out of attempts, which is coverage going down. And fewer wrong outputs get through, which is accuracy going up. All five columns move because one boundary moved, and they move in the pattern that boundary implies — so a proposal claiming to raise accuracy without touching conformance is either changing the generator or has not been measured.

The class that survives the invariants#

Nothing here removes the fourth error class. A quantity copied accurately from the wrong row satisfies the arithmetic when the rows happen to be compatible, appears verbatim in the document, and belongs to a real supplier, so it sits inside the new accept set exactly as it sat inside the old one. The measured rate is around 1.2% of accepted outputs, it is the largest remaining contributor to procurement's rework, and closing it would require either a layout-aware extractor that reasons about column geometry or a person. Writing that sentence down in the same place as the metrics is what keeps it from being rediscovered in another two quarters.

The reporting discipline that follows is small and has to be enforced, because the short form of any of these numbers is the misleading one. Conformance and labelled accuracy travel together with the interval and the sample date attached, and the checker version is named beside them, since a change to the accept set moves both numbers in opposite directions and a reader comparing quarters otherwise sees a regression where there was an improvement.

A conformance rate is worth having and it is not a quality metric. Once it is called one, the levers it makes visible are all levers on the population it already rejects, and a team can spend two quarters moving a number that was never attached to the errors anybody was complaining about. The measurement that reconnects them costs a week of labelling; the fix it points at always makes the conformance rate worse, which is why it needs the price of a downstream error sitting next to it.

Everything in this topic so far has been one pipeline at a time — a SQL agent, a provisioning writer, an extractor. The pattern underneath them is the same one, and most systems already contain three or four instances of it that were built separately by different people and are never discussed together. The next chapter goes looking for them, and uses the amount of checking machinery a team already owns as the criterion for deciding which feature to build first.