Record-level and per-field scoring

An extractor that emits a fixed set of typed fields per document can be scored two ways, and most suites carry both without reconciling them. Record-level exact match asks what fraction of records had every field right; mean per-field accuracy asks how often an arbitrary field was right. On one insurance claims extractor emitting twelve fields per claim as JSON, over 800 claims, those two numbers are 41% and 94%.

Both are correctly computed from the same data, and they feel contradictory. Reconciling them is the first thing to do, because the reconciliation is also the diagnosis — it tells you whether you are looking at a broad quality problem or at two specific fields nobody is watching. This chapter does that arithmetic, then builds the per-field scoring that follows from it: the outcomes a comparator has to distinguish beyond "the value is different", and the weights that decide which of twelve numbers the team actually works on.

Reconciling 41% with 94%#

The belief that has to go first is that a mean per-field accuracy of 94% describes a typical record. It is the natural reading, because an average feels like a statement about the middle of a distribution and 94 is a mark most people would be pleased with. It is wrong here for a structural reason rather than a numerical one. The average is taken across fields, and a record is not an average across its fields; it is an and across them. A record with eleven fields right and one wrong is not most of a record. It is a failed record, and record-level exact match cannot tell it apart from one that got nothing right.

Start smaller than twelve, because the operation is visible at two. Take an extractor emitting two fields, each right on 0.94 of records. Of a hundred records, ninety-four clear the first field. Of those ninety-four, the second field is wrong on six in every hundred, so about six of the ninety-four fall away and a shade under eighty-nine survive both. That last step is where multiplication comes from: the survivors of the second gate are a fraction of the survivors of the first, not a fraction of the original hundred. Two rates that apply in sequence combine by multiplying, rather than by averaging them or by adding their error rates together.

Two conditions have to hold for that step to be legal, and the second is what the rest of this section turns on. The gates have to be exhaustive, meaning a record passes only if every field passes — which is what record-level exact match means by definition, so that one is free. And the fields have to be independent, meaning that knowing one field went wrong tells you nothing about whether another did. Independence is an assumption about this extractor, not an observation of it. It fails whenever one upstream cause spoils several fields at once, and the direction in which it fails turns out to be readable off the numbers.

Extending the pass from two fields to twelve adds rows and nothing else, and repeated squaring gets to the end of it quickly. Two fields is a shade under nine-tenths; four fields is that figure squared, a little over three-quarters; eight fields is that squared again, a bit over three-fifths; and twelve fields is the eight-field figure multiplied by the four-field one.

Twelve independent fields at 0.94 each produce a fully correct record with probability 0.94^12 = 0.476. That is the whole gap: a per-field accuracy most people would call good multiplies down to a coin flip at the record level, and there is nothing anomalous about seeing 41% next to 94%.

The rough figure usually carried around for that product is about 0.46, and the honest one is 0.476. The difference is rounding accumulated over twelve steps rather than a second effect, and it is small enough not to change any argument here — but it is worth doing the multiplication rather than trusting the recollection, because 47.6% is about to be used as a ceiling, and a ceiling misremembered downwards makes an observation look closer to it than it is.

What is worth noticing is that 41% sits below 47.6%, and the direction is informative. For a fixed mean, the product of twelve numbers is largest when they are all equal, so 47.6% is the ceiling that a mean of 0.94 permits under independence. Landing under the ceiling means either the fields are uneven or the errors are correlated, and correlated errors push record accuracy the other way — a claim scanned crookedly ruins several fields at once, which produces fewer, worse records and therefore a higher record-level score than independence predicts. Unevenness is the explanation that fits.

Both halves of that inference are asserted rather than shown, and both are checkable in a few lines. Take the claim that a fixed mean is best served by equal numbers, which is what makes 47.6% a ceiling rather than just one possible value. At two numbers the whole argument is visible. Fix a pair whose average is 0.94: they can both be 0.94, or they can be pulled apart by the same amount in opposite directions, one raised and one lowered, which leaves the average exactly where it was. Multiplying the pulled-apart pair multiplies something above 0.94 by something below it, and the product of such a pair is always the equal pair's product minus the square of the distance each one moved. A square is never negative, so pulling the pair apart can only subtract. The twelve-field case follows by repetition: if any two of the twelve differ, replacing both with their average leaves the mean untouched and raises the product, so the only arrangement that nothing can improve is the one where all twelve are already equal.

The direction correlated errors push is the counter-intuitive half, because the instinct is that correlation makes everything worse. Hold the per-field accuracies fixed, which fixes the total quantity of wrong fields in the whole set. Those wrong fields have to land on records somewhere. Scattered across as many different records as possible, each one ruins a record of its own, and the number of ruined records is as large as it can be. Piled onto the same records — a claim scanned crookedly ruining several fields at once — the same quantity of error ruins fewer records, and every record it does not touch is a clean one. Correlation therefore produces fewer, worse records and a higher record-level score, not a lower one.

So the observation does the eliminating. Correlation would push the record-level number up from the independent prediction and 41% sits below it, which rules correlation out and leaves the only other candidate: the fields are uneven.

Which matters enormously, because the two worlds behind those numbers need opposite responses.

Neither number tells you which world you are in. Both are compatible with either, and a team that reads a flat 41% as evidence of general mediocrity will spend its effort on the model's general behaviour, which in World B is already excellent.

The per-field table is what tells them apart#

This table is also where the two worlds stop being hypothetical. Add the twelve accuracies in it and they come to a shade over eleven and a quarter; a twelfth of that is 0.94, which is the mean per-field accuracy the dashboard reports, arrived at from the individual fields rather than assumed. Multiply the same twelve together instead of averaging them, and the product is 42.8%. World B in the table above is not an illustration of a possible extractor; it is this one, computed from its own numbers. The observed 41% sits only a little under that, which is the second useful thing the table says: the residual correlation between fields is small, and unevenness alone very nearly accounts for the whole shortfall.

The shape of the fall is worth seeing as well as its size. Multiply the ten good fields together and the running product only just drops to nine-tenths, because none of them subtracts more than a couple of parts in a hundred. Multiplying by 0.710 and then by 0.660 removes roughly half of what is left, in two steps. A record-level score built from twelve factors is decided by its worst factors and not by its typical ones, and any factor close to one is very nearly invisible in it.

Two rows carry the entire gap between 94% and 99%, and they are the two rows nobody was looking at, because both fields are rare. A prior-authorisation code, which is the insurer's advance agreement to pay for a named procedure, appears on 12% of claims, and a coordination-of-benefits flag, which marks a member covered by a second insurer that has to pay first, on 8%. An earlier version of the dashboard had shown both at around 0.96, computed over all 800 claims, and 0.96 on a rare field is mostly a measurement of how often the model correctly emits nothing.

That last clause carries a mechanism worth slowing down on. A prior-authorisation code is absent from nearly nine claims in ten, so on nearly nine claims in ten the correct output for that field is nothing at all. An accuracy computed over every claim pools those with the claims where a code has to be located on the document and read off it, and if the scorer awards a point for correctly emitting nothing, most of the points it awards are for the easy population. The floor that sets is the thing to see. A component that emitted nothing for prior_auth_code on every claim — no reading, no model, no cost — would be marked correct on every claim that has no code, which is nearly nine in ten of them. A figure in the mid-nineties is not far above what that floor buys for free, and bundling a correct null in with a correct value, the outcome this chapter later gives a counter of its own, is what put it there.

Breaking the 29% error on prior_auth_code by failure mode explains why the unconditional number was so flattering: about 6 points of it are wrong codes on claims that have one, and about 23 points are codes invented on claims that have none. The model has learned that claims of a certain shape usually carry an authorisation code and supplies one. That is a fabrication in a structured field, it is the single most expensive defect in this extractor, and a record-level pass/fail number reports it as part of an undifferentiated 59% failure rate.

Those two components are worth tying back to how often the field appears, because each is capped by a different population and the two populations are very different sizes. Invented codes can only happen on claims that have none, which is nearly nine claims in ten, and the model supplies one on about a quarter of them; a quarter of nearly nine-tenths is a little over a fifth of the whole set, and that is the twenty-three points. Wrong codes can only happen on the claims that do carry one, and those are 12% of the set — so six points of error against twelve points of opportunity means the model reads the code wrong on roughly half of the claims that have one. Both failure modes are severe. Only the larger one is visible in a headline accuracy, and it is larger because it has more than seven times as many claims to happen on, not because the model is worse at it.

An explicit policy for each way a field goes wrong#

Scoring field by field forces a decision that record-level exact match lets a team avoid: what counts as wrong, exactly. Four things can happen that are not "the value is different", and they behave differently downstream.

key_missing and null_for_value look like the same failure and are not: one leaves the destination column at its default and the other overwrites whatever was there. correct_null needs its own counter because bundling it with correct is what produced the misleading 0.96 above. And extra_key is not a field outcome at all — there is no field to attribute it to — so it belongs to the record.

from enum import Enum

class Outcome(str, Enum):
    CORRECT        = "correct"
    CORRECT_NULL   = "correct_null"
    KEY_MISSING    = "key_missing"
    NULL_FOR_VALUE = "null_for_value"
    VALUE_FOR_NULL = "value_for_null"
    VALUE_WRONG    = "value_wrong"

def score_field(name, prediction: dict, reference: dict, comparators) -> Outcome:
    if name not in prediction:
        return Outcome.KEY_MISSING
    pred, ref = prediction[name], reference.get(name)
    if ref is None:
        return Outcome.CORRECT_NULL if pred is None else Outcome.VALUE_FOR_NULL
    if pred is None:
        return Outcome.NULL_FOR_VALUE
    return Outcome.CORRECT if comparators[name](pred, ref) else Outcome.VALUE_WRONG

def score_record(prediction: dict, reference: dict, schema, comparators):
    outcomes = {f: score_field(f, prediction, reference, comparators) for f in schema}
    extra = sorted(set(prediction) - set(schema))
    return outcomes, extra

The order of the tests in score_field is not arbitrary, and reading them in order is the quickest way to see what the six outcomes are actually distinguishing. The missing-key test comes first because a key that is not present cannot be compared to anything — there is no value to hand to a comparator, so no other question can be asked yet. The reference is consulted next, and it is the reference rather than the prediction that decides which pair of outcomes is in play, because the reference is what defines whether this claim was supposed to carry a value at all. Only when both sides hold a value does the field's own comparator get called. That last point bounds how much damage a bad equality decision can do here: however permissive a normalizer or a parser is, it can only ever move a pair between correct and value_wrong, and the other four outcomes are settled before it is consulted.

The comparators argument is where the first two chapters of this topic are wired in: date_of_service gets the billing job's date parser, billed_amount gets the Decimal coercion, claim_id gets raw string equality because nothing downstream interprets it, and prior_auth_code gets a case-insensitive comparison because the routing table upper-cases it. One dictionary, twelve entries, each one a decision someone had to make deliberately.

Weight by what a field costs, not by how often it shows up#

Averaging the twelve field accuracies treats a coordination-of-benefits flag and a place-of-service code as equally important, which no one believes. The obvious repair is to weight, and the choice of weights is where the score is won or lost.

Weighting by frequency, so that the score reflects the real distribution of work, sounds principled and is wrong here in a way that is easy to demonstrate rather than argue about. Weighted by how often each field appears, the two broken fields contribute 0.12 and 0.08 against ten fields contributing 1.0 apiece, and the composite comes out at 98.5%. The extractor's worst defect — invented authorisation codes on a quarter of all claims that have none — moves the reported number by one and a half points.

Following that arithmetic is more convincing than the number is, because what does the damage is where the weight goes rather than what the accuracies are. A weighted mean multiplies each field's accuracy by its weight, adds the products, and divides by the sum of the weights, and that last division is what keeps the result reading as an accuracy rather than as a total. The ten common fields carry a weight of one apiece, so they contribute their ten accuracies unchanged, and those ten add up to a shade over nine and nine-tenths. The two broken fields carry 0.12 and 0.08, so 0.710 and 0.660 are shrunk between them to about a seventh of a point. Dividing by the summed weight — ten and a fifth, since the two rare fields add a fifth of a weight unit to the ten — gives 98.5%.

The step to read twice is the weighting itself. The two fields the extractor is worst at were handed a fiftieth of the weight of the ten it is best at, for a reason that sounds like fairness rather than like a thumb on the scale: they come up less often. One and a half points is the whole distance between a perfect score and this one, so one and a half points is the entire room the metric has for the defect this chapter exists to find. The composite is arithmetically correct, defensible line by line, and almost blind by construction.

Frequency is not consequence, and on this data the two run in opposite directions. The fields that appear on every claim are the routine ones a reviewer fixes in seconds; the rare fields are the ones that misroute money, precisely because they are exceptions to the normal path and the normal path is what is automated. Frequency weighting concentrates the team's attention on whichever field is most common, which is almost always the field that is already at 99%.

Weighting by consequence means assigning each field a number proportional to what it costs when wrong, which is a product decision rather than a statistical one. Here the three automatically consumed fields get weight 5, the two routing fields get weight 8 because an unauthorised payment is both expensive and hard to claw back, and the seven human-reviewed fields get weight 1.

Totalling those weights is what shows the difference from frequency weighting, because the composite they produce is not readable off the individual numbers. Three fields at 5 is 15 weight units, two fields at 8 is 16, and seven fields at 1 is 7, so the scheme distributes 38 units across the twelve fields and the two routing fields hold 16 of them. That is more than two-fifths of the total, against the fifth of a single unit that frequency weighting gave the same two fields. Multiplying each accuracy by its weight, adding the products and dividing by 38 gives 86%.

Where that fourteen-point shortfall comes from is the part worth reading off. The seven reviewed fields give back almost all of the seven units available to them. The three automatic fields give back close to the full 15, since they sit at 0.993 and above. The two routing fields have 16 units available and return about eleven of them, and that single deficit is very nearly the whole of the shortfall. Two fields out of twelve, weighted by what they cost rather than by how often they occur, now account for essentially all of the distance between this extractor and a perfect one — which is the same fact the record-level product was reporting, arriving through a different door.

All four are correctly computed and none is the score. What the team reports is the per-field table plus the consequence-weighted composite for tracking, and the record-level number stays on the dashboard as a separate gate, because there is a consumer that needs it: the ERP posts a claim as a unit, and a record with one wrong field is a record that cannot post. Record-level exact match is a bad diagnostic and a perfectly good acceptance criterion for whole-record ingestion, and the mistake was never computing it — it was computing only it.

One more distribution is worth keeping alongside, because it drives staffing rather than engineering: of the 800 claims, 41% are perfect, 34% have exactly one wrong field, and 25% have two or more. That shape tells an operations lead how many claims a reviewer must open and how long each takes, and it is invisible in every average above.

Those three shares account for every claim in the set, which is what makes the distribution readable: a claim has zero wrong fields, or exactly one, or more than one, and no claim can sit in two of the buckets. The middle bucket is the operationally interesting one, because a claim with a single wrong field is a claim a reviewer fixes in one edit, and it is a third of the set. The record-level gate collapses that third into the same verdict as the quarter with two or more wrong fields — correct for the ERP, which cannot post either of them, and useless to anyone deciding how many reviewers to hire. This is the same loss the chapter opened on, one level up: an and across fields discards how many of them failed, exactly as an average across fields discards which ones did.

The weights come from what survives review#

Assigning 5 and 8 and 1 by intuition is better than assigning 1 to everything, and it is not where the numbers should come from. What makes a field expensive is not its accuracy in isolation but how much of its error reaches the outside world, and that is a product of three things: how often it is wrong, how much of that a human catches, and what an escape costs.

The middle column is what reorders the list, and it has nothing to do with the model. Ten of the twelve fields sit behind a reviewer who opens every claim and fixes obvious errors in seconds, so their error rates are attenuated by a factor of twenty before they reach anyone. The two worst fields are consumed by an automatic routing step with no human in the path, so their error rates pass through undiminished — and they are, by two orders of magnitude, the largest source of escaped defects in a system whose worst per-field accuracy is otherwise 98.2%.

The factor of twenty is the review rate read backwards, and doing that conversion makes the fourth column reconstructible rather than something to take on trust. A reviewer who catches about 95% of a field's errors lets the remaining one in twenty through, so fifteen wrong diagnosis codes per thousand claims reach the outside world as about three-quarters of one, and eighteen wrong allowed amounts as a little under one. The two routing fields have no such divisor anywhere in their path. Nothing stands between the extracted value and the action it triggers, so 290 and 340 wrong values per thousand claims are 290 and 340 escapes per thousand claims, and the ratio between those and the fraction of one that the reviewed fields leak is what "two orders of magnitude" is measuring.

That is a claim about the system rather than about the model. Reviewing a field divides its error rate by however much of it a person catches; automating a field multiplies it by one. Two fields at identical accuracy can differ twentyfold in escaped defects on the strength of who reads them, which is why the accuracies in the per-field table are the wrong column to sort the backlog by.

Where per-field scoring stops#

Scoring each field against its own comparator has nothing to say about relationships between fields. A claim whose allowed_amount exceeds its billed_amount, or whose date_of_service falls after the claim's received date, is internally incoherent no matter how each field scores in isolation — and it is quite possible for every field to be individually plausible and the record to be impossible. Those are invariants, they are cheap deterministic assertions, and they belong in the suite next to the field scores rather than inside them.

Per-field accuracy also averages over documents, which hides whether the errors are spread thinly or concentrated. Two builds can report the same twelve field accuracies while one fails a little on many claims and the other fails catastrophically on the 6% of claims that arrive as photographs of a fax. The per-record distribution above catches some of this; a breakdown by document source catches the rest, and neither is automatic.

What a suite has after all this is a set of numbers that mean what they say: comparators matched to consumers, failure modes distinguished, weights tied to cost, and aggregates that name the question they answer. The single most useful habit in it is refusing to publish one number where the underlying data has twelve. Aggregates are fine; an aggregate nobody can decompose is one nobody can act on, and six weeks is a realistic price for finding that out.

What the suite does not yet have is any guarantee that those numbers will keep meaning that. Scores decay in place — a comparator survives a change in the data it was written for, a metric gets swapped for a friendlier one during a bad week, and the resulting number goes flat and stays flat while the behaviour underneath it drifts. That failure has a shape worth recognising, and it is almost always noticed by a customer first.