Normalization and exact match

An eval suite for an extraction system holds a set of documents with the correct value of every field written out by hand beside each one. The hand-written value is the reference, what the model emits is the prediction, and the comparator is the function that reads one such pair and decides whether the prediction counts as the same value as the reference. The score is the fraction of pairs the comparator accepts, which makes it exactly as meaningful as that function and no more. This topic is about choosing those functions, and it starts here because the cheapest comparator is also the one whose repair is most often done by instinct.

That comparator is exact match, which accepts a prediction when its string is identical to the reference's character for character, including case, spacing and punctuation. It costs nothing to run, is deterministic, and needs nobody's judgement on a borderline case, so a team can put it in the build, have it score twelve fields across 500 invoices — 6,000 field comparisons — in well under a second on every commit, and gate a release on the percentage that comes out. The price of that simplicity is that exact match has no notion of two strings standing for one value, so every difference in shape is counted as an error. The standard repair is normalization: rewriting both sides into a canonical form — lowercased, trimmed of stray spaces — before comparing them, so that differences the team regards as cosmetic stop being counted against the model. Which of those rewrites a team is entitled to make is the whole subject of this chapter.

Two properties of exact match are worth separating before the repair is discussed, because only one of them is the reason it is cheap. It is deterministic: the same pair of strings produces the same verdict on every run, on every machine, and again in six months, because the function consults nothing outside its two arguments. That is what makes scoring at this volume safe — twelve fields on each of 500 invoices is 6,000 pairs, and a suite that graded them differently on Tuesday than on Monday would be measuring itself rather than the model. It is also fixed: the equality it encodes was decided once, by whoever wrote the function, and nothing at run time revises it. Those two properties together have a consequence that is easy to state and easy to forget. Determinism means the instrument cannot drift; it does not mean the instrument is right. Everything that can be wrong with a score computed this way is therefore sitting in the definition of equality, which is the one part of the arrangement the run never varies and the one part nobody re-reads.

Reading the failures before writing the normalizer#

Here is what the failing pairs look like on a vendor_name field scored at 71% across 500 invoices, printed before anything was changed.

     prediction              reference
   1 ACME Corp.              Acme Corp
   2 " Acme Corp"            Acme Corp
   3 Acme  Corp              Acme Corp
   4 NORTHWIND TRADERS       Northwind Traders
   5 Bellweather Ltd.        Bellweather Ltd
  ...
  97 Wesson & Smith          Smith & Wesson
  98 Blue Ridge Supply       Blue Ridge Supply Co
  99 (empty)                 Kesterline Fabrication

The 71% is a count and nothing more elaborate, and it is worth doing the count by hand once before arguing about what it means. Each of the 500 invoices carries one vendor_name pair, so there are 500 verdicts; 145 of them are failures, which leaves three hundred and fifty-five passes, and three hundred and fifty-five out of 500 is a shade over 71%. There is no weighting in it, no partial credit, and no roll-up from some finer unit: a field either matched character for character or it did not, and the percentage is the share of the pairs that did.

Sorted into buckets, the 145 failures came out as 39 case-only differences, 24 whitespace differences, 19 trailing punctuation or a dropped period on a suffix, 3 where the words were right but in the wrong order, and 60 where the model had genuinely produced a different company.

Those five counts add up to 145 exactly, and the exactness is the part to check rather than to skim past. A failing pair can differ in more than one way at once: row 1 of the printed list is in the wrong case and carries a trailing period the reference does not have. A bucketing that asked each question independently would file that row under case and again under punctuation, and the five totals would then come to more than the number of failures there are. Landing on 145 is what tells you each pair was filed under one heading only, and that in turn is what makes the counts subtractable: fix what causes a bucket and the failure count drops by exactly the size of that bucket, with no double-counting to unpick afterwards. Every recovery figure in the rest of this chapter leans on that property.

Of those five buckets, the first three are the reason exact match on raw strings is almost never the right check for a human-written field, and the obvious fix is to run both sides through a normalizer.

import re

def normalize(s: str) -> str:
    s = s.strip()
    s = s.lower()
    s = re.sub(r"[^\w\s]", "", s)          # strip punctuation
    s = " ".join(sorted(s.split()))        # collapse whitespace, sort tokens
    return s

assert normalize("ACME Corp.") == normalize("Acme Corp")     # passes
assert normalize("  Acme   Corp ") == normalize("Acme Corp") # passes

What that function does to the comparison is worth stating precisely, because "normalizing" sounds like a softening of the standard and it is not one. Exact match is still exact match: the final test is still character-for-character identity, and no tolerance has been introduced anywhere. What changed is that both strings are rewritten before that test, by the same function, so a pair whose only difference is one the rewrite erases arrives at the test as two identical strings and passes on the original terms. The rewrite being applied to the reference as well as the prediction is what makes this an equality relation rather than a concession — it is symmetric, and a pair the rewrite does not touch scores exactly as it did before.

Eighty-five of the 145 failures now pass and vendor_name reads 88%. That is the same count as before with one group moved across: the three hundred and fifty-five pairs that were already passing stay where they are, because rewriting two identical strings in the same way leaves them identical, and 85 of the 145 failures join them, which makes four hundred and forty of the 500. The denominator has not moved and could not have. The suite is scoring the same 500 invoices it was scoring a moment ago; normalization changes the comparator, not the set. The number is better and it is also less true than it looks, because the four lines of that function are four separate decisions and only two of them were ever justified.

The four steps are not one decision#

Each transformation collapses a set of strings into a single bucket, and the question that matters for each one is not "does this feel like it shouldn't count against us" but "which strings does this make indistinguishable, and are any of them genuinely different values?" Laid out that way, the four steps do not look alike at all.

Read the second column downwards and it accounts for the 85 recoveries exactly: 24 whitespace, 39 case, 19 punctuation, 3 word-order. That the rows sum to the total is not decoration. It means the four steps carve the recovered failures up between them without overlapping, and that is what lets each step be priced on its own — take one line out of the function and the score falls by exactly that row, no more and no less. The decision about the normalizer is therefore four decisions, and the only reason it gets taken as one is that the four lines were written in one sitting.

The last row is the one that should stop you. Three invoices in the set had the vendor's words in the wrong order, and every one of them was a real extraction error: the model had read a two-part name off a stamped letterhead and reassembled it backwards. Token sorting does not recover those; it hides them. The 88% therefore contains 82 genuine recoveries and 3 manufactured ones — the first three rows of the table added together, against the last row standing alone — and the manufactured ones are precisely the cases a reviewer would have wanted to see.

Three failures out of 500 is not the problem. The problem is that the eval now has a rule stating that word order in a vendor name carries no information, and that rule will apply to every future model, every future invoice, and every vendor whose name is a pair of surnames. Nothing in the suite will ever report a word-order error again, and nobody will remember that a line of code decided this.

Ask the consumer, not your intuition#

The two justified steps and the two unjustified ones are not distinguished by how reasonable they sound. Stripping a trailing period sounds at least as reasonable as lowercasing. They are distinguished by what happens to the extracted value after the extractor emits it.

vendor_name in this system is written to a staging table and then matched against a vendor master by a loader whose lookup is a single predicate:

SELECT vendor_id
FROM   vendor_master
WHERE  lower(trim(name)) = lower(trim($1));

That predicate is the definition of "the same vendor" for everything downstream of the extractor. Two strings that the predicate maps to the same row are, for every practical purpose in this system, the same answer; two strings it maps to different rows (or to nothing) are different answers, however similar they look. Lowercasing and trimming in the eval are not charitable adjustments, then — they are a copy of the consumer's own key function, and using them makes the score answer a question the business actually has. Stripping punctuation and sorting tokens are inventions of the eval. "Acme Corp." does not match "Acme Corp" in that lookup; it produces an unmatched invoice that lands in a review queue. Scoring it as a pass tells the team a problem is solved when the review queue is still filling up.

The sentence a reader is likely to write after this, and the one worth blocking before it does any damage, is that normalizing both sides makes the comparison fairer. The reason it is tempting is real: the operation is symmetric, both sides get the same treatment, nobody is being let off anything, and symmetry is what fairness usually looks like. What the operation actually does is declare a set of strings interchangeable, and that is a claim about the system downstream of the extractor rather than a courtesy extended to the model. Such a claim is either true of that system or false of it. Lowercasing is true here because the loader lowercases; token sorting is false here because nothing between the extractor and the vendor master ever puts a vendor's words into alphabetical order. Fairness is not the axis the decision runs along, and a step can be perfectly even-handed and still leave the score describing a system nobody operates.

The rule that falls out of this is narrow and mechanical. Every normalization step in a scorer must correspond to something the consumer of that field already does. If the consumer lowercases, lowercase. If the consumer resolves aliases through a table, run the same table. If the consumer does nothing, compare the raw strings and accept the score you get. When a normalization step has no counterpart downstream, you are not measuring the system — you are measuring a hypothetical system in which someone fixed the loader.

That framing also tells you what to do about the 19 punctuation failures instead of normalizing them away. They are real: "Acme Corp." really does fail the lookup, and the model really does emit trailing periods. Either the loader should strip them (in which case update the loader first, and let the eval follow) or the extraction prompt should stop producing them. Both are fixes. Rewriting the comparator is not.

The same function on an amount field is a bug#

A four-line normalize() does not stay on the field it was written for. Put it in a shared scoring/ module and within a week it is being applied to every string-valued field, including amount_due.

Invoice amounts arrive in whatever shape the vendor's PDF used, and references transcribed from the source documents inherit that, so some read 100,000.00 and some read 100000. Run the normalizer over an amount and the punctuation strip removes both the thousands separator and the decimal point:

>>> normalize("1,000.00")
'100000'
>>> normalize("100000")
'100000'
>>> normalize("1,000.00") == normalize("100000")
True

Following the four lines in order shows why those two land in the same bucket, and it is worth doing because the result looks like a rounding quirk and is not one. "1,000.00" is stripped of surrounding spaces and lowercased, neither of which touches a digit. Then re.sub(r"[^\w\s]", "", s) deletes every character that is neither a word character nor whitespace, and \w covers letters, digits and the underscore — so the comma and the decimal point both fall outside it and both disappear. What remains is the digits in their original order and nothing else, and the final line has no work to do, because a string with no spaces left in it is a single token and sorting one token returns it. The reference "100000" passes through all four steps unchanged. The two results are not merely close; they are the same six characters.

What makes that a hundredfold error rather than a formatting quibble is which two characters were thrown away. A thousands separator and a decimal point are punctuation to a character class and place value to a reader, and the regular expression classifies by character rather than by what the character is doing, because a regular expression has no way to ask. Remove them and two amounts a hundredfold apart are written with the same digits in the same order, which is precisely the situation a decimal point exists to prevent.

A prediction of one thousand dollars now scores as a pass against a reference of one hundred thousand dollars. The eval reports the amount field at 97%, which is roughly the number it reported before anyone touched it, so nothing looks wrong; the check is simply no longer capable of failing on a misplaced separator. This is a two-order-of-magnitude error passing a correctness gate, and it passes because a function written for company names was reused on a field whose consumer parses the string into a Decimal.

The consumer's key function for amount_due is the parse:

from decimal import Decimal

def to_amount(s: str) -> Decimal:
    return Decimal(s.replace(",", "").replace("$", "").strip())

Under that function "1,000.00", "1000.00" and "$1,000.00" are the same value and "100000" is a different one, which is exactly the set of equalities the scorer should have. The general shape is visible here even though the mechanics are not yet: for a field the consumer parses, the comparison belongs downstream of the parse rather than upstream of it.

The decisions have to be written down somewhere#

A twelve-field extractor scored this way contains twelve separate equality decisions, and while they live inside a scoring module they are invisible to everyone who reads the dashboard rather than the code. That invisibility is what lets a function written for company names spread to a currency field without anyone signing off on it, and the cheapest available fix is to make the decisions per-field by construction rather than by convention.

COMPARATORS = {
    "vendor_name": lambda p, r: key(p) == key(r),   # matches vendor_master lookup
    "amount_due":  lambda p, r: to_amount(p) == to_amount(r),
    "po_number":   lambda p, r: p == r,             # opaque key in the ERP
    "line_desc":   lambda p, r: p == r,             # no consumer parses it
}

Two properties of that registry matter more than its shape. There is no default entry, so a new field cannot inherit somebody else's notion of equality by being added to the schema; if a field has no comparator the suite fails loudly rather than guessing. And every relaxation carries a comment naming the consumer that justifies it, which turns the code review question from "does this seem reasonable" into "is that really what the loader does" — a question with a checkable answer.

The registry also makes an operational consequence visible that is otherwise easy to miss. When the loader changes — say someone strips trailing punctuation in the lookup, which would be a perfectly good fix for the 19 failures above — the scorer has to change with it, and on the day it does, the reported accuracy jumps for reasons that have nothing to do with the model. A score series is only comparable across the interval in which its comparators were fixed, so the change belongs on the dashboard as an annotation rather than as an unexplained step in the line. Teams that keep the comparators in code and the numbers in a spreadsheet discover this the first time somebody attributes a four-point jump to a prompt they shipped that week.

That constraint follows from the definition this chapter opened with rather than from any convention about dashboards. The score is the fraction of pairs the comparator accepts, so it is a function of two things and not one: what the model emitted, and what the function counts as the same value. Two numbers in a series are therefore comparable only if one of those two was held still between them. A series in which the model moves and the comparator does not is a measurement of the model, which is what everybody reading it assumes they are looking at. A series in which both moved is a measurement of neither, and it cannot be repaired after the fact, because the only thing the run wrote down was the total.

Canonicalization stops at surface form#

Normalization is a canonicalization step: it rewrites a value into one agreed form, so that two writings of the same thing arrive at the comparison as one string. Canonicalization only ever merges strings that differ in shape. It has nothing to say about strings that differ in content, which is why it is not a fuzzy matcher and should not be pushed in that direction. "IBM" and "International Business Machines" are the same vendor, and no amount of case-folding will bring them together; that is an alias-resolution problem, solved with a table, and the eval should use whichever table the loader uses. Nine of the 60 remaining failures in the invoice set are exactly this — a trading name against a registered name — and the loader already resolves them, through a vendor_alias table it consults when the direct lookup misses. Calling that resolver from the scorer is legitimate for precisely the reason token sorting was not: it is something the consumer does, so the equality it creates is an equality the system has.

The boundary falls exactly there because of what a normalizer is. Every step in one is a function of the input string and nothing else: it reads characters, rewrites characters, and has no access to any fact that is not already in the string. Two strings differing only in shape still hold the same characters in some arrangement a rewrite can reach, which is why case-folding and whitespace collapsing work at all. "IBM" and "International Business Machines" do not stand in that relation — the second holds letters the first does not, in an order no rewrite of the first could produce — and the only thing making them one vendor is a fact recorded somewhere outside both strings. No function of the characters can discover a fact the characters do not contain. That is why alias resolution needs a table rather than a cleverer regular expression, and why the table has to be somebody else's.

"Acme Corp" and "Acme Corp LLC" may or may not be the same vendor depending on whether the master data distinguishes the entities, and the eval cannot decide that on its own. Reaching for edit distance at this point — accepting anything within two characters — buys back a handful of passes and imports a much larger blind spot, since two characters is also the distance between two genuinely different account codes.

Normalization also tells you nothing about why a field is wrong. The 60 remaining failures include blanks, values lifted from the wrong region of the page, and the shipper's name where the vendor's should be, and those are three different defects with three different fixes. A pass/fail comparator, however carefully its equality is defined, compresses all of them to one bit. Later chapters give the score more structure than that; this one is only about not corrupting the bit.

What a normalizer encodes, then, is a claim about the world: that the system downstream cannot tell these two strings apart. That claim is checkable, it is checkable by reading one function in somebody else's service, and it takes minutes. Every step that survives the check makes the score more useful, and every step that does not makes it permanently less able to report a class of error that nobody will think to look for again.

The rule has a hard edge, and the edge is where the next problem starts. For vendor_name there was a key function to copy, because the consumer compared strings. For a date field such as effective_date there is no such thing: a billing job does not compare the string at all — it parses it into a date and charges from the result. "2024-03-01", "March 1, 2024" and "1 March 2024" are one value to that job and no sequence of lowercasing, trimming or stripping will unify them, while "March 2, 2024" differs from all three by a single character and must fail. String rules run out here, and the comparison has to move.