Scorer validation and canary tests

A metric that does not move is what a healthy system looks like on every other dashboard a team owns, which is why a flat score is read as reassurance rather than as a question. It is the one place that reading is unsafe, because a comparator can be structurally incapable of failing, and a comparator that cannot fail produces a beautifully stable line.

This chapter is about establishing, before a customer does, whether a number is carrying the information it is assumed to carry. The technique is borrowed from mutation testing — corrupt the thing under test deliberately and assert that the check notices — and it is cheap enough to write in an afternoon. The chapter also traces how a comparator gets chosen for the number it produces rather than for what it models, because that substitution is the usual origin of a flat line, and it leaves a paper trail with an identifiable week in it.

A flat number is not a healthy one#

Take an answer-quality dashboard showing mean semantic similarity of 0.82 against reference answers across a 500-example suite, holding between 0.81 and 0.83 every week for three months. What that flatness tells you is that the outputs kept sounding like the references. That is the entire content of the number. It never carried information about whether the facts in those outputs were true, so it could not have moved when the facts stopped being true.

That last step is worth slowing down on, because it is a claim about what a comparison of two strings is capable of rather than about how good this particular comparison happens to be. Semantic similarity reads the model's answer and the reference answer, turns each into a vector — a fixed-length list of numbers standing for the text — and reports how closely the two point in the same direction. Every input to that calculation is a string. Whether a section reference in the answer points at a section the source document actually contains is not a property of the string at all; it is a property of the string's relationship to a document the comparison never opens. No function of the two strings alone can report it, and a better embedding model does not change which inputs the function is given.

The second half of the mechanism explains the tightness of the band rather than its level. Fabricating a section number alters a handful of characters in an answer that is otherwise several hundred words long, and a vector built from an almost identical string is an almost identical vector. The corruption that mattered most to the customer was, in the only terms this metric can read, among the smallest edits in the suite. The line stays where it is.

The steadiness does real work, and it is worth being precise about the mechanism, because this is how a metric causes harm rather than merely failing to help. When somebody raises a concern — an engineer notices an odd citation, a support lead forwards a customer question about a section number — the dashboard is the first thing consulted, and an unchanged number ends the conversation. A plausible aggregate does not just fail to detect a problem; it terminates the investigation that would have.

What makes this sting is that the specific defect rarely needs an expensive check. A citation is not free text — it is a key into a source document's section index, which is enumerable, and the check is a regex and a set membership test:

CITE = re.compile(r"§\s*\d+(?:\.\d+)*(?:\([a-z]\))?")

def bad_citations(answer: str, valid_sections: set[str]) -> list[str]:
    return [c for c in CITE.findall(answer)
            if normalise_section(c) not in valid_sections]

Open text is rarely uniformly unstructured; it is prose with typed substrings embedded in it — section references, dates, amounts, account numbers, product SKUs — and each of those substrings can be checked deterministically against the thing it is a key into, using exactly the machinery that already applies to structured fields. A summary that resists judgement as a whole may still contain six values that a code-based check can settle. Reaching for a single global comparator over the whole answer, and then choosing the cheapest available one, skips past all of them.

The same failure with money in it#

A second team hit the identical shape with a structured pipeline, and their version has a paper trail that shows where the decision went wrong.

Their extraction suite scored amount with exact string match, and it was miserable. Roughly a third of the failures were $1,250.00 against 1250.00, 1,250.00 against 1250.0, and other renderings of the same number that the ledger parses identically. The suite reported 62%, everyone knew most of the gap was noise, and nobody triaged the failure list any more — which is a genuine engineering problem, because a suite that people have learned to ignore provides no protection at all.

So they replaced it. Embedding similarity between the predicted string and the reference string, threshold 0.85, and the number came out at 0.93. It sat between 0.92 and 0.94 for a quarter. Then a customer reported their ledger was off by a factor of a hundred on a subset of invoices: the model had been emitting amounts with the separator and the decimal point dropped, writing 1,250.00 as 125000, and the ledger's parser read that as one hundred and twenty-five thousand dollars. The similarity gate never moved, because 1,250.00 and 125000 are both short numeric strings and sit at cosine 0.98 of each other.

The metric change bought something real. It removed about 190 false failures from a 500-row suite and made the remaining list worth reading, and for a quarter the team acted on that list. What it cost was every failure mode that preserves the shape of a money string, which is nearly all of them: a dropped separator, a transposed pair of digits, a lost minus sign, a value from the wrong column of the invoice. The suite went from failing on differences that did not matter to being unable to fail on differences that did.

The 190 in that account is not an independent measurement. It is the 62% read the other way round: a suite of 500 rows scoring 62% has 62 rows in every hundred passing and 38 in every hundred failing, and 38 in every hundred of 500 is 190. So the failure list nobody was opening was not a subset of the gap between the reported number and a perfect one — it was the whole of it, and the claim the table below makes about that list is correspondingly strong.

The load-bearing moment is week nine, and the mistake there was not choosing cosine over exact match. It was choosing a comparator on the basis of the number it produced. The stated justification — "exact match keeps failing on formatting differences nobody cares about" — is a claim about the field's semantics, and it is correct: the ledger parses $1,250.00 and 1250.00 to the same Decimal. That claim has a direct implementation, which is to compare the parsed values. Nothing about it recommends comparing embeddings; embeddings were reached for because they were on hand and because they made the dashboard green.

The belief that makes week nine feel like ordinary engineering is worth stating in full, because it is the one a reader carries away unless it is named: that of two comparators run over the same data, the one producing the better number is the better comparator. It is tempting because the number is the only axis on which the two are visible at the moment of the decision. Both ran over the same 500 rows, both returned a figure between zero and one, and one of them was higher. What that comparison cannot show is the property the choice actually turns on, which is not how the two comparators score the rows already in the suite but which pairs they are able to tell apart. Two comparators can return the same verdict on every row of a suite and still differ completely on a row nobody has written yet, and no aggregate over the existing rows separates them. Separating them requires constructing an input on which they must disagree, which fixes where to intervene: at the moment a comparator is proposed, when somebody can still be asked which pairs it will stop distinguishing, rather than at the moment its number moves.

Week six deserves its own line. A suite whose failure list has stopped being opened is already broken, and it is broken in a way that predicts the next failure: the pressure to make the number look better is now purely cosmetic pressure, and cosmetic pressure selects for comparators that cannot fail. Watching whether anyone triages failures is a cheap and neglected health check on an eval suite.

The fix that keeps the wins#

Reverting to exact match repeats the original problem in the other direction. It reinstates 190 failures that nobody will act on, and a suite nobody acts on is exactly how this team talked itself into similarity the first time. Bolting a numeric tolerance onto the similarity score is worse, because it welds a special case onto a comparator that was never modelling the field's type — and the special case will have its own threshold, chosen the same way the 0.85 was. What is wanted is a comparator that keeps the false failures gone without reintroducing the blindness, and neither of those two moves is one.

The fix is the typed comparison of chapter 2, applied to the field the ledger actually parses:

from decimal import Decimal, InvalidOperation
from ledger.ingest import parse_amount        # the consumer's own function

def amount_matches(prediction: str, reference: str) -> bool:
    expected = parse_amount(reference)        # raises on a bad fixture
    try:
        return parse_amount(prediction) == expected
    except (InvalidOperation, ValueError):
        return False

assert amount_matches("$1,250.00", "1250.00")     # the false failures are gone
assert not amount_matches("125000", "1250.00")    # and this one fails, loudly

Three numbers on the same 500 rows: exact match 0.62, cosine 0.93, typed comparison 0.99 before the model update and 0.79 after it. The typed number is the only one of the three that carries information about the regression, and it is worth noticing that it is lower than the similarity number the team had been reporting. A comparator that models the field will usually produce a less flattering number than one that does not, which is the whole reason the substitution was tempting.

Break the output on purpose and watch the number#

Every claim above reduces to one testable property: when the output changes in a way that matters, the score must move. That is a property of the scorer, it can be tested directly, and almost nobody tests it.

The reason it needs a test of its own, rather than falling out of the tests a comparator already has, is a property of what a comparator is. It is a total function: hand it a prediction and a reference and it returns a verdict, for every pair, always. There is no input on which it declines to answer. A comparator that has been quietly loosened past the point of usefulness therefore does not raise, does not log and does not slow down. It returns a number of the same shape, in the same range, at the same cost as the one it returned last week. Nothing about the output distinguishes a working comparator from a broken one, which is why the flat line at the top of this chapter was unreadable in both directions.

If the output carries no signal about the instrument, the only remaining move is to supply an input whose correct verdict is already known from somewhere other than the instrument. That is what a canary is. The pair 125000 against 1250.00 has a verdict fixed by the ledger's parser rather than by the scorer: the parser reads those two strings as two different amounts, so any comparator claiming to model this field must return a failure on that pair, and a comparator that returns a pass is wrong regardless of what its dashboard says. The authority has to come from outside the scorer, and for a typed field it does — the consumer of the value is the authority. That is the same fact the typed comparison rests on (chapter 2, where the comparison moves downstream of the parser the consumer already runs), used here to fix a verdict rather than to choose a rule.

Which cases the canary set contains is then not a matter of taste, and the asymmetry is easy to get backwards. Every relaxation of a comparator works by merging strings that were previously distinct into a single bucket, and merging can only convert a failing pair into a passing one. It cannot do the reverse, because two strings a stricter rule already accepted are still in the same bucket once the buckets get bigger. A canary set built entirely out of pairs that ought to pass is therefore green before the loosening and green after it, by construction, however far the loosening goes. The cases carrying the information are the ones the comparator must fail — and those are the ones nobody writes, because asserting that a scorer rejects something feels like a test of the obvious.

import random

MUTATIONS = {
    "benign_reformat":  lambda s: s.replace("$", "").replace(",", ""),
    "drop_separators":  lambda s: s.replace("$", "").replace(",", "").replace(".", ""),
    "swap_two_digits":  swap_two_digits,
    "flip_sign":        lambda s: "-" + s.lstrip("-"),
}

def canary(metric, predictions, references, mutate, k=100, seed=0):
    rows = random.Random(seed).sample(range(len(predictions)), k)
    mutated = list(predictions)
    for i in rows:
        mutated[i] = mutate(mutated[i])
    return metric(predictions, references), metric(mutated, references)

base, after = canary(typed_amount_score, preds, refs, MUTATIONS["drop_separators"])
assert base - after > 0.15, f"comparator is blind to dropped separators: {base:.3f} -> {after:.3f}"

Run the four mutations against 100 of the 500 rows, under each of the three comparators, and the result is the whole topic on one page:

Exact match falls by twenty points on all four, including the benign one, so it reports change without distinguishing damage from formatting. Cosine does not move to two decimal places on any of them; the largest movement across all four mutations is three thousandths, which is inside the run-to-run variation of the embedding endpoint. The typed comparator holds on the benign perturbation and drops by exactly the injected error rate on each of the three harmful ones — and the fact that it drops by exactly twenty points is itself a useful check that the scorer is counting what you think it counts.

That twenty points is worth deriving, because the derivation is what makes the assertion in the harness checkable rather than magical. The mutation is applied to 100 rows out of 500, which is one row in five. Each corrupted row that had been passing now fails, since the typed comparison parses 125000 to a different Decimal from 1250.00 and returns false. No row moves the other way, because corrupting a prediction cannot repair one that was already wrong. The score therefore falls by the share of rows that were both corrupted and previously passing, and at a baseline of 0.99 that is very nearly the whole hundred — a fifth of the suite, which is twenty points. The equality between the drop and the corruption rate is a condition rather than a law: where a comparator's baseline is low, fewer of the corrupted rows had a pass to lose, and the fall is correspondingly smaller than the share corrupted.

The same reasoning is why the assertion in the harness reads base - after > 0.15 rather than testing for the full fifth. The expected movement is twenty points, the guaranteed movement is smaller by however many of the corrupted rows were failing already, and a test demanding the theoretical maximum would go red on a suite that had merely got slightly worse in the ordinary way. What the threshold asserts is that most of the available movement happened. That is a coarse question and it does not need a delicate number, because a comparator blind to the injected defect returns a difference of zero rather than a difference of a few points, and there is a great deal of room between the two verdicts.

Cosine's non-response has a mechanism of its own, and it is the threshold rather than the similarity score that does the work. The comparator counts a row as a pass when the similarity between prediction and reference clears 0.85, and 1,250.00 and 125000 sit at 0.98 of each other. Corrupting a row leaves it clear of the line rather than merely near it, so the row is still counted as a pass and the mean over 500 rows does not move. The three thousandths that do move come from the handful of rows whose corrupted similarity happened to land on the other side of the threshold. Moving the threshold is not the repair either: any threshold low enough to keep accepting the benign reformatting is also low enough to accept a dropped separator, because this comparator has no representation in which the harmless edit and the harmful one differ in size.

Deliberate corruption transfers to the open-text side, where the technique answers a question nobody can answer by inspection: how often does the expensive checker catch the thing you built it for. Take 50 archived answers, increment one digit in a real citation so that it refers to a section the document does not have, and run both checks. A section-index lookup flags 50 of 50, because a missing key is a missing key. A claim-verification pipeline flags 46 of 50, and the four it misses are all cases where the fabricated section number sits in a sentence whose substance is otherwise correct, so the decomposer folded the citation into a claim that came back supported. A recall of 0.92 on deliberately planted defects is a fact about the checker that no amount of reading its outputs would have produced, and it is the number to quote when someone asks how much the correctness gate is worth.

The reason 0.92 is quotable, where a recall figure computed on production traffic usually is not, is the denominator. Recall is the share of the defects that were there which the checker found, so computing it at all requires knowing how many were there. On real traffic that count is precisely what nobody has, because the defects the checker missed are the ones nobody has seen. Planting removes the problem by construction: 50 answers were altered, so 50 defects exist, and 46 of 50 is 0.92 with a denominator that is not an estimate. Corrupting inputs on purpose rather than waiting for real ones is what buys the denominator, and the denominator is what makes the number mean anything.

The two checks in that run are pinned to very different degrees by the same exercise, and the difference is the line between what can be validated and what can only be sampled. The section-index lookup flags 50 of 50 because its verdict is decided by something outside itself: a section reference either appears in the document's enumerated index or it does not, the check is a set-membership test over that index, and incrementing a digit produces a key that is absent from it. There is no input on which its correct answer is unknown, so the exercise does not sample its behaviour — it fixes it, in the same way the ledger's parser fixes the verdict on an amount. The claim-verification pipeline has no such authority standing behind it. Its 0.92 is recall on fifty defects somebody constructed, and the four it missed show why that is a bound rather than a property: all four were fabricated section numbers sitting inside sentences whose substance was otherwise correct, which is one shape of defect among several the planting could have produced. A different fifty, planted differently, returns a different figure, and nothing in the exercise says which fifty resemble the defects a customer will find.

The same run prices the other side. Across the 450 untouched answers the pipeline raised 11 contradictions it had not raised before the mutation run, because the verifier is stochastic; 3 of those turned out to be real defects nobody had noticed and 8 were noise. That ratio is the false-positive tax on the check, it belongs in the same report as the recall, and a team that knows both numbers can decide how hard to gate on the result.

Those counts come off one 500-example suite, split two ways. Fifty answers were altered and the other 450 were left alone, which is why the untouched 450 are the ones that price the noise: anything the pipeline raises there is a flag it would have raised on an ordinary run, with nothing planted to explain it. Eleven new flags across 450 answers is under three in every hundred, and 3 of the 11 were real, so the remaining 8 are what running the check costs when it is working correctly — eight answers somebody reads and dismisses, per five hundred, before any genuine defect is acted on. Recall says how much of the problem the check finds. This says how much of somebody's week it takes to use what it finds, and a gate designed from the first number alone gets abandoned because of the second.

Where a canary suite stops helping#

A canary measures whether a scorer responds to the corruptions you thought of, which is a real guarantee and a bounded one: the mutations come out of the same model of how the output can go wrong as the comparator did, so a failure mode nobody imagined is as invisible to the canary as it is to the metric. The defence is to seed the mutation list from real defects as they arrive, so that every escaped bug becomes a perturbation.

A metric that moves is also not thereby a metric that is right. Exact match in the table above responds to all four perturbations, including the benign one, and responding indiscriminately is its defect. The canary is a test for blindness rather than for validity, and it has to be read as a pair: hold on the perturbations the consumer would not notice, drop on the ones it would. A comparator that moves on everything and one that moves on nothing are both unusable, and only the two rows together tell them apart.

The shuffled-pairs number from the scenario is the one instrument in this chapter that is not bounded by what anybody imagined, and the contrast is why it is worth publishing permanently rather than running once. A canary requires a hypothesis: somebody has to have imagined the corruption, written it as a mutation, and predicted which way the number should go. Shuffling requires none. It never asks how the output might go wrong, only what the metric reads on pairs that are certainly not right, and that question can be answered on a suite whose failure modes nobody has thought about yet. The two are complements rather than alternatives. The canary reports whether the metric responds to the defects you can name; the floor reports how much room it has to respond in at all.

A score is a claim that a team will act on for months without re-examining, which makes choosing one an act of system design rather than of reporting. The design work is small and it is all up front: for each output, write down what the consumer does with the value, what equality means under that operation, which failure modes must be distinguished, what the metric reads on unrelated pairs, and what a deliberate corruption must do to the number.

Five lines per field, decided before the scorer is written and checked in beside it. Everything in this topic falls out of those five lines — the normalizer that copies a lookup, the parser imported from its consumer, the overlap metric demoted to a drift alarm, the claim-level check that reads the source. And every failure in it came from skipping them and picking the comparator that made the number look right.