Your fixtures never included an output built to pass wrongly

A false positive for a verifier is a pass issued on an output that should not have one: the output is well-formed, satisfies every constraint, and is wrong about the thing the feature promised. The claims extractor of chapter 1 produced one on its first attempt — an insurance claim with a transposed policy number, a loss date chosen from two conflicting ones without a note, and a fabricated third line item that made the arithmetic reconcile — and the constraint list had nothing to say about any of it.

The thing worth understanding is not that such outputs exist, which is obvious once one is in front of you, but why the ordinary way of testing a check cannot produce one. A fixture set is the stored example outputs a test suite runs the check against, real ones and broken ones together. When those fixtures are real extractions plus hand-broken variants of them, the set measures the verifier against the distribution it was designed against, and that is the whole difficulty. Follow the reasoning one step at a time. The author of the check wrote a constraint for each way an output can go wrong that they thought of. An output that passes and is wrong is one they did not think of — if they had, there would be a constraint for it, and the output would be rejected. The fixtures were also written by that author, out of the same picture of the problem. So the set of outputs a fixture author would produce and the set of outputs that slip past the check are disjoint. Carelessness has nothing to do with it; the two sets are defined against each other. Testing harder inside the fixture set cannot help; the set has to come from somewhere else.

That somewhere else is an adversarial corpus, meaning one built deliberately by someone whose goal is to get past the check rather than to demonstrate that it works. This chapter is about how to build one, what it measures, and what it does not.

The word points in two directions, so say which one you mean#

Two conventions collide here and teams lose hours to it in review meetings. If the positive event is the verifier passes, a false positive is a pass on a wrong output, which is what the previous section means and what this chapter is about. If instead you think of the verifier as a defect detector, the positive event is a defect was found, and a false positive is a spurious rejection of a perfectly good output. Both usages are common and both name real problems. The habit that survives contact with a room full of people is to drop the word in favour of the two unambiguous ones — a wrong pass and a wrong rejection — and to keep both columns on the same report, because as the last section of this chapter shows, inside a retry loop they are connected.

Four ways an output gets a pass it does not deserve#

The adversarial corpus is organised by mechanism, because each mechanism suggests its own construction technique.

The four are not variations on one idea. A vacuous constraint fails to separate anything, so its verdict carries no information about any output. A parser quirk separates fine and separates the wrong population, because the bytes the check reads and the bytes the consumer reads are decoded differently. Satisfy-by-construction is a constraint whose two sides are both under the generator's control, so making it true costs nothing. A degenerate boundary is a constraint that is correct about its range and silent about whether a value in that range is meaningful. Knowing which of the four you are looking at tells you what a fix would have to change.

The parser row is the one that surprises people who write the check in one language and consume the output in another. Two of these are properties of Python that most working engineers have never had reason to notice.

import re
POLICY_RE = re.compile(r"^POL-\d{8}$")

POLICY_RE.match("POL-４０１１８５２６")   # matches: \d is any Unicode decimal digit
POLICY_RE.match("POL-40118526\n")      # matches: $ also matches before a final newline

STRICT = re.compile(r"\APOL-[0-9]{8}\Z")
STRICT.match("POL-４０１１８５２６")      # None
STRICT.match("POL-40118526\n")         # None

Both lines are worth taking apart, because the general lesson is that a pattern encodes a decision its author probably did not make deliberately. In a Python str pattern, \d means "any character with the Unicode decimal-digit property", which includes the full-width forms used in Japanese and Chinese typography as well as the digits of a dozen other scripts. The team believed they had written "eight ASCII digits" and had written "eight things Unicode calls digits". The full-width string carries eight characters that str.isdigit() also agrees are digits, satisfies the constraint they believed they had written, and finds nothing in the policy table. [0-9] in the strict pattern names the eight characters they meant.

The second line is a different decision hiding in a different symbol. $ in Python's re matches at the end of the string and immediately before a newline at the end of the string, so "POL-40118526\n" matches a pattern that looks anchored. \Z matches only at the very end and admits no trailing newline; \A is the corresponding start-of-string anchor that ^ is not, once multiline behaviour is in play. The trailing-newline case is worse in a quiet way than the full-width one, because it passes verification, gets stored, and matches nothing on lookup while looking identical to the correct value in every log line and every terminal.

The duplicate-key case belongs to the same family and needs two components to bite, which is worth following as a chain rather than accepting as a claim. A model emits {"total_claimed": 8480.00, "line_items": [...], "total_claimed": 12000.00}. Python's json.loads reduces repeated keys to a single entry holding the last value, so the object the verifier sees has total_claimed of 12,000. The verifier sums the line items, compares against 12,000, finds a mismatch, rejects, and the loop retries — which is the good outcome, and it is the outcome as long as one parser is involved. The failure needs a second one. If the storage layer parses the same bytes with a library that keeps the first occurrence, and some streaming parsers do, then the value that was verified and the value that was persisted are different numbers read from identical bytes, and no log line anywhere records a discrepancy because no component saw two values. The defence follows directly from where the chain breaks: parse once, at the edge, with a strict parser that rejects duplicate keys and non-standard tokens, and pass the parsed object to both the verifier and the consumer, so that there is only ever one decoding to disagree about.

import json

def strict_load(raw: str) -> dict:
    def no_dupes(pairs):
        seen = {}
        for k, v in pairs:
            if k in seen:
                raise ValueError(f"duplicate key: {k}")
            seen[k] = v
        return seen
    return json.loads(raw, object_pairs_hook=no_dupes, parse_constant=_reject)

def _reject(token: str):
    raise ValueError(f"non-standard JSON token: {token}")   # NaN, Infinity, -Infinity

Measuring how much each constraint actually rules out#

A constraint list looks like a wall and behaves like a sieve. That is an analogy and it is worth stating what it buys and where it stops. What it buys is the right expectation about leakage: a sieve is not a failure when something gets through, that is what a sieve does, and asking whether the list "works" is the wrong question. Where it breaks is on which things get through. A sieve's holes are a fixed size, so what passes is whatever is small, and that is a property of the things being sieved. A constraint list's holes are shaped by what its author did not imagine, so what passes is whatever is shaped like the author's blind spot — a property of the author rather than of the outputs. Keep the leakage and drop the size ordering, or the picture will tell you that the dangerous outputs are the marginal ones, and they are not.

The way to see which holes are load-bearing is to run each constraint separately against both corpora and record its rejection rate. Take the malformed fixtures as one column and 80 hand-built adversarial outputs as another, and the difference between the columns is the number worth arguing about.

Add the first column up before reading anything else. It comes to exactly 60, which is the size of the malformed corpus, and the equality is not a coincidence — it means each hand-broken fixture is rejected by exactly one constraint. That is what happens when a fixture is made by taking a correct output and breaking one thing, which is how anybody builds them. So the first column is not a measurement of the check at all. It is a census of what its author decided to break, redrawn as a bar chart, and a constraint scores high in that column precisely when the author found it easy to imagine violating.

The second column does not add up to its corpus size, and the constraints in it overlap, so the arithmetic gives a range rather than a number. The outputs that survive the whole list are at most the 29 that the source constraint alone lets through, and at least what is left of the 80 once the column is totalled, which would be the count if no two constraints ever rejected the same output. The true figure sits between those, and where exactly depends on overlap nobody measured. A range is the honest output here, and it is more useful than a point estimate would be, because both ends of it say the same thing: a substantial fraction of outputs built to pass wrongly do exactly that.

Now read the two columns against each other rather than down. They rank the constraints in opposite orders. The required-fields rule is the strongest constraint by the fixture column and the weakest by the adversarial one, because fabrication fills every field — a required field is a defence against omission and fabrication is not omission. The source constraint is the weakest by the fixture column, scoring zero, and the strongest by the adversarial one by an order of magnitude. That inversion is the finding, and it explains why the source constraint did not exist in the original verifier: against real extractions it never fires, so on the only evidence anybody had it looked like a constraint that does nothing.

The 51 has a licence and a residual and both matter. What it licenses is a floor under what the constraint is worth: adding it rejects at least that share of this class of wrong output, which is a lower bound on the benefit and can be put next to the cost of computing it. What it leaves is the 29 the constraint lets through, and those are not a random remainder. They are the outputs that select a real string from the wrong place on the document — a genuine policy number belonging to a different claim, a date lifted from the wrong box — and the constraint cannot touch them by construction, because they consist entirely of strings that do appear verbatim in the document. That residual is the class the previous chapter priced, and closing it needs a constraint that reads the input again by an independent route.

The same measurement settles the tempting fix for the text-to-SQL case from that chapter. Adding a non-empty-result rule to a check that already requires successful execution rejects almost nothing on an adversarial corpus, because the reported failures were wrong numbers rather than missing ones and a query joining the wrong table returns plenty of rows. It converts one vacuous constraint into a slightly less vacuous one, and the honest way to discover that costs an afternoon of writing twenty wrong-but-runnable queries by hand and watching all twenty pass.

Where the adversarial outputs come from#

Three sources, in ascending order of value. Hand construction is the cheapest and gets you the mechanism table above: take a correct output, and for each constraint ask what the smallest edit is that keeps the constraint satisfied while making the output wrong. Past incidents are better because they are drawn from the real failure distribution — every wrong answer a user has ever reported is an adversarial fixture with a free label attached, and a team that files them into the corpus as they arrive accumulates a set that no amount of imagination would have produced. The third source is the loop's own history, which is the subject of the next chapter and turns out to supply these outputs faster than anyone can write them.

Who writes them changes what gets written, and the reason is the same argument that opened this chapter. An engineer constructing adversarial cases against a verifier they wrote themselves is drawing from the same picture of the problem that produced the gaps, so their cases land inside the region the constraints already cover. Two people break the symmetry: one brings the constraint list, the other brings a correct output and tries to break it while keeping every constraint satisfied, and neither can see round their own blind spot but each can see round the other's. Twenty cases is a realistic yield for an hour, and the first hour finds the most, because the cheapest exploits are the ones the check's author never had to think about.

The suite that holds them looks like an ordinary test file, with one structural difference: each case carries the truth about the output alongside the expected verdict, so the file records both what the verifier does and what it should do.

ADVERSARIAL = [
    Case(id="fabricated-contents-line", output=FABRICATED,
         truth="wrong", expect="pass", note="known gap: no source check"),
    Case(id="fullwidth-policy", output=FULLWIDTH,
         truth="wrong", expect="reject"),
    Case(id="negative-credit-line", output=CREDIT_NOTE,
         truth="correct", expect="pass", note="regression: was rejected in v3"),
]

@pytest.mark.parametrize("case", ADVERSARIAL, ids=lambda c: c.id)
def test_verifier_verdicts(case):
    problems = verify(case.output)
    assert ("pass" if not problems else "reject") == case.expect, problems

Cases where truth is "wrong" and expect is "pass" are the interesting ones, because they document a known gap rather than a bug, and they turn into a red test the day someone strengthens the verifier enough to close it. A gap you have written down is a gap the next engineer can price.

What the corpus does not tell you#

An adversarial suite is not a security exercise, and framing it as one leads teams to the wrong threat model. There is no attacker choosing inputs, no incentive to evade, and nothing to defend; the outputs that slip through are produced by a model doing ordinary next-token prediction under pressure from a constraint list, which is why they cluster around plausible-looking values rather than around exotic encodings. The full-width digits case is in the corpus because a document scanned from a Japanese form produced them, not because anyone was being clever.

It is also not a measurement of how often wrong passes happen in production, and the reason is in the construction. The corpus has no distribution — 80 hand-built outputs weighted by nothing, chosen because they are interesting. A rate needs a denominator drawn from real traffic, and this denominator was drawn from an afternoon. So the corpus tells you which mechanisms your check is blind to, and it can tell you that a change closed a mechanism, and it cannot tell you that 3% or 11% of shipped outputs are wrong. That figure needs labels on a representative sample of real accepted outputs, which is a separate and considerably more expensive exercise. The two are complementary in a precise way: the corpus tells you what to look for and the sample tells you how much of it there is.

The third thing it does not do is stay valid. Every constraint added or loosened invalidates part of the corpus, since a case constructed to slip past version 4 may be rejected outright by version 7 for an unrelated reason, and a case that documented a known gap may now be a live regression. Re-running the whole suite on every verifier change is what keeps this honest, and it is cheap enough — the claims team's 80 cases run in under two seconds, evaluating every constraint separately against every case rather than short-circuiting on the first failure — that the only real cost is remembering to update the truth labels when the definition of correct changes.

Wrong rejections cost more inside a loop than outside one#

The third case in the snippet above is a correct output: a claim with a negative line item, because the insurer had already paid a €900 advance and the adjuster recorded it as a credit. Version 3 of the verifier rejected it, on the reasonable-sounding rule that amounts must be positive.

Outside a loop, that rejection is a visible error and someone triages it. Inside one it behaves quite differently, and the mechanism is worth tracing because it is not obvious and it runs in the opposite direction to intuition. The loop responds to a rejection by asking the model to try again, with the constraint list attached as text. The model reads that amounts must be positive. The correct output has an amount that is not positive. The cheapest edit that satisfies the stated rule is to remove the offending line, and removing it makes total_claimed disagree with the line items, so the model recomputes the total to match. The second attempt now satisfies every check, and the claim is overstated by €900. Nothing malfunctioned. A tight constraint was stated to a generator that had been asked to satisfy it, and it was satisfied.

The team measured this on a batch of 1,000 documents after they found the first one. Twenty-three correct outputs were rejected by constraints that were too tight, and fifteen of them came back on a later attempt in a form that passed and was wrong. The other eight exhausted their attempts and were seen by a person, which is the outcome a wrong rejection is supposed to have. So the ledger has two entries and only one of them appears on any dashboard: outside a loop, a wrong rejection costs a retry or a human triage; inside one, it applies steady pressure on a correct output until it deforms into a passing one, and the deformed output is indistinguishable from a first-attempt pass.

That connection is why both error columns belong on the same report. A constraint tightened to close a wrong pass has to be checked against the correct outputs it starts rejecting, because a fraction of those rejections will not stay rejections. It is also the reason the two-corpus measurement above has to be re-run after every tightening: the adversarial column tells you what the new constraint buys, and the fixture column of real extractions tells you what it costs, and inside a loop the cost has a multiplier the fixture column cannot see.

None of the eighty adversarial outputs required a model to produce. They were written by hand, over an afternoon, by someone trying to get past a check — which is worth holding onto, because the cheapness is the point of the exercise and also its limit. An afternoon buys twenty or eighty cases. What the next chapter takes seriously is that the loop is doing the same thing continuously, at every attempt, on every input, without being asked and without anybody scheduling an hour for it.