Objective verification signals

An objective signal, in the sense this chapter uses it, is a check whose verdict is produced by code rather than by a model: a schema that a field either satisfies or does not, an arithmetic identity that either holds within a tolerance or does not, a string that either appears in the source document or does not. It sits on the fourth rung of the independence ladder in chapter 2, where a checker's errors are uncorrelated with the generator's by construction, and it is the only rung where "what does this catch" has an answer that can be written down in advance.

The reason to care is not primarily accuracy. It is that postmortems all ask one question — at what point could the system have caught this, and why did it not — and only a check with stateable coverage can answer it. The worked case below is an invoice extractor, where the fields are typed, the arithmetic is real, and the failure costs money.

Two properties have to hold before that question has an answer, and separating them is what makes "computed rather than judged" more than a slogan. The first is that the verdict is reproducible: the same input yields the same verdict every time, so a verdict seen once tells you what would happen on every other input of that shape. The second is that the verdict is enumerable in advance: the conditions under which the check comes back FAIL are readable off the code without running it, because they are written there as comparisons on values.

A model asked to review its own output has neither property, and the reasons are worth stating separately because they fail independently. Its verdict is sampled from a distribution, so the same page can produce {"ok": true} on one call and a flagged field on the next, which means one observed verdict licenses no conclusion about any other request. And its behaviour on inputs nobody has tried is not written down anywhere — it is a property of the weights, and weights cannot be read the way a comparison can. Both failures point at the same missing thing: there is no artefact you could show someone that says what the check does.

A review step of that kind survives design review because it produces exactly the artefact a real control produces. There is a boolean, it arrives at the right point in the pipeline, it can be logged and alerted on and graphed, and on the documents where the extraction was fine it says so. Everything visible from outside matches. What differs is the provenance of the boolean. One is computed from the page by a rule; the other is generated by the same process whose work is in question, which means a systematic misreading of a faint separator produces a confident all-clear for precisely the reason it produced the wrong total. A control that is at its weakest on the inputs it was installed for is worse than no control, because it also removes the pressure to build one.

The check is the trigger and the acceptance test#

Consider an extractor that reads invoice 4471 from a packaging supplier, whose total of 12,480.00 was booked as 1,248.00. What a control that caught it looks like, in the log, is two lines:

2026-03-11T09:14:22Z invoice=4471 check=totals_reconcile status=FAIL
  line_items_sum=12480.00 extracted_total=1248.00 delta=11232.00
  action=revise field=total
2026-03-11T09:14:26Z invoice=4471 check=totals_reconcile status=PASS
  revised_total=12480.00 action=accept_revision

Nothing in those two lines involves an opinion. The sum of the line items is a number that exists on the page and can be computed without a model; the extracted total either matches it within tolerance or does not; and the revision was accepted because the same computation passed afterwards. That is a control with stateable coverage, and its coverage is exactly the set of invoices where the line items are extracted well enough to sum.

The arithmetic in those lines is worth doing rather than glancing at. The line items on the page add up to 12,480.00, the extractor reported a total of 1,248.00, and the check records the difference between them: 12,480.00 minus 1,248.00 is 11,232.00, which is the delta field. The two figures are the same digits with one group missing, which is the signature of a misread separator rather than of a model inventing a plausible number.

Subtraction is the right operation there for a reason that generalises past invoices. The line-item sum and the reported total are not two independent readings of the same region of the page. They are tied by an identity that holds on every well-formed invoice, because the total is defined as what the parts come to. That identity is what hands the check something the model cannot supply: a second route to the same quantity. The extractor obtained the total by reading one part of the page; the check obtains it by adding up a different part, under a rule that never reads the total at all. Two independent routes to one number is what makes a disagreement between them informative, and it is why the code can be right about the total while the model that produced it is wrong.

The same mechanism locates the check's blind spot precisely, and it is worth stating in the same breath. A mismatch proves the two routes disagree; it does not say which route is wrong. A subtotal misread by one digit group and a line item dropped from a correctly read subtotal both make the line-item sum differ from the subtotal, and the comparison that fires is the same comparison in both cases. It reports that the identity does not hold and hands both numbers to the revision step. Coverage here means detection, not diagnosis.

Three checks carry this pipeline, and they are worth naming before the diagram uses them. Schema validation asks whether every field has the type and shape the downstream system requires. The totals recompute adds up the line items and compares the result with the total the extractor reported. Provenance, the third, asks whether a value the model claims to have read can be found in what it read, which for a string field is a substring test against the page text. The structural move is to use the same objective signal twice. It decides whether a revision happens at all, and it decides whether the revision is kept. Neither decision passes through the model's assessment of its own work.

The revision prompt is narrow. It receives the page text, the field that failed, the failure message, and nothing else — no discussion of the extraction's overall quality, no invitation to improve anything. Scoping it to a single field is what keeps a fix to total from quietly rewriting supplier_name, and the rule that a revision is discarded when the checks still fail is what stops the loop from trading one failure for another.

from decimal import Decimal
from pydantic import BaseModel, Field

class LineItem(BaseModel):
    description: str
    quantity: Decimal
    unit_price: Decimal
    amount: Decimal

class Invoice(BaseModel):
    supplier_name: str
    invoice_number: str = Field(pattern=r"^[A-Z0-9][A-Z0-9\-/]{2,31}$")
    invoice_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    line_items: list[LineItem]
    subtotal: Decimal
    tax: Decimal
    total: Decimal

TOLERANCE = Decimal("0.02")

def run_checks(inv: Invoice, page_text: str) -> dict[str, str]:
    """Field name -> failure message. Empty dict means everything passed."""
    fails: dict[str, str] = {}

    for li in inv.line_items:
        if abs(li.quantity * li.unit_price - li.amount) > TOLERANCE:
            fails["line_items"] = (
                f"{li.description!r}: {li.quantity} x {li.unit_price} "
                f"!= {li.amount}")

    items_sum = sum((li.amount for li in inv.line_items), Decimal(0))
    if abs(items_sum - inv.subtotal) > TOLERANCE:
        fails["subtotal"] = f"line items sum to {items_sum}, subtotal is {inv.subtotal}"
    if abs(inv.subtotal + inv.tax - inv.total) > TOLERANCE:
        fails["total"] = (
            f"subtotal {inv.subtotal} + tax {inv.tax} = "
            f"{inv.subtotal + inv.tax}, total is {inv.total}")

    if inv.supplier_name.lower() not in page_text.lower():
        fails["supplier_name"] = f"{inv.supplier_name!r} does not appear on the page"

    return fails

The failure messages are not decoration. They are the external evidence that the revision round gets to condition on, and their specificity determines whether the round can do anything. Sending back "the total appears incorrect" produced a correct revision 51% of the time; sending back the computed line-item sum, the extracted total and the delta produced one 84% of the time, on the same 340 failing invoices. The difference is that the second message contains the answer implicitly — the re-extraction is being asked to find 12,480.00 on the page, which it can do — while the first asks the model to work out for itself what is wrong with a number it just produced, which is the failure this whole topic is about.

The step the specific message removes is the one the model is worst at. Told only that the total appears incorrect, the model must first decide what is wrong, and the only evidence available for that decision is the same reading of the same page that produced the error in the first place. Told the line-item sum, the extracted total and the delta, no decision is required: the message names the quantity to look for, and the round collapses from a judgement about its own work into a search of the page for a figure that is on it. Recovering a little over four in five is what a lookup scores where a judgement scored barely half.

The supplier_name check is the cheapest useful thing in that function and the one teams most often leave out. It is a substring test against the source text, it takes microseconds, and it catches the failure where the model produces a supplier that is genuinely a customer of the business, spelled correctly, drawn from its own knowledge rather than from the page. That check is a retrieval comparison in miniature: every string field an extractor claims to have read should be required to appear in what it read, and a field that fails it was invented, whatever it looks like.

The reasoning under that substring test is short, and making it explicit is what shows where it stops. A value the extractor claims to have read from the page must have been present on the page, because reading is the only way that claim can be true. Presence is therefore a necessary condition for the claim, and a value that fails it was not read at all — it came from somewhere else, which for a language model means its own priors about which company names exist. Presence is not a sufficient condition, and the check makes no attempt to be one: a string that appears anywhere on the page passes, whatever part of it the string was lifted from. The class this check earns its place on is the one described above, where the name is correctly spelled, genuinely a company, and absent from the document entirely.

What the checks cover and what is left over#

Running this against 1,400 invoices with human-verified fields gives a coverage map, which is the artefact the postmortem question actually wants.

The second column is a share of the errors rather than a share of the invoices, and conflating the two is the ordinary way a coverage map gets misread. The extractor is wrong on 8.0% of invoices, and the percentage against each row is a slice of that 8.0% — not a slice of all invoices. Converting a row into a rate over the whole population therefore means multiplying the two shares together, because a slice of a rate is itself a rate. That is where the last row's 4% share of an 8.0% error rate becomes the 0.3% of invoices quoted below it. The conversion matters because it puts a row on the same footing as the check that would cover it: a check's cost is paid on every invoice, and its return is earned only on the errors, so the two can only be compared once both are expressed per invoice.

Five of the six classes are covered by twenty lines of deterministic code with no tokens and no latency. The last row is not, and that is the honest output of this exercise: a well-formed date taken from the wrong part of the page — the purchase-order date, the delivery date, the date the scan was made — satisfies every check written here and is wrong. Its 4% share of an 8% error rate is 0.3% of invoices, and the options for it are to cross-check against the purchase order in the ERP — the enterprise resource planning system that already holds the company's orders and ledgers — which is another objective signal from a system that has never seen this page, or to route those invoices to a person. What is not an option is believing that a critic prompt handles it, since the same-model critic caught 6% of that class.

Note the shape of that last column. The critic prompt is strongest on malformed dates and currency codes, which is the class the schema catches for free, and weakest on everything expensive. That is the pattern from the independence ladder in chapter 2 showing up in a specific system: it is good at rule-application when the rule is determinate, and poor on the errors that follow from its own priors about what an invoice looks like.

Where a critic prompt still belongs#

None of this makes a separate critic prompt worthless, and it is worth being precise about the job it can hold. It sits on the second or third rung of the independence ladder, it has measurable recall and a measurable false-alarm rate, and its coverage extends to classes no deterministic check covers — tone, completeness, whether an answer addresses the question asked. What it cannot do is be the reason a revision was accepted, because "the model thought the new version was better" does not survive being written down next to a payment error.

The clean division is that objective checks adjudicate and critic prompts triage. A critic prompt is a reasonable thing to put in front of an expensive check, or to use as a signal for sampling outputs into a human review queue, and it is the wrong thing to put in the position where the pipeline decides to accept a changed value.

That division also settles the contract-clause argument that has been running since chapter 1, between a second turn in the same prompt and a fresh call given only the clause and the extracted value. The fresh call is the better of the two for that feature, since the extractor's chain-of-thought is exactly the thing most likely to have carried the error and there is no process error worth catching in a single-field extraction. The choice is worth making and it is not the important decision. Neither setup catches the error class that actually costs money there: a notice period read from the wrong clause, well-formed, plausible, and consistent with what the model expects contracts to say. For that, the extractor should be required to return the character span it read the value from, and a deterministic check should confirm that the span falls inside the clause the classifier labelled as auto-renewal. A second check should confirm that no later amendment in the document set supersedes that clause, which is a document-set query rather than a language task. Both of those are rung-four checks, both are enumerable, and both are things a critic prompt was never going to provide.

The same reasoning applies to a pricing-explanation feature where a self-critique pass fixes 40% of formatting violations and 4% of numerical errors. Those two numbers are not two instances of the same evidence. The 40% is a rule-application result on determinate rules, which is the regime where a model critic performs, and even there a deterministic formatter is cheaper and complete. The 4% is indistinguishable from noise on most eval sets, and the numbers in a pricing explanation have a source of truth: the pricing engine that produced the bill. Recomputing every figure quoted in the explanation from that engine and requiring an exact match is a rung-four check that catches 100% of transcription errors, costs nothing per request, and can be described in a postmortem.

That last claim needs its conditions attached, because "100% of transcription errors" is only true of the class those words name. Requiring every quoted figure to match one the engine produced catches every figure the engine did not produce, which is what transcription means here. It does not catch a figure the engine did produce that has been quoted in the wrong role — the tax presented as the total, last cycle's charge presented as this cycle's — since both sides of the comparison are then genuine engine output and the comparison passes. Closing that gap means checking each figure against the role it was quoted in rather than against the set of all figures, which is a larger piece of work, and knowing that before the check is described as complete is the difference between a stated coverage and an assumed one.

Objective does not mean correct#

Deterministic checks fail in their own ways, and pretending otherwise trades one blind spot for another. A tolerance set too loose passes real errors — 0.02 is fine for currency and useless if the extractor confuses a percentage with a rate. A totals check on invoices that include rounding adjustments or multi-currency lines will fire constantly on correct extractions, and a check that cries wolf gets muted within a quarter, which is a slower and more expensive failure than not having written it. The checks themselves need the same treatment as the critique pass in chapter 3: measured against labelled data, with a false-alarm rate anyone can quote.

Calibrating them is ordinary work with an ordinary answer: run the check suite over the labelled set and record, per check, how often it fires on extractions a human confirmed correct. On the 1,400 invoices the totals check fired on 1.9% of correct extractions, almost all of them credit notes where a negative line item and a rounding adjustment made the arithmetic legitimately not tie, and the fix was an explicit branch for credit notes rather than a looser tolerance. The provenance check fired on 3.1% of correct extractions, driven by suppliers whose page name is a trading style and whose registered name appears in the footer, which normalisation handled. Those two numbers are the check suite's false-alarm rate, they belong in the same document as its coverage, and a check whose false-alarm rate nobody has measured is one release away from being disabled by someone on call.

Preferring a branch to a looser tolerance is the substantive decision in that paragraph, and the two moves are not interchangeable. Raising TOLERANCE reduces false alarms by making the comparison harder to fail, which is the same operation as making it harder to fail on a real error — coverage and false-alarm rate are governed by that single number, so they move together and in the same direction. A branch that recognises credit notes and applies the identity that actually holds for them changes which arithmetic is checked rather than how strictly it is checked. False alarms on credit notes go away and coverage on ordinary invoices is untouched. The test for which move is available is whether the correct extractions that fire the check share a property a predicate can name. When they do, write the branch; when they do not, a looser threshold is the only lever left, and it buys quiet by giving up detection.

Passing every check is also not a claim of correctness. It bounds the error to the classes nothing covers, which for this pipeline is 0.3% of invoices and one known cause. Being able to state that sentence is the entire value of the design: the checks are worth having because they are cheap and they work, and they are worth building this way — same signal as trigger and as acceptance test, coverage written down, false-alarm rate measured — because that is what turns a set of cheap checks into an account of what the system guarantees.

The invoice case is comfortable because the checks are hard pass-or-fail predicates on a single field. Plenty of tasks have no such predicate — a support answer, a summary, a rewritten paragraph — and the checks available are partial and scored rather than binary. The loop still has to decide whether to keep what the revision produced.