A merged header span relabels every column beneath it

Every failure earlier in this topic left a mark. Chapter 2's fabricated invoice cell broke a total. Chapter 5's fabricated data point broke a stated range or contradicted the prose beside it. Those are detectable because something in the extracted object disagreed with something else in it, and a validator is a program that looks for exactly that kind of disagreement.

This chapter is about a failure class that leaves no mark at all, and that is what makes it distinct from everything earlier in the topic. It breaks nothing checkable, produces values that are individually correct, and survives every totals validator anyone would think to write. The table is well-formed, the arithmetic closes, and the answer built on it is wrong — because the labels moved and nothing in a table constrains a label. Getting there needs one term defined first. A two-level header is a header of two stacked rows, where the upper row groups the lower: two fiscal years across the top, revenue and margin under each. Flattening is the step that turns such a header into the flat list of column names an index can store, one name per value column. The failure is in that step, and the reason it matters more than its size suggests is that a corpus of annual reports and filings is mostly tables with stacked headers.

One source cell, several columns, and the miscount between them#

Work the mechanism forward, because each link is checkable and skipping any of them leaves a rule that cannot predict the next case.

A table on a page has a fixed number of value columns, and the store that receives it holds each value under a name. So the flattener has one obligation: emit exactly as many header labels as there are value columns, in order. That is a requirement, not a design choice — a value with no name cannot be retrieved by name, and a name with no value is a column that does not exist.

Now the source. A merged header cell is not stored as several cells that happen to say the same thing. It is stored once, with a width — a colspan in HTML, a merged range in a spreadsheet, a single wide text box on a rendered page. One cell of storage, several columns of page. That mismatch is the whole problem, and it is a property of how tables are represented rather than a bug in any particular tool.

So the flattener has to expand. It must walk the header and, for a cell of width two, emit two labels; for a cell of width one, emit one. The count it is maintaining is a running column index, and that index advances by the cell's width rather than by one. Advance it by one per cell instead — which is what walking a header row cell by cell does if nobody thought about spans — and every column after the first span is offset. A span of width two consumes one step where it should have consumed two, so everything downstream of it lands one column early, and the labels slide.

The consequence follows without a further step. If the labels slide left by one group, the columns under the second fiscal year receive the first fiscal year's name. Both year columns now carry the same year. A reader asking for that year's revenue matches a column label, gets a number, and the number is the other year's — correctly transcribed, correctly totalled, correctly cited, and attached to the wrong period.

The last link is why nothing fires. Consider what a validator could check. Cell counts: unchanged, since no cell was added or lost. Empty cells: none, since every column got a label. Column totals against the printed total row: they still match exactly, because every value is right and a sum does not consult the header. Types: unchanged. There is no arithmetic invariant anywhere on a financial page that constrains a year label, because a year is not a quantity that participates in any of the page's sums.

That is the belief worth naming, and it is a comfortable one to hold: a table whose columns add up to its printed total feels verified. What makes the comfort misplaced is that a totals check returns "pass" for a correctly labelled table and "pass" for a table whose labels have all slid one column, so its verdict does not move when the thing it is supposed to discriminate moves. A check that returns the same answer whether the system is right or wrong is not weak evidence; it is no evidence. The same goes for a shape validator that asserts a rectangular grid with no empty cells, and for a spot-check that a known figure appears somewhere in the extraction — the figure is there either way.

Which leaves one place a defect like this can be made visible, and it is a decision about the output format rather than about validation. Flattening to markdown is a design choice, and markdown table syntax has no way to say that two columns share a name — a duplicated header reads as ordinary text and nothing downstream can object. Emitting a column_labels list, as the schema in chapter 3 does, is a different design choice with the same information content plus one property: a list can hold the same string twice, and a duplicate in a list is three lines to detect. The requirement is identical under both. The choice of representation is what decides whether the requirement can be checked at all.

What reproducing it would take#

The first question after any incident like this is not "how do we fix the page", it is "how many other pages carry this". Answering it means re-running the cohort — every page extracted by the same model version under the same prompt with a detected two-level header — and diffing. Here is what that needs, against what a pipeline built without this question in mind actually retained.

That table is best read as a set of gaps rather than as an inventory. Three rows say yes, four say no, and the four are not interchangeable with the three. A re-run is a conjunction: it needs the same pixels and the same model and the same instruction, and a conjunction is satisfied only when every term is. So the value of the retained rows is not additive with the missing ones — having the prompt version and the source PDF buys nothing while the image and the servable model are absent, and the row that would have been cheapest to keep is the one whose absence costs the most.

Two of these are fatal on their own. Without the page image or a pinned renderer, you cannot put the same pixels in front of any model. Without a servable model snapshot, you cannot put anything in front of the model that produced the output, which means a clean re-run on the current model demonstrates that the current model handles two-level headers and demonstrates nothing about March.

That distinction is the one that gets skipped, and it is worth separating the two questions precisely, because they use the same evidence and answer different things. Re-running a single page today asks whether the current configuration handles this page class. Re-running a cohort asks how many pages already in the index are wrong. The first is answered by one call; the second is answered only by comparing new output against old output over the whole population that shared the old configuration — which requires that the old configuration still exist somewhere to be run.

What to keep, and what it actually costs#

{
  "page_id": "filing_88213_p63",
  "evidence": {
    "page_image": "s3://ingest-evidence/2026/03/12/filing_88213_p63.jpg",
    "crops": ["s3://ingest-evidence/2026/03/12/filing_88213_p63_table0.jpg"],
    "image_sha256": "9f2c...c41a",
    "render": {"dpi": 300, "encoder": "jpeg", "quality": 85, "colour": "gray",
               "renderer": "pypdfium2==4.30.0"},
    "raw_response": "s3://ingest-evidence/2026/03/12/filing_88213_p63.json.gz"
  },
  "model": "claude-sonnet-4-5-20250929",
  "prompt_version": "extract_page_v4",
  "router": {"rule_version": "r7", "reasons": ["table_figure_area"]},
  "retention": {"class": "customer_document_derivative", "delete_with": "doc_88213"}
}

Each field in that record answers one row of the table above, which is the only principle behind its design. The image and its image_sha256 restore the pixels and prove they are the pixels; the pinned renderer version makes a re-render reproducible if the image itself is ever dropped; the raw_response preserves what the model emitted before any post-processing, which is the difference between diagnosing the model and diagnosing your own parser of its output. The retention block is not diagnostic at all — it is what makes the rest legally storable, and its role is taken up below.

The second row looks wrong on a first pass and is not, and the reason is worth having because it decides how the whole table reads. Crops exist on 22% of pages, so a per-crop size multiplied by a month of pages would overstate the storage several-fold. It does not overstate it, because the 9 KB is already an average over every page, crops and no crops alike — which is the only convention under which the bytes-per-page column can be added downwards at all. The parenthesis names where those bytes come from; the number describes what they cost per page of corpus.

Read that way the column closes. The four item rows sum to a shade under the bolded total, which is shown rounded up to the nearest whole kilobyte, and the monthly column sums to its own bolded total exactly. Multiplying the rounded per-page total by a month of pages gives a tenth of a gigabyte more than the monthly column sum, and that gap is the rounding of the per-page total and nothing else. Naming the residual is what makes it safe to quote either figure: the arithmetic is closed, and the difference between the two routes is a display convention rather than a missing line item.

At object-storage prices with a lifecycle rule moving objects to infrequent-access after 90 days, a year of this is about $9 a month in storage and $6 a month in write requests. Against a $1,101 monthly model bill, the bytes are a rounding error, and any argument against retention framed as a storage cost is not the real objection.

The real costs are elsewhere and are worth naming precisely. You are now storing images of customer documents, which moves this data into a higher classification: the erasure path for a deleted document has to delete the evidence objects too, the access controls have to match the source document's, and any region-residency commitment now applies to the evidence bucket. That is a policy change requiring a real review, and it is the reason most teams do not have this data. The second cost is a re-extraction job that can take a list of page ids, load their stored images, run a named model and prompt version, and diff against what is in the index — perhaps two weeks of engineering, and useless until the day it is needed.

A cheaper variant exists when the source documents are immutable and you can pin the renderer: store the source hash and the exact render parameters, and re-render on demand instead of retaining pixels. That drops storage to under 3 KB per page. It fails when the source is a URL rather than a file, when the renderer's output changes across versions, and when the document was deleted at the customer's request but you still need to characterise the defect — all three of which happen.

Should the page have been refused?#

Under the gates a pipeline runs today, yes, and by a wide margin. The emitted column_labels list would have contained FY2023 · Revenue twice, and the duplicate check is three lines. Where the emitted artefact is markdown, in which duplicate labels are not expressible, nothing could have fired.

Failing closed means a page that trips a gate is held: not indexed, flagged, and visible as a gap. Measured over one month of traffic:

The first row shows its own derivation and the pattern is worth extracting. The vision path takes 8.1% of pages, as the previous chapter's shipped rule establishes, and 1.9% of the pages on that path trip a gate. Multiplying the two shares is the right operation because the second is conditional on the first — "of the pages that went this way, this fraction failed" — and a share of a share is a product. Doing that multiplication lands on the cell.

The table then closes twice over, in two columns that were computed independently. Reading the pages column, the two held rows added together and the recovered row subtracted leave exactly the bolded permanent figure: everything held either comes back or does not, with nothing unaccounted for. Reading the share column, the same addition and subtraction lands exactly on the bolded share. What the closure licenses is a specific reassurance — that the permanent figure is a residue of the three measurements above it rather than a fourth measurement that might have been taken differently, so an argument about it has to be an argument about one of the three.

There is a two-page discrepancy between the columns on the recovered and permanent rows, and it is informative rather than an error. Divide the recovered page count by a month of volume and the share lands a hair above the displayed three decimal places; do the same for the permanent count and it lands a hair below. The pages are the primary measurement and the shares are the rounded derivative, and the two roundings go in opposite directions — which is why the share column still closes exactly despite neither of its last two entries being exact. A residual that cancels in a subtraction is a rounding artefact; one that accumulates would be a missing row.

Roughly 550 pages a month are never indexed, out of 400,000. What that costs is not "0.14% worse answers", because the effect is concentrated rather than spread: a held page is usually a table or a chart, so it is a page users are disproportionately likely to ask about. The right way to state the cost to a stakeholder is that about 550 pages a month become explicit gaps, surfaced in the document view as "1 page could not be extracted" with a link to the source, and answerable only by opening the original.

Set against that, indexing those same 550 pages produces some number of confidently wrong answers with correct-looking citations, and a client acting on a misdated segment figure is the shape of that outcome. A visible gap has a bounded cost that the user can route around; an invisible error has an unbounded one that surfaces as a support ticket weeks later, with the extra property that nobody knows how many more of them are in the index. The previous topic made the same argument about parser output, and the case is stronger here because generative extraction produces failures that look exactly like successes.

"You cannot know in advance which extractions are wrong"#

This objection is correct and it does not do the work people expect of it. None of the gates claims to know which extractions are wrong. Every one of them is either a pre-extraction property of the input or a post-extraction property of the output, and each identifies a class of pages with an elevated defect rate rather than adjudicating an individual page's correctness.

Base rate of at least one defect across all pages is 1.8%. A page in the cross-run-disagreement class has a 46% chance of carrying a defect, which is a 26-fold enrichment, and holding a class with a 46% defect rate is a defensible policy without anyone claiming to know which of the 46 is broken. That is the same logic as any screening test, and the same trade-offs apply: the flagged class contains 54% false alarms, which is what the 546 permanently held pages are mostly made of.

The enrichment figure is a ratio and dividing is the right operation, for a reason worth stating rather than assuming. What a flag is worth is entirely a question of how much it changes your belief, and that is the defect rate inside the flagged class compared against the defect rate you would have assumed without the flag. Dividing one by the other is that comparison, and the units — defects per page in both cases — cancel, leaving a bare multiple. A signal whose flagged-class rate equalled the base rate would divide to one, meaning it told you nothing, and the fact that every row in that table divides to well above one is what distinguishes these signals from the totals check discussed earlier, which would divide to exactly one if you measured it.

The last column is also the only column that needs measuring. The false-alarm share quoted for the cross-run row is what remains after subtracting its defect rate from a whole, so the two are one measurement expressed twice, and every row of that table carries the same pair. The lowest-scoring row is the one that generates most of the permanently held pages, and that is not a defect in the signal — a signal that fires on a large class with moderate enrichment will always contribute more false alarms in absolute terms than a rare, sharp one.

Two of the five rows cost nothing at all, because they are deterministic checks on data already in hand. The resolution floor runs before you spend anything. The schema invariants are arithmetic on the emitted object. The cross-run check is the only expensive one, and it applies to 8.1% of pages, so running it doubles the cost of the vision path on the escalated subset, which is $837 a month here — a real number to weigh rather than an obviously correct purchase.

Correlated failures, and the limit of cross-run agreement#

Cross-run agreement is the broadest signal and its failures are correlated, which is a serious limitation rather than a footnote. Two samples from the same model looking at the same flattened header will flatten it the same way, because the misjudgement is systematic rather than noisy. On page 63, five re-runs against the deprecated snapshot would have produced the same wrong labels five times, and the disagreement gate would have passed the page cleanly. Using two different models narrows this — architectures trained on different data flatten different spans — and it does not close it, because the convention that a merged span applies leftward is a genuine ambiguity in the document rather than an artefact of one model.

Sampling twice is worth thinking about as an analogy to measuring twice with the same instrument, provided the limit of the comparison is stated. Repeating a measurement detects a wobble in the instrument and cannot detect a mis-calibration in it, because a constant offset is present in both readings and cancels out of their difference. Resampling a model has the same structure and the same blind spot. Where the analogy breaks is that an instrument's calibration error is usually constant across everything it measures, while a model's is conditional on the input: the same model that flattens this span wrongly may handle the next one correctly, so the flaw cannot be characterised once and subtracted off. That is why the answer here is sampling pages rather than calibrating a model.

Chapter 5's four residual figures are the same shape of problem: stable across runs, internally consistent, and wrong. There is no automated gate for a page where the model is confidently and reproducibly mistaken, and claiming otherwise is how a validation layer becomes a source of false assurance.

What remains is sampling. A fixed monthly budget — 150 pages per source, stratified by page type, hand-checked against the originals — yields a defect rate per source with an interval attached, and that number is a characterisation of the corpus rather than a guarantee about any page. It is also the only thing in this topic that will tell you when a model upgrade has quietly changed behaviour on a page class, since the day a provider deprecates a snapshot is the day your extraction distribution shifts with no code change on your side.

The through-line of this chapter is that a defect has to be representable before it can be caught, and reproducible before it can be counted. A merged span slid one column is representable in a list of column labels and not in a markdown table, which is a choice made when the output format was picked, months before any wrong answer. It is countable only if the pixels and the model that read them still exist, which is a choice made about retention, also months before. Neither choice looks like a correctness decision at the time it is taken, and both of them are.

You now have the pieces: a specified call with a schema that makes defects representable, page-level ground truth over a stratified sample scored per dimension, a corpus-weighted reading of the results, real per-page cost and wall-clock figures, a router with thresholds swept from the benchmark, gates on both paths, evidence retained per page, and a hold state for what fails. Assembling them in your own corpus starts in the same place it started here — 200 pages, annotated by hand, from the documents you actually ingest.