Side-by-side markdown outputs are not a measurement

Two paths can turn a page of a document into text. A layout parser reads the character data the file already contains, works out where each glyph sits, and sorts those glyphs into a reading order. A vision model is handed a picture of the rendered page and generates a transcription of it the way it generates any other text. Choosing between them is a decision about money and quality together, and the usual way the quality half gets settled is that somebody runs both over a sample, pastes the two outputs into a shared document, and circulates it for comment.

That artefact is genuinely informative about one thing and silent about everything else, and the gap between those two facts is where extraction programmes go wrong for six months at a time. What follows is what a comparison has to contain before it can decide anything, what it costs to build, and how much of it you are entitled to believe.

What a side-by-side comparison can and cannot see#

A reader comparing two blocks of markdown can perceive whether the prose flows, whether tables are aligned, whether headings look like headings and whether anything is obviously mangled. Those are properties of the writing, and the vision model is a much better writer than the parser by construction, since one of them is producing prose and the other is emitting glyphs in the order it managed to sort them. A side-by-side document measures fluency, and fluency is the one dimension on which the comparison was never in doubt.

It is worth being precise about why this is not a measurement rather than a bad one, because "we compared them and the new one was better" sounds like evidence. A measurement of quality needs three components. It needs a unit of judgement — the thing being scored, one at a time, so that two judgements can be added. It needs a denominator — how many such things there were, so a count of errors becomes a rate that can be compared across samples of different size. And it needs a decision rule fixed before anyone looks — a statement, written down in advance, of what would count as right and what would count as wrong.

Reading two outputs side by side supplies none of the three. There is no unit, because a reader takes in a whole page and forms an impression of it. There is no denominator, because nothing establishes how many facts that page contained in the first place, so a missing value and a page with no values look identical. And there is no advance rule, so the criterion is whatever difference happened to be salient on the screen — which is length, layout and fluency, in that order, because those are the differences a person perceives without reading carefully. The verdict is a function of what stood out rather than of what was true.

That leads directly to the belief worth naming, which is that human review is a safeguard. It is tempting for a good reason: a person looking at the output is the last defence in most pipelines, and in a great many settings a person looking at the output really does catch things. The trouble here is that the signal a reviewer responds to and the property that needs checking are independent of one another. Fluent-and-correct and fluent-and-fabricated look the same on a screen, because a fabricated value is a well-formed number in a well-formed table. A check whose verdict does not change when the thing it is supposed to detect changes is not a weak check; it is not a check at all, and running it more times does not help.

The demonstration that settles this takes an afternoon. Take twenty pages from the sample, strip the labels, hand each pair to three reviewers, and ask only "which is the better extraction". The vision output wins 17 of 20. Then check the same 17 against the printed pages: six of them contain at least one value that does not appear anywhere on the page. Preference and correctness are not merely different measurements, they diverged in the same direction on nearly a third of the pages the reviewers preferred.

What has to be written down before anything is scored#

Ground truth for this problem is per page, and it is not a transcript. A transcript would let you score text and nothing else, and text is the dimension where the two paths differ least. What has to be recorded is a set of relationships, one record per page:

{
  "page_id": "ar2023_p63",
  "page_type": "table",
  "render": {"dpi": 300, "source": "embedded_scan_300"},
  "text": "Segment results\nSegment reporting is presented ...",
  "reading_order": ["heading:Segment results", "body:Segment reporting...",
                    "table:segment_results", "footnote:1", "sidebar:none"],
  "cells": [
    {"row": "Infrastructure", "col": "FY2023 · Revenue",          "value": "4,182"},
    {"row": "Infrastructure", "col": "FY2023 · Operating margin", "value": "11.4%"},
    {"row": "Infrastructure", "col": "FY2022 · Revenue",          "value": "3,970"}
  ],
  "figures": [
    {"id": "fig_12", "caption": "Revenue by segment, FY2019–FY2023",
     "legend": ["Infrastructure", "Cloud", "Services"],
     "y_units": "£m",
     "facts": ["Cloud revenue rose in every year shown",
               "Infrastructure revenue in FY2023 is approximately 4,200",
               "Services is the smallest series in all five years"]}
  ],
  "unreadable_regions": []
}

Each list in that record is a unit of judgement, which is the first of the three missing components supplied. A cell is one scorable thing, a reading-order pair is one scorable thing, a stated chart fact is one scorable thing. Because the record enumerates them, it also supplies the second component: the length of each list is the denominator, so an extraction that emits nothing scores zero rather than scoring nothing.

The cells list is the expensive part and the load-bearing one. Writing 4,182 into a transcript takes a second; writing that 4,182 is the FY2023 revenue of the Infrastructure segment takes a judgement about a two-row header, and it is the judgement that both paths get wrong in different ways. The facts list is discussed at length in the next chapter; note for now that it is a list of assertions rather than a list of data points, because a bar chart with no printed values does not have ground-truth numbers, it has ground-truth claims.

Sampling is stratified rather than random, because a random 200 pages from a corpus that is 94% ordinary prose gives you roughly twelve pages carrying tables and two or three carrying charts, and no useful estimate on either. Four strata of fifty pages each — single-column prose, layout-heavy pages, table pages, chart and figure pages — costs the same annotation effort and produces a usable per-stratum estimate.

It produces a usable one, not a precise one. At n = 50 with a score around 0.8, the standard error on a proportion is about 5.7 points, so a 95% interval is roughly ±11 points. Both of those figures come out of one formula worth carrying, because it tells you what sample size buys before you spend the annotation budget. A score that is a proportion has a spread that depends only on the proportion itself and the sample size: take the score, take what is left over on the other side of 1.0, multiply the two together, divide by the number of pages, and take the square root. With a score of 0.8 the two factors are 0.8 and its complement, and dividing their product by 50 and taking the root gives 5.7 points. The 95% interval is a shade under two of those either side, which lands at ±11. A per-stratum gap of 15 or 20 points is real at that sample size; a gap of 3 points is not resolvable and should be reported as a tie rather than as a direction. Teams routinely quote a two-point difference from a fifty-page stratum as though it settled something, and it is worth writing the interval next to every number in the report so that nobody has to remember.

The formula also explains why the interval is not the same width everywhere. The product of a score and its complement is largest in the middle of the range and shrinks toward either end, so a stratum scoring near the ceiling carries a much narrower interval than one scoring near 0.8 with the same fifty pages. That is why the prose stratum, where both paths score in the high nineties, can support a conclusion about a difference of well under a point, while a stratum in the middle of the range cannot.

Five dimensions, five metrics, scored separately#

Three of those metrics need naming before they can be read. Character error rate is the number of single-character insertions, deletions and substitutions needed to turn one string into the other, divided by the length of the reference; NFKC is a Unicode normalisation that collapses typographic variants so that a curly quote and a straight one do not count as an error. Kendall's tau compares two orderings by taking every pair of blocks, asking whether the two orderings put that pair the same way round, and reporting agreements minus disagreements as a fraction of all pairs, so it runs from minus one for a perfect reversal to one for a perfect match. F1 combines precision — the share of what a path emitted that was right — with recall — the share of what was there that it found — by taking their harmonic mean, which is the average that a low value drags down hard, so a path cannot buy a good F1 by being excellent at one and dreadful at the other.

Five separate numbers, and there is no defensible way to average them. The last row has a column with almost no parser counterpart: the parser emits close to nothing on chart facts, and so emits nothing false at all, and any weighted blend that includes a false-assertion count will reward the path that says nothing. Blending also fails on the first two rows, which move in opposite directions between the paths, so a single number would report their cancellation as parity.

The triple-based table score is the one worth showing in code, because the choice of key is the whole method:

def triples(cells):
    return {(c["row"].strip().casefold(),
             c["col"].strip().casefold(),
             c["value"].strip().replace(",", "").rstrip("%"))
            for c in cells}

def table_f1(gold_cells, pred_cells):
    g, p = triples(gold_cells), triples(pred_cells)
    if not g and not p:
        return 1.0
    tp = len(g & p)
    prec = tp / len(p) if p else 0.0
    rec  = tp / len(g) if g else 0.0
    return 0.0 if prec + rec == 0 else 2 * prec * rec / (prec + rec)

Because the key includes both labels, a value transcribed perfectly but attached to FY2023 · Revenue when it belongs to FY2022 · Revenue scores as one false positive and one false negative, which is what it deserves. A metric keyed on the value alone — or on grid position — would score that page at 1.0, which is precisely how a merged-header defect passes a benchmark. The previous topic showed that flattening a table deletes the binding between header and cell; this metric is what makes that deletion visible in a score rather than in a customer complaint.

Precision and recall are also worth reporting unblended per path, because they separate the two failure kinds this topic keeps returning to. The parser's table errors are almost entirely recall — cells it could not bind, so it emitted nothing for them. The vision model's are split, and its precision losses are cells it emitted that the page does not contain. An F1 of 0.61 reached through precision 0.98 / recall 0.44 and an F1 of 0.61 reached through precision 0.71 / recall 0.54 are different products.

Those two are worth putting through the formula, because the collision is exact rather than approximate and that is the whole argument for reporting the parts. Take precision 0.98 and recall 0.44: their product doubled, divided by their sum, comes to 0.61 when rounded to two places. Take precision 0.71 and recall 0.54: the same operation on those comes to 0.61 as well. One path is emitting very little and getting almost all of it right; the other is emitting most of what is there and getting about seven cases in ten right. Those are different systems with different downstream consequences, and the summary statistic maps them onto the same value. A number that two materially different behaviours both produce cannot be used to choose between them, and no threshold placed on it recovers the distinction.

What the pooled numbers look like#

Over the 200-page stratified sample, pooled across strata:

Read the table row by row and the two paths are not close on any of them, and they are not close in the same direction on any two adjacent rows. The vision model transcribes characters roughly two and a half times worse, orders pages far better, binds table cells much better while inventing some, attaches captions almost perfectly, and is the only path that says anything at all about charts — while emitting roughly one false chart assertion for every three figures.

The table-structure row is a small model of itself, and checking that closure is what licenses reading the other rows at face value. Put the parser's precision of 0.98 and recall of 0.27 through the harmonic mean and 0.42 comes back; put the vision model's 0.71 and 0.54 through the same operation and 0.61 comes back. Both cells are reproduced from the two figures printed beside them, with no residual, so the summary and its parts are the same measurement rather than two measurements that happen to sit in one cell.

One pair of numbers in the paragraph above the table looks as though it should appear in this one and does not, and the mismatch is deliberate. The parser's row here reads precision 0.98 with recall 0.27; the illustration used precision 0.98 with recall 0.44. The second is not a mis-transcription of the first. It is a constructed case — the parser's real precision paired with a recall it does not have — chosen so that its F1 lands on the vision model's actual 0.61. That is what makes the collision demonstrable from figures the chapter already contains rather than from invented ones.

Read the same table by the gaps rather than the levels and the argument against blending gets sharper. The first two rows are locked together in most people's intuition: a path that reads characters accurately is assumed to lay them out accurately. Here they come apart completely and in opposite directions, with the parser ahead on characters and the vision model far ahead on order. Any average of those two rows reports the pair as roughly tied, which is a description of neither path. The last two rows come apart differently: the false-assertion row has no informative parser value to compare against, because a path that emits almost no chart facts cannot be caught emitting false ones, so the parser's 0.00 is a consequence of its 0.02 rather than an independent virtue.

There is no compression of that table into a scalar that a routing rule could safely consume, and the attempt to produce one is where most of these evaluations go wrong. The public write-ups worth reading are the ones that resist it: the Modern OCR vs VLMs for Document Understanding benchmark from CalQuity reports its criteria separately for exactly this reason, and its usefulness comes from the disaggregation rather than from any headline figure.

Finding the fifty pages that are worth annotating#

Fifty hand-annotated table pages is a real budget, and spending it on fifty easy tables measures nothing. Disagreement between the two paths is a free way to find pages where the measurement will be informative, and it costs one run of each over a much larger unlabelled sample.

def disagreement(parser_out, vision_out):
    return {
        "text": char_error_rate(parser_out["text"], vision_out["text"]),
        "coverage_gap": 1 - coverage(parser_out["text"], vision_out["text"]),
        "cell_jaccard": jaccard(triples(parser_out["cells"]),
                                triples(vision_out["cells"])),
        "table_count_delta": abs(len(parser_out["tables"]) - len(vision_out["tables"])),
    }

The reasoning behind that function is a chain worth making explicit, because it is the one step in the method that uses no ground truth at all. Two paths that produce the same output on a page cannot be told apart by any scoring of that page, whatever the ground truth says, since both would score identically against it. So annotation spent on an agreed page buys no resolution. Disagreement is therefore a proxy for informativeness that can be computed before a single page is labelled, which is exactly the property a sample-selection rule needs.

Run both paths over 4,000 unlabelled pages, rank by disagreement within each stratum, and annotate a mix: thirty pages sampled from the top disagreement quartile and twenty sampled uniformly. Those two counts add to the fifty pages a stratum is allowed, so the mix is a split of a fixed budget rather than an addition to it. The uniform half keeps the estimate honest, since a benchmark built only from disagreements would report the two paths as far apart everywhere. The disagreement half is where the annotator's time buys resolution, because pages where both paths produce the same output are pages where ground truth cannot separate them.

This also produces a useful side artefact before any annotation happens. On this corpus, the two paths agreed within tolerance on 88.6% of pages, which bounds the entire question: whatever the benchmark eventually says, it can only be about the other 11.4%, and any claim that one path is dramatically better across the board is already contradicted. The two shares add to the whole with nothing left over, which is what makes it a bound rather than an estimate — there is no third category of page, agreed-or-disagreed exhausts them, so the disagreeing fraction is the entire space in which any difference can live.

What ground truth costs, and how good it is#

The annotation column adds up. Fifty prose pages at 4 minutes, fifty layout-heavy at 9, fifty table pages at 21 and fifty figure pages at 26 comes to 3,000 minutes, or about 50 hours — call it six or seven person-days for a first benchmark, with a domain-literate annotator rather than a general one, because deciding that a column labelled Adj. means adjusted operating profit is a finance question.

That total closes exactly, and the way it closes says something about the column. The four per-page figures add to sixty minutes, so fifty pages of each kind is fifty pages times one hour, and the fifty-hour figure is an identity rather than a rounded estimate. Three of the four match a single row of the dimensions table: prose pages are charged the text-fidelity cost, table pages the table-structure cost, figure pages the chart-facts cost. The fourth does not appear in that table at all, and it is the sum of two rows that do — reading order and caption-and-legend attachment — which is what a layout-heavy page actually has to be annotated for. Reading the column as additive per-dimension cost therefore reproduces every one of the four stratum figures, and that is what licenses using it to price a stratum you have not annotated yet.

The same additivity exposes what the total leaves out. Table and figure pages are charged one dimension each, though a table page also carries text and a reading order that a complete record would have to capture. Fifty hours is the cost of annotating each stratum on the dimension that governs it, which is what the benchmark scores, and it is a floor rather than a ceiling for anyone who wants every dimension on every page.

Double-annotating a fifth of the sample to measure agreement is not optional overhead, because it tells you which of your five numbers you are allowed to believe. On this corpus, two independent annotators agreed on 0.93 of cell triples, 0.97 of reading-order pairs, and 0.79 of chart facts. The chart-fact figure is the honest weak point of the whole benchmark: annotators disagreed about how precisely to state a value read off an unlabelled bar, and about whether a claim like "Services is the smallest series in all five years" is one assertion or five. Any conclusion drawn from chart-fact recall carries that 0.79 underneath it, and a chart-fact gap of five points between two paths is inside annotator noise.

Automating annotation with a second model is the obvious economy and it has a hard constraint: the grader must never be the model under test, and its agreement with human annotators has to be measured on the same double-annotated fifth before its numbers are used. A grader that agrees with humans on 0.71 of chart facts is not a substitute for humans, it is a third noisy annotator.

Extraction is not answers, and pages are not documents#

This benchmark measures extraction, not answers. A page can score 1.00 on every dimension and still produce a system that cannot answer questions about it, because chunking, embedding and retrieval sit between extraction and the user, and the previous topic covered how a perfectly extracted table can still fail to retrieve. Improvements on this benchmark are necessary for downstream quality and nowhere near sufficient for it.

It also does not measure documents. Every metric here is per page, which is the right unit for a routing decision and the wrong unit for questions like "does the extraction of this contract support answering about termination rights", which depend on cross-page continuity — a table split across a page break, a footnote marker on page 12 resolved on page 14. Those failures are invisible to a page-level benchmark and need their own.

And it says nothing yet about cost or latency, which are measured on different pages and by different means, and which turn out to decide more of this than quality does.

The three components a side-by-side lacks are the three this record supplies, and supplying them is most of the work: the unit of judgement is the cell, the pair, the assertion; the denominator is the length of the list those come from; and the decision rule is the ground truth itself, written down before either path was run against it. Everything after that is arithmetic. What the arithmetic returns is five numbers rather than one, and the refusal to collapse them is not fastidiousness — two of those numbers move in opposite directions between the paths, and a sixth number made by averaging them would describe a system that does not exist.

One dimension in the table above behaves unlike the other four. Chart facts have the lowest annotator agreement, the only false-assertion count, and no useful parser baseline to compare against, and those three anomalies have a single common cause: the quantities a chart depicts are not written anywhere on the page as text. The next chapter is about what that does to every check you might want to run.