Per-document OCR rate reveals a vendor that started scanning

A parser reaching a page can do one of two things with it. If the file already holds real character data for that page — a text layer, the actual codepoints the producer wrote in, rather than a picture of the marks — the parser reads them out. If there is no text layer, or the text layer is empty, the only remaining route is optical character recognition: rasterise the page into an image and run a recogniser over the pixels to guess which characters the shapes represent. Both routes end in a string of text, and both end in a parse that raised no exception. A pipeline that records only whether an exception was raised therefore cannot tell you which route ran, and the route is the difference between characters that are certainly right and characters that are probably right.

That distinction matters because it is not stable over time. A supplier who has been exporting documents straight out of a word processor produces pages with text layers; the same supplier who starts printing, signing and scanning those documents produces pages without them, and nothing about the filename, the file size, the extension or the delivery mechanism changes. Parse metrics — one record per document per parse, describing what the parser found and what it had to do about it — are how that change becomes visible. The instrumentation in this chapter is a single row written at parse time, and the questions it answers are all of the same form: has this source's shape changed, and what is that costing.

One row per document, emitted at parse time#

The classifier from chapter 2 already computes per-page routing decisions. Instrumentation is mostly a matter of keeping them, aggregating to the document, and writing a row.

import statistics, time
from dataclasses import dataclass, asdict

@dataclass
class ParseMetrics:
    doc_id: str
    source: str                 # the vendor or connector, not the file path
    ingested_at: float
    parser_version: str
    producer: str               # PDF /Info /Producer, "" if absent
    pages: int
    ocr_pages: int
    blank_pages: int
    failed_pages: int
    chars_per_page_p50: float
    tables_detected: int
    parse_ms_per_page: float
    ocr_cost_usd: float

    @property
    def ocr_rate(self) -> float:
        return self.ocr_pages / self.pages if self.pages else 0.0


def parse_and_measure(path, doc_id, source, parser_version) -> ParseMetrics:
    t0 = time.perf_counter()
    doc = open_pdf(path)
    page_stats, failed = [], 0
    for page in doc:
        try:
            page_stats.append(classify_and_extract(page))
        except ParseError:
            failed += 1
    elapsed_ms = (time.perf_counter() - t0) * 1000
    ocr_pages = sum(1 for p in page_stats if p.route == "ocr")
    return ParseMetrics(
        doc_id=doc_id, source=source, ingested_at=time.time(),
        parser_version=parser_version, producer=doc.metadata.get("producer", ""),
        pages=len(doc), ocr_pages=ocr_pages,
        blank_pages=sum(1 for p in page_stats if p.route == "blank"),
        failed_pages=failed,
        chars_per_page_p50=statistics.median([p.chars for p in page_stats] or [0]),
        tables_detected=sum(p.tables for p in page_stats),
        parse_ms_per_page=elapsed_ms / max(len(doc), 1),
        ocr_cost_usd=ocr_pages * 0.0015,
    )

The one derived quantity in that record is ocr_rate, and it is worth saying exactly what is divided by what. The numerator is ocr_pages, the count of pages in this document that the classifier sent down the recognition route. The denominator is pages, which is len(doc) — every page the file contains, including the ones that failed. Dividing by the document's own page count is what makes the number comparable between a two-page cover letter and a two-hundred-page agreement: it converts a count, which is dominated by how long the document happens to be, into a proportion, which is a statement about how the document was produced. Length is exactly the property you are not asking about, so length is exactly the property the denominator has to remove.

Choosing that denominator has a consequence the record does not announce. A page that raised a ParseError is counted in failed but never appended to page_stats, so it can never be counted in ocr_pages — yet it is still in pages, because pages is the length of the document rather than the length of the successful sample. A document with failed pages therefore reports an ocr_rate that is too low, and the size of the understatement is exactly the failed share. That is the reason failed_pages is carried as its own field and alerted on separately rather than folded into the rate: the two signals are complementary by construction, and the rate is only a complete description of a document when the other is zero.

The four route counts are the other place to read the record carefully, because one of them is missing. ocr_pages, blank_pages and failed_pages are stored; pages that were read straight out of a text layer are not. They are the residual — pages minus the other three — and naming that residual is what licenses leaving it out of the schema, since a value recoverable by subtraction adds no information. The subtraction is exact only if the classifier's routes are exhaustive and mutually exclusive, so that every page lands in one bucket and no page lands in two. If a later version of the classifier gains a fourth route and no field is added for it, the subtraction silently starts returning text-layer pages plus that new route, and nothing in the row will say so.

There is one degenerate case, and the record handles it in two places at once. A document with no pages at all makes pages zero, so ocr_rate returns 0.0 from its guard rather than dividing by zero, and statistics.median is fed [0] by the or [0] fallback rather than raising on an empty list. The result is a row reporting an OCR rate of zero and a median character count of zero. A perfectly healthy born-digital document also reports an OCR rate of zero, which means ocr_rate on its own has no value that says "there was nothing here" — the pair of signals discriminates where neither member does.

Two design choices in that record matter more than the rest, and both are choices rather than requirements — the requirement is only that a source's change of shape becomes visible. source has to be the logical origin — the vendor, the connector, the mailbox — rather than a bucket prefix or a filename pattern, because every question worth asking of this data is asked per source. And the rows are facts about one parse of one document, never overwritten, so parser_version lets you compare the same document across parser upgrades and ingested_at lets you see when a source's shape changed.

Why the average has to be taken per source#

Rolling per-document rates up to something a person reads means averaging them, and the grouping chosen for that average decides what the average is capable of detecting. Grouping per source is the load-bearing decision here, and the reason is arithmetic rather than taste.

An average taken across every source at once is a weighted mean: each source contributes its own rate multiplied by its share of the documents in the window, and those shares sum to one. The most a single source can move that mean, by flipping from a rate of zero to a rate of one, is its own share. A source supplying a tenth of the week's documents can go from entirely born-digital to entirely scanned and shift the corpus-wide figure by a tenth, and no more. A threshold set above that share therefore returns the same verdict — nothing to see — whether the source flipped completely, flipped halfway, or did not flip at all.

That is the stumble worth naming, because it is a defence that feels like one and is not. A corpus-wide OCR-rate alert reads as a general safety net: it watches everything, so surely it watches this. What it actually does is hold its verdict constant while the thing it is supposed to discriminate varies across its entire range, and a check whose output does not move when its subject moves carries no information about the subject. The same defect is what made the success dashboard useless. Reporting the share of parses that raised no exception returns a healthy number whether the pages were read from text layers or guessed from pixels, so the metric is constant across precisely the distinction it is being trusted to make.

Grouping per source removes the weighting. Inside one source's rows that source's share is one, so a complete flip moves its group mean by the full distance and any threshold below that distance fires. This is also why the grouping cannot be finer than the process that produces the documents: two suppliers who change process on different days will cancel each other out in a shared bucket, and one supplier split across two connectors will halve its own signal.

Reading a step change out of a weekly rollup#

With rows in a table, the question becomes a two-line query. This one groups by source and week over a recent window.

SELECT source,
       date_trunc('week', to_timestamp(ingested_at)) AS wk,
       count(*)                       AS docs,
       round(avg(ocr_pages::numeric / pages), 3) AS ocr_rate,
       round(avg(chars_per_page_p50))            AS chars_p50,
       round(sum(ocr_cost_usd), 2)               AS ocr_cost
FROM parse_metrics
WHERE ingested_at > extract(epoch FROM now() - interval '10 weeks')
GROUP BY 1, 2
ORDER BY 1, 2;

 source    |     wk     | docs | ocr_rate | chars_p50 | ocr_cost
-----------+------------+------+----------+-----------+---------
 meridian  | 2024-02-26 |   61 |    0.021 |      2380 |     0.19
 meridian  | 2024-03-04 |   57 |    0.018 |      2411 |     0.16
 meridian  | 2024-03-11 |   64 |    0.968 |      1902 |    78.44
 meridian  | 2024-03-18 |   59 |    0.994 |      1876 |    74.10
 halloway  | 2024-03-11 |  103 |    0.007 |      2502 |     0.11

Two suppliers appear here: meridian, a legal-services source sending executed contracts, and halloway, shown for the same week as a control. The ocr_rate column steps from 0.018 to 0.968 in a single week, which is a change no gradual drift produces and no threshold tuning is needed to see. The chars_p50 column corroborates it, p50 being the median — the value half the pages fall below — and it drops from about 2,400 characters a page to about 1,900: OCR of a printed page typically yields ten to twenty percent fewer characters than the born-digital original, because headers, page furniture and low-contrast small print recognise poorly or not at all. The cost column turns the same event into a finance number — $78 a week, against $0.19 — which is often the fastest way to get the change taken seriously.

The two columns corroborate each other precisely because they are not two views of the same quantity, and the gap between how far each moves is the evidence. The rate multiplies by roughly fifty; the character median falls by about a fifth. A routing count and a per-page yield have no reason to move together by the same proportion, so their agreeing on the direction and the week is an independent confirmation rather than a restatement. If they did move in lockstep, one of them would be redundant.

The observed drop sits at the top of the ten-to-twenty-percent band, and a little past it, which is worth reading rather than smoothing over. That band describes what recognition typically costs a page; a single supplier's contracts are one draw from that distribution, and the residual above the band is attributable to the causes already named — this source's headers, page furniture and small print happen to be among the parts that recognise worst.

The cost column is a re-denomination rather than a fifth signal. ocr_cost_usd is ocr_pages multiplied by a constant, and that constant closes exactly against the price quoted in prose further down this chapter: at a rate per thousand pages the per-page figure in the dataclass is the same number. So the weekly cost is total recognised pages times a fixed price, and it can detect nothing that the page counts could not. Its whole value is that it states the event in a unit a budget holder acts on. It is also why cost cannot be recovered from the rate column alone: the rate is an unweighted mean over documents, giving a two-page file the same vote as a two-hundred-page one, while the cost is proportional to the pooled total of recognised pages. Converting between them needs pages per document, which this rollup does not carry — and that is a reason to select both rather than a discrepancy between them.

The producer field gives independent confirmation when it is populated, since it moves from Microsoft® Word for Microsoft 365 to Canon iR-ADV C5560 III on exactly the same boundary. It is a weaker primary signal because plenty of producers leave it empty, plenty of tools rewrite it, and a document that passes through a redaction or watermarking step carries that tool's name rather than the original. Use it to explain a step change that ocr_rate has already detected.

The shapes these signals take#

Reading the table above requires knowing what normal looks like, and normal is specific to each signal rather than being a single band.

This table is a small parameter model, and it reproduces every cell of the query output above without being tuned to it. The healthy ocr_rate mass near zero covers meridian's two early weeks and halloway's control row; the scan-heavy band covers both of meridian's later weeks; the healthy character band contains all three born-digital medians and the scan-heavy band contains both post-flip medians. Five observed rows, ten predicted values, no misses. What that licenses is the strong reading of the diagnosis: identifying meridian as a source that has started scanning is a classification against bands stated in advance, not a judgement call about where to draw a line after seeing the data.

The rightmost column earns its place by not being derivable from the others. The pathological-file cell for chars_per_page_p50 — near zero with ocr_rate also zero — is exactly the degenerate branch in the dataclass, where an empty document returns a rate of zero from the guard and a median of zero from the or [0] fallback. The table and the code arrive at that pair by routes that share no step, one from what a real corpus looks like and one from what the arithmetic does when handed nothing.

The bimodality of ocr_rate is the property to internalise, because most documents genuinely are all one thing or all the other. Documents in the middle of the distribution are the hybrids — the born-digital agreement with a scanned signature rider, the report with a photographed appendix — and they are worth a dedicated query rather than an average, since averaging them away is exactly how the Vantis file from the last chapter stayed invisible.

SELECT doc_id, source, pages, ocr_pages,
       round(ocr_pages::numeric / pages, 2) AS ocr_rate
FROM parse_metrics
WHERE pages >= 5
  AND ocr_pages BETWEEN 1 AND pages - 1
  AND ocr_pages::numeric / pages < 0.5
ORDER BY ingested_at DESC;

The three conditions in that WHERE clause are doing three different jobs. The page floor discards short documents, where a single scanned page is a large share of a small denominator and the rate is noise. Requiring ocr_pages strictly between one and one less than pages keeps only genuine mixtures, excluding both poles of the bimodal distribution in a single expression. The rate ceiling then keeps the half of the mixtures nearer the born-digital end, which is where a scanned insert hides inside a document that otherwise looks fine.

A pathological file has a different fingerprint again, and parse_ms_per_page is where it shows. On one corpus the median text-path document parsed at 11ms per page and one document parsed at 34,000ms per page: a 6-page engineering drawing set where each page held roughly 180,000 vector path operations, which the table detector walked exhaustively looking for ruling lines. That single file consumed 3.4 minutes of the ingestion run and produced 240 characters. Nothing failed; the run simply took longer, which on a nightly batch is invisible until the batch stops finishing before the morning.

Those two figures close on each other exactly, and the closure is worth checking because it rules something out. Six pages at thirty-four thousand milliseconds each is a shade over two hundred thousand milliseconds, which is the stated wall clock to the reported precision. The per-page normalisation is therefore not manufacturing the outlier out of an ordinary rate spread over an unusually long document — the file really is six pages, and each one really is slow. Character yield says the same thing from the other side: two hundred and forty characters across six pages is a per-page figure far below even the slide band in the table above, so a file consuming minutes of compute returned less text than a title slide.

The alert that catches this is a ratio rather than an absolute, because absolute latency depends on hardware and on the mix of routes. Flagging any document whose parse_ms_per_page exceeds forty times the trailing median for its source and route catches the drawing set, catches the file with a 12,000 × 9,000 pixel embedded scan, and catches the PDF with a corrupt font that the extractor rebuilds on every page, while staying quiet through an ordinary shift from the text path to the OCR path — because that shift moves the median for the OCR route rather than creating an outlier within it. The drawing set clears that multiple by roughly two further orders of magnitude, which is the margin that makes the threshold uncontroversial: any factor from a few dozen upwards would have caught it.

What to alert on, and what merely to look at#

Most of these signals belong on a page somebody reads weekly rather than in a pager rotation. Three deserve to fire.

A per-source step change in ocr_rate is the first: compare each source's weekly mean against its trailing eight-week mean and alert on a move of more than 0.15 with at least twenty documents in the week. That rule fires on the Meridian change with a margin of 0.95 and does not fire on the ordinary week-to-week variation of a mixed source, which on this corpus had a standard deviation of about 0.04. The minimum document count is what stops a source that sends three files in a quiet week from paging someone.

Both numbers in that rule are recoverable from figures already on this page, which is what a threshold should be. The move of 0.95 is the difference between the post-flip rate and the pre-flip rate in the query output, so the stated margin is the table's own step read back out rather than a separate measurement. The threshold of 0.15 sits at close to four times the ordinary week-to-week spread, far enough above the noise that routine variation cannot reach it and far enough below a complete flip that the flip clears it many times over. Every week in the output above carries well more than twenty documents, so the minimum count never suppresses this event; it exists for the quiet weeks that are not shown.

The second is failed_pages greater than zero at all, since a page that raised an exception is a page whose content is definitively absent from the index. It happens rarely enough to be worth a ticket each time, and the OCR timeout that swallowed page 12 of the Vantis agreement is precisely this signal, which had nowhere to be reported at the time.

The third is cost, aggregated per source per month against a budget rather than a baseline. OCR at $1.50 per 1,000 pages sounds negligible per document and becomes a line item at corpus scale; the Meridian change alone was a $340-a-month increase that nobody had approved, and a hosted layout parser applied to the same volume would be several times that. The monthly figure is the weekly cost column carried up over a month's worth of weeks, and it reads as an increase rather than a total because the same source's weekly cost before the change was pennies.

Retention deserves a decision rather than a default. These rows are small — under 300 bytes each — so a corpus of 40,000 documents re-ingested monthly produces about 12 MB a year, which is cheap enough to keep indefinitely and valuable enough to justify it. The comparison you will want in eighteen months is this quarter's ocr_rate and chars_per_page_p50 for a source against the same source two years ago, and that comparison is impossible if the table is a rolling ninety-day window. Keeping every parse rather than the latest per document is what makes a parser upgrade evaluable after the fact, since the same documents appear twice under two parser_version values.

Everything else in the table is diagnostic context, read when something else has already gone wrong. Putting chars_per_page_p50 or tables_detected on an alert produces noise, because both move for legitimate reasons — a source starts sending appendices, a template gains a summary table — and neither indicates a defect on its own.

Routing, throughput and cost are not correctness#

Instrumentation of this kind reports on routing, throughput and cost. It does not report on correctness, and treating it as a quality dashboard is the mistake it invites. A source can run at ocr_rate 0.0, zero failed pages, 2,600 characters per page and 11ms per page while emitting two interleaved columns in every chunk, exactly as chapter 1 described, and every number here will look excellent. The metrics that would catch that — reading order fidelity, table binding accuracy — cannot be computed without ground truth, which means a human comparing output to the printed page, which is the sampled characterisation from chapter 6 rather than anything a pipeline emits about itself.

The reason is structural rather than a gap somebody could close with another field. Every quantity in the parse record is computed from the parser's own output, so each one is a statement about what the parser did and none is a statement about whether what it did was right. Reading order is a property of the relation between the output and the page, and the page is not an input to any of these numbers. A metric derived entirely from one side of a comparison cannot score the comparison, however many such metrics are added.

The honest division of labour is that instrumentation tells you when something changed and what it costs, and sampling tells you whether it is right. Meridian's switch was a change, so instrumentation was the correct instrument and would have caught it in week one instead of week seven. A parser that has always mangled two-column layouts is not a change, so no amount of instrumentation will surface it, and only somebody reading output beside a page will. The practical form of that rule is that a new source needs both — the sampled read once, to establish whether the output is right at all, and the metrics thereafter, to detect the day that answer stops holding.

What the metrics table does deliver, once it has run for a few weeks, is a description of the corpus precise enough to make a purchasing decision with. It says which of your sources are scan-heavy and which are born-digital, how many tables per page each one carries, what each currently costs per page to process, and how long each takes.

Those four numbers, per source, are the inputs to the question of which parser to use — and the answer turns out not to be a single parser. Each of the four is a property of a source rather than of the corpus, which is the same reason the alert had to be grouped per source: an average over sources that were produced by different processes describes none of them. A decision taken on that average inherits the defect.