OCR pipelines and vision models

OCR — optical character recognition — turns pixels into characters with positions, and a rules layer on top of it maps those characters to fields by where they sit on the page and what they look like. A vision model, by contrast, answers a question about the page directly from the pixels, with no character extraction step in between. Both produce a supplier name and an invoice total. They fail in opposite ways, and at volume that asymmetry decides everything.

The setting for this chapter is a concrete one, because the answer is a function of its numbers: an extractor handling 50,000 scanned pages a month, with a 5-second per-page latency budget, feeding fields into an accounting system that reconciles to the cent, and a target schema that changes roughly every quarter, which triggers a reprocess of the trailing twelve months. Change any of those four facts and the conclusion moves. State them out loud and the comparison stops being about which extractor is smarter.

Running the comparison, and knowing what it settles#

Take 200 pages from real traffic, hand-label every field, and run both extractors. This is the same protocol as chapter 3's resolution experiment, pointed at a different axis, and it is worth doing because it produces the numbers the conversation needs. Split the sample by page character before you look at the totals, because the aggregate hides the shape.

The overall accuracy row is the one that gets quoted, and it favours the vision model by 2.9 points. That row is also the least informative thing in the table. It is a weighted average of two very different populations, and the weighting is a property of this month's document mix rather than of either system. On the 88% of pages that are clean printed output, the two are within half a point of each other, which is inside the noise of a 176-page sample — you would need several thousand pages to establish a real difference there. Where they genuinely differ is on the 12% of awkward pages, and that is a small enough slice that a decision made on the aggregate is really a decision made on a twelfth of the corpus.

The overall row is a sum with two terms, and writing it out shows where its authority comes from and how little of that authority belongs to either extractor. A hundred and seventy-six of the two hundred pages were clean and twenty-four were awkward, so the clean pages are eighty-eight in every hundred and the awkward ones twelve. Each extractor's overall figure is its clean score multiplied by eighty-eight hundredths, added to its awkward score multiplied by twelve hundredths. For the parser that is a shade under eighty-seven plus a bit over seven, which is the 94.1. For the model it is a shade over eighty-six plus a bit under eleven, which is the 97.0. Only the two scores in each sum are properties of an extractor. The two multipliers are a count of what happened to be in this month's post bag: take the awkward fraction to zero and the parser finishes ahead by half a point, raise it and the model's lead widens, and neither system has changed in either direction. A number that moves when your suppliers change their scanners is not a number to choose an architecture with.

Charts sharpen the same point from the other direction, and are worth including in the sample if any of your pages carry them. A parser applied to a rendered chart returns the axis labels and the legend text and nothing about the data, because there is no text on the page saying that APAC overtook EMEA in Q3 — that fact lives in the relative heights of two rectangles. The vision model reads it directly and is right about the direction of the comparison almost always, while being noticeably less reliable about the exact tick values it reads off the axis, which are 9-point text and sit in the fragile regime chapter 3 measured. So on a chart the two systems are not competing at all: one recovers text and the other recovers the relationship, and the sensible design takes the labels from the parser and the relationship from the model.

What decides it at 50,000 pages a month#

Three things, none of them accuracy.

The first is determinism. Twelve pages in 500 produced a different extraction on a second identical run — a 2.4% disagreement rate with nothing changed. For most applications that is tolerable. For one that reconciles to the cent it is corrosive, because the quarterly reprocess re-extracts the trailing year and produces numbers that differ from the ones already in the ledger, for reasons that have nothing to do with the schema change you were making. You then get to explain to finance why 14,000 historical invoices moved. The parser produces byte-identical output on a byte-identical input, every time, which means a reprocess changes exactly what the schema change intended and nothing else. That property is not a nice-to-have in an accounting path; it is most of why an existing rules extractor is trusted.

Two and a half pages in a hundred reads as a rounding error until it meets the reprocess, which is where the fourteen thousand comes from. Twelve in five hundred is a shade under two and a half in a hundred. The trailing twelve months is six hundred thousand pages. Two and a half in a hundred of six hundred thousand is on the order of fourteen thousand, so that is the number of pages a reprocess can be expected to re-extract differently on a corpus that has not changed at all. The size is not the alarming part. The alarming part is that those entries move for no reason anybody can name, all at once, in a system whose entire purpose is that entries do not move on their own.

That difference between the two systems is not a matter of engineering quality, and the mechanism matters because it also decides which fixes are available. The parser is a function of its input in the strict sense: the same bytes enter the same code and the same bytes come out, and nothing outside the page is consulted. A model produces its answer one piece at a time by computing a distribution over the candidates for the next piece and drawing from it, and a draw is a draw — two draws from one distribution are under no obligation to agree. Setting the sampling temperature to zero, which is the usual response, removes that source of variation and not the other one. The distribution itself is the result of an enormous number of floating-point additions, and floating-point addition does not give the same answer in every order; the order depends on how the provider batched your request alongside everybody else's, which changes minute to minute and which no provider undertakes to hold fixed. Two candidates whose scores are nearly equal can therefore swap places between runs. Twelve pages in five hundred is what that looks like from outside, and it is why "turn the temperature down" is a mitigation rather than a fix.

The second is cost multiplied by reruns, which is where the per-page figures stop being small. At 50,000 pages a month the vision model costs about $525 a month, against $75 for the parser — a difference nobody would argue about. Then the schema changes and you reprocess 600,000 pages, which costs $6,300 in vision calls, four times a year. The parser's reprocess costs approximately nothing, because the expensive step is turning pixels into a text-and-layout representation and that representation is cached per page: a schema change re-runs the rules over stored text, which is compute you already own. Annual vendor spend comes out around $31,500 against $900, and the ratio is driven entirely by the reprocess, which is invisible in a per-page comparison.

Every one of those figures is a single multiplication, and they are worth doing in the open, because the annual number is the only one that decides anything and it is the one nobody computes. Fifty thousand pages at the model's price: fifty thousand whole pennies is five hundred dollars, and the extra half-thousandth charged on each page adds twenty-five more, which is the $525. The parser's per-page price is one seventh of the model's, so its month is one seventh of $525, which is the $75. A reprocess covers the trailing twelve months, so it is twelve months of pages — twelve times $525, which is the $6,300. A year of live traffic is also twelve months of pages, so a year of live traffic costs that same $6,300; four reprocesses cost four more; and five lots of $6,300 is the $31,500. The parser's year is twelve months at $75, which is $900, and its reprocesses add nothing measurable, because the step that costs money — turning pixels into text and layout — already ran when the page first arrived and its output is on disk. What that leaves is a ratio far wider than the seven-to-one the per-page row advertises, and every bit of the extra width is reprocessing.

Wall-clock time follows the same pattern. Rules over 600,000 cached pages finish in tens of minutes on a handful of workers. Six hundred thousand vision calls at a median 2.8 seconds, run at fifty concurrent and subject to rate limits, is on the order of half a day, during which the ledger is being rewritten. And the p95 of 6.1 seconds is already over the 5-second per-page budget, so the vision path fails the stated latency requirement on one page in twenty before any of this.

The half-day figure is the same kind of multiplication, and it is worth checking rather than accepting, because it is the one that decides whether a reprocess is an overnight job or a working week. Six hundred thousand calls at a median of 2.8 seconds each is a bit under two million seconds of generation. Running fifty of them at a time divides that total by fifty, which leaves something over thirty thousand seconds, and thirty-odd thousand seconds is nine hours and change — before rate limiting, which can only make it longer. Rules over cached text finish in tens of minutes because there is no model in the loop at all: the expensive step ran once, when each page first arrived, and a schema change re-runs only the cheap half.

The third is where the errors land. The parser's failures announce themselves. An OCR engine scores every character it emits by how cleanly the pixels matched a glyph and rolls those scores up into a confidence for each extracted field, so a smudged digit produces a low per-field confidence, and on this sample a threshold at 0.75 caught 91% of its errors while flagging 4% of fields for review. The vision model's failures are well-formed numbers. Asked to attach its own confidence, it rated 84% of its wrong answers as high confidence — the same phenomenon chapter 5 described on downscaled invoices, where a model that cannot resolve a character returns a plausible one, now appearing at full resolution on genuinely hard pages. A system whose errors carry a usable signal can be operated at 94% accuracy safely. A system whose errors are indistinguishable from its successes cannot be operated at 97% safely, because you do not know which 3%.

Where that per-field confidence comes from is worth a step, because everything this chapter ends up recommending rests on it. An OCR engine does not read a page. It decides, region by region, which character a patch of pixels is. However that decision is implemented — a classical engine comparing the patch against stored glyph shapes, a modern one running it through a classifier trained on images of characters — it is made by scoring the candidates and taking the highest, and the winner's score falls out of the process at no extra cost, because it had to be computed in order to choose. A crisply printed digit wins its region by a wide margin. A digit that has been smudged, clipped by a fold, or rendered at the sizes chapter 3 measured wins by a narrow one, because the pixels that would have separated it from its nearest lookalike are the pixels that were lost. The rules layer rolls the character scores of a field into one number for the field. So a low field confidence says something specific and checkable: the pixels in that region did not clearly resemble any one character. That is a statement about the input, which is exactly why it predicts errors — the regions where the evidence was thin are the regions where the wrong candidate can win.

The vision model has no such number to give, and the absence is structural rather than an omission somebody could fix. Nowhere in its processing is a patch of pixels scored against a set of characters; the page goes in whole and an answer comes out whole. What it can report is how strongly it favoured one piece of its answer over the alternatives at each position, and that is a measurement of something else. It is conditioned on everything the model has available — the rest of the page, the prompt, the layout, and everything it absorbed in training about what invoices look like. A model that cannot resolve a digit falls back on that last part, and the fallback can be strong: totals are often round, a supplier's invoices resemble each other, a subtotal usually agrees with its line items. The answer produced from the prior is written with the same firmness as the answer produced from legible pixels, because from the inside they are the same operation. That is why asking the model to rate itself put 84% of its wrong answers in the high-confidence band. The number is not a poorly calibrated version of the OCR engine's number. It measures a different quantity, and no amount of prompting turns one into the other.

The belief that builds a hybrid the wrong way round is that those two numbers are the same kind of thing, one merely noisier. It is tempting because both are called confidence, both run between zero and one, and both tend to fall on hard pages. What is true is that one reports on the evidence and the other reports on the model's own conviction, and only the first can be thresholded.

That distinction inverts the table's headline, and the arithmetic is short enough to do here. A parser at 94.1% gets a shade under six fields in a hundred wrong. Its confidence threshold catches ninety-one of every hundred of those, so nine in every hundred escape, and nine hundredths of a shade under six in a hundred is a little over half of one field in a hundred reaching the ledger unreviewed. The model at 97.0% gets three fields in a hundred wrong and catches none of them, because it has nothing to catch them with, so all three in a hundred reach the ledger unreviewed. The instrument with the worse accuracy delivers roughly a fifth as many unreviewed errors. Accuracy counts mistakes; an accounting path cares about mistakes that arrive looking like facts, and those are two different quantities that coincide only when neither system has an error signal.

What a vision model buys that no rule can#

None of that is an argument that the parser is better. It is an argument about what each instrument is for, and the vision model's half of it is real. A rule is written against a layout. When the layout holds, the rule is exact, fast, free, and repeatable. When the layout does not hold, there is no rule, and no amount of engineering produces one — you cannot write a regular expression for a table with no ruling lines whose columns are established by whitespace that varies per supplier, or for a handwritten note in a margin, or for a chart whose underlying numbers nobody kept, or for the layout a supplier silently changed last quarter and will change again.

That is precisely the 24 pages where the parser scores 61% and the model scores 89%. It is also the population that grows every time you onboard a supplier, and the population a rules team spends most of its time on. Replacing the parser throws away everything the rules do well; refusing the model leaves the awkward pages permanently in a manual queue.

The belief a demo installs is worth naming separately, because it is the one that gets acted on. A page the incumbent cannot handle, handled, reads as evidence that the new instrument contains the old one — that everything the parser does, the model also does, plus the handwriting. What the demo actually establishes is narrower: on that page, the model is better. Containment would additionally require the model to be at least as good everywhere else, and three rows of the table say it is not. It is slower, it is seven times dearer per page, and it disagrees with itself. None of those three can appear in a demo, because each needs either a second run of the same page or a second month of invoices, and a demo is one run of one page.

The split those two strengths imply#

The parser stays primary and handles the pages it is good at, deterministically and almost free. The gate is not the model's opinion; it is the parser's own confidence plus checks you can compute.

from decimal import Decimal

def needs_vision(page) -> bool:
    if page.min_field_confidence < 0.75:
        return True
    if page.table is None or page.table.row_count != page.stated_row_count:
        return True
    line_sum = sum(item.amount for item in page.line_items)
    if abs(line_sum - page.subtotal) > Decimal("0.01"):
        return True
    if abs(page.subtotal + page.tax - page.total) > Decimal("0.01"):
        return True
    return False

Every branch there is deterministic and computable from the parser's own output: character-level doubt from OCR confidence, structural doubt from a missing or short table, and semantic doubt from arithmetic that should close and does not. The arithmetic checks are the strongest of the three, because they catch a digit read confidently and wrongly, which confidence alone never will. Roughly 7% of pages route onward, which puts the vision spend at about $42 a month rather than $525, and puts the vision model exactly where its 89% beats the parser's 61%.

Why the arithmetic checks are the strongest of the three deserves its own step, because it is not obvious that a sum can outperform a purpose-built confidence score. Confidence is high whenever the pixels were clear, and pixels can be perfectly clear and still be read into the wrong place: a digit that genuinely is that digit, lifted out of a column the rules assigned to the wrong field, produces a legible, confident, wrong number. The sum of the line items is a different kind of evidence. It is a fact the page states twice — once as a list and once as a total — and a check that the two statements agree does not care how any of the digits were obtained. A single misread or misplaced digit breaks the agreement whatever confidence was attached to it. Character confidence audits the reading; the arithmetic audits the result.

The spend figure follows from the flagged fraction and from nothing else, which is worth seeing written out because it tells you which lever exists. Seven pages in a hundred out of fifty thousand is three and a half thousand vision calls a month, and three and a half thousand at a shade over a penny each is a little under forty dollars; the $42 above allows for a flagged fraction sitting slightly over seven on this sample. The exact figure matters less than the shape. The vision bill is the model's per-page price multiplied by a fraction the router decides, and the price is not yours to move, so every change you can make to that bill you make by moving the fraction. That is also why the fraction has to be watched rather than set once.

The reprocess story stays intact because both intermediate layers are cached by page hash: the OCR text and, for routed pages, the vision model's raw extraction. A schema change re-runs rules and re-maps stored outputs. Nothing is re-inferred, nothing moves that should not move, and the whole thing finishes over lunch.

When the four facts change#

Change the volume and the answer changes. At 200 pages a month with no reprocess, one vision call per page is the correct architecture, and building a rules extractor to save $20 a month is malpractice; the constant cost of maintaining two systems dominates everything. Change the tolerance and the answer changes: a spend-analytics dashboard that wants approximate category totals does not need determinism or per-cent reconciliation, and the model alone is fine.

The hybrid is also not free. You now own two extractors, two sets of failure modes, and a router whose thresholds drift as the document mix drifts; the flagged fraction needs monitoring, because a supplier changing template can take it from 7% to 30% overnight and quietly turn your $42 into $300 while the accuracy still looks fine. And the whole design leans on one asset that took real effort to build: a signal that says this page is likely wrong. The parser supplies it natively. The vision model does not, which is why it sits downstream rather than upstream.

The comparison worth running, then, is not "which one is more accurate" but "which one's errors can I see, and what does a rerun cost me". Accuracy is the easiest number to produce and the one most likely to be dominated by a document mix that will look different next quarter. Determinism, the cost of the reprocesses your schema changes force, and the presence or absence of a per-field doubt signal are properties of the instrument rather than of this month's traffic, which is why they are the ones that should decide the architecture.

That asymmetry invites an obvious question. If a confidence signal is what makes the architecture work, and the model does not provide one, could you manufacture one — run the same image twice, in two different ways, and treat agreement as confidence? It is a reasonable-sounding idea, and the specific version involving two resolutions is worth taking apart carefully.