An OCR'd chart is topically right and evidentially empty

A chart stores its numbers as positions. A line sits at some height above a baseline, and the height is what the number is; the characters printed around it name the units, the periods and the series, and none of them is a value. When a page carrying that chart is turned into text, the characters come through and the positions do not. What is left is a paragraph that names everything the chart is about and states nothing the chart shows. That paragraph is an excellent match for a question about the chart's subject and a useless answer to a question about its contents. The distance between those two facts is the subject here.

The arrangement in which this matters has a name. Retrieval-augmented generation, almost always shortened to RAG, does not ask a language model to remember anything about a particular business: given a question, a retriever searches a corpus for the passages most likely to bear on it, those passages are pasted into the prompt as evidence, and the model is asked to answer from them and cite them. Getting a document into such a corpus normally means turning each page into characters, splitting the characters into chunks of a few hundred words, and storing a vector for each chunk so that a question can be matched against it by meaning rather than by keyword. The step at the front is OCR — optical character recognition, which reads the marks on a page image and returns a string of characters. On a born-digital PDF it is closer to a lookup in the embedded text layer than to genuine recognition, and either way the thing it hands downstream is a string.

An index whose unit of storage is a string#

Multimodal RAG, which is the subject of this topic, is the same arrangement with the text-only assumption taken out of it. The units in the index and the evidence handed to the model can be images — a whole PDF page rendered as a picture, a cropped figure, a scanned table, a screenshot of a dashboard — retrieved by encoders that compare a query against an image rather than against a string, and read by a model that can see. It exists because a substantial share of what an organisation knows is drawn rather than written, and every step that turns a drawing into a sentence discards most of it. You meet it wherever the honest answer to "where did that number come from" is "the chart on page 14": internal assistants over board decks and quarterly filings, field-service tools over scanned hardware manuals, support systems over product screenshots, review workflows over engineering drawings and floor plans.

What the absence of that machinery costs follows in four steps, each checkable on its own. A retriever ranks by comparing the query's vector against the vectors it holds. Each of those vectors is computed from a chunk of text. Each chunk is a substring of whatever the extractor emitted for that page. Therefore anything the extractor did not emit is not merely ranked badly — it has no representation in the space the ranking happens in, and no ranking function can promote something that was never a candidate. The last step does the damage, and the first three are so ordinary that nobody stops on them.

That chain rests on a decision rather than on a requirement, and separating the two is what makes the rest of this predictable. The requirement is to answer questions about an organisation's own documents and cite where the answer came from. Nothing in it says the unit of storage has to be a string. Text became the unit because text encoders were cheap and mature when most of these systems were built — a good engineering reason, and not a necessity. A requirement quietly satisfied by one representation keeps looking satisfied until the first question whose answer that representation never held.

What survives a chart when the page becomes text#

Here is the entire text an OCR stage extracted from one page of a board deck, verbatim, as it was stored in the index:

Monthly Churn Rate — Trailing 18 Months
Source: Billing warehouse, snapshot 2024-09-30

8%
6%
4%
2%
0%

Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

■ SMB   ■ Enterprise   --- Blended

Note: SMB cohort excludes trial-to-paid conversions in the first 30 days.
Churn is calculated on logo count, not ARR.

Two hundred and thirty characters, and not one of them is a value from the series. The title tells you the page is about churn. The axis ticks tell you the plot spans 0% to 8%. The month row tells you the time range. The legend tells you three lines exist. The footnote tells you how churn was defined. All of that is metadata about the chart, and none of it is the chart.

The time range is the one item on that list that does not quite close, and the way it fails is the same defect one level down. The title says eighteen months; counting the labels in the month row gives sixteen. Both readings are ordinary — a renderer suppresses labels at the ends of an axis to stop them colliding with the frame, and an extractor drops labels that are clipped or rotated — and nothing in the string distinguishes them. So the month row reports the granularity outright, and reports the span only up to whatever happened at the edges.

Put the question next to the block and the shortfall becomes specific. "When did churn first cross 5%" needs a month, and the position of a line relative to a gridline in that month. The month row supplies month names with no association between any name and any height. The tick column supplies gridline labels with no association between any label and any point on the line. Both facts the answer is made of are relations between things the string mentions separately, and a relation between two glyphs is not itself a glyph. That is how the extraction can be complete and the evidence absent at the same time.

Why the retrieval score cannot move#

An embedding of that text sits very close to the query "when did churn first cross 5%" — it shares the domain vocabulary, the metric name, the unit, and the time granularity. That is why it retrieves at rank 1. The similarity score is measuring whether the chunk is about the right thing, which it is. It has no way to measure whether the chunk contains the answer, which it does not.

The reason it has no way is mechanical rather than a matter of tuning. The score is a function of two strings and nothing else: the query as written and the chunk as stored. The chunk as stored holds naming content only. A function whose inputs do not include a quantity cannot vary with that quantity, so the score is formally independent of whether the evidence is present, and no threshold on it can separate the two cases.

Imagine the same chart drawn with data labels printed above each point, which is the one change that would make the page self-sufficient. The extractor now emits the values too, and the chunk goes from holding none of the evidence to holding all of it. The similarity score barely moves: the added characters are bare numerals and percent signs, which share no vocabulary with the question and carry almost no signal for an encoder trained on meaning. Evidential sufficiency swings from empty to complete while the monitored number sits still. A quantity that stays constant while the dangerous property varies across its whole range is not a weak signal. It is not a signal.

The generator then does what generators do with an underdetermined prompt. It has the axis range, it has the month labels, it has a footnote about the SMB cohort, and it has the general knowledge that SaaS churn tends to drift upward. It produces a plausible date. In a run of 40 chart-backed questions over this corpus, 31 came back with a confidently stated number, 9 with a hedge, and 0 with a refusal. The model rarely says "the page contains a chart whose values I cannot read", because from its position there is no chart — there is a paragraph of text about a chart.

Those three counts sum to the run exactly, and the arithmetic licenses one claim and not another. Confident, hedged and refused exhaust the outcomes, and adding them recovers every question asked, so there is no fourth category and no silent drop-out. That buys a statement about composition: a monitor watching refusal rate would have seen a flat zero across the whole run. It buys no statement about accuracy, because nothing in a sum of outcome counts says how many of the thirty-one confident answers were right.

Topical relevance and evidential sufficiency#

It helps to name the two things separately, because almost every retrieval metric in common use measures the first and is silent about the second.

Topical relevance is the property your retriever optimises: this passage concerns the same subject as the query. It is what the familiar ranking metrics all score — recall@k, which asks whether a relevant passage appeared anywhere in the top k results, and the position-sensitive pair MRR and nDCG, which reward putting it near the top — because those metrics are computed against relevance judgements, and a human labelling a chart page as relevant to a question about that chart's metric is correct to do so.

Evidential sufficiency is the property the answer depends on: this passage contains enough information for the specific claim to be derived. A page can be maximally relevant and contain zero of the required evidence, and that is exactly what happens when the evidence lives in pixels and the index holds text.

The gap between these two shows up as a characteristic pattern in evaluation numbers. On a 600-question set over a corpus of financial filings and board material, one system measured recall@5 at 0.94 and end-to-end answer correctness at 0.61. That 33-point spread is not a generation problem in any useful sense — the generator was handed insufficient evidence and did its best with it. Slicing the same set by where the evidence lived made the shape obvious: on questions whose answer appeared in body text, correctness was 0.88; on questions whose answer appeared only in a figure, correctness was 0.19, while recall on that same slice was 0.91. Retrieval was working. The representation was not.

Read those four numbers as levels and they are four facts to remember. Read them as gaps and they collapse into one. Within the figure slice, retrieval and answering are measured on the same questions, so subtracting one from the other is meaningful: 0.91 against 0.19 is a gap of seventy-two points between finding the material and using it. The corpus-wide gap of thirty-three points is that same quantity diluted by every question whose answer was ordinary prose. The figure slice carries more than twice the average gap, and the average exists only because it is mixed with a population that has almost no gap at all.

The two slice figures also reproduce the headline, which is the check worth doing before believing any of it. Write f for the share of questions whose answer lives in body text, so 1 − f is the share whose answer lives only in a figure. The aggregate is the two slices weighted by those shares: 0.88f + 0.19(1 − f) = 0.61. Expanding the bracket puts 0.19 in twice, once alone and once multiplied by −f, so collecting the f terms leaves 0.19 + f(0.88 − 0.19) = 0.61 — permitted because both terms carry the same unknown to the first power and the slice scores do not themselves depend on the mix. The coefficient of f is the distance between the slices, sixty-nine points, and the aggregate stands forty-two points above the figure slice, so f is forty-two sixty-ninths: a shade over three questions in five.

Nothing is left over, and that licenses a negative claim of some strength. The aggregate is fully accounted for by two populations and one weight, so there is no third band of questions failing for a reason the modality split does not name. The same shortfall arrives from the other side as a check: the aggregate sits twenty-seven points below the body-text slice, and figure questions, being about two in five, drag it down by two fifths of the sixty-nine-point slice gap, which is a little over twenty-seven points. Two routes sharing no step land on one number.

Two residuals survive that arithmetic. Correctness can be earned without evidence — a model that guesses a well-known figure is scored right, which inflates 0.61 and makes both gaps look smaller than the evidential situation warrants. And the mix was inferred here rather than reported, so a consistent weight existing is a consistency check and not a measurement; three-in-five is not a property of the question set to be quoted onward.

The document types that fail this way silently#

The failure is not evenly distributed. Some material survives text extraction essentially intact, and some loses its entire information content while leaving behind a convincing residue of labels.

Read down the third column and the common shape appears: what is lost is always the relational or quantitative content, and what survives is always the naming content. That asymmetry is what makes the failure so hard to catch by inspection. A chunk that lost its data still reads like a chunk about data. A reviewer skimming the index sees domain-appropriate text on every page and concludes the parse went fine.

The table is more useful read across than read down, because the second and third columns are the same distinction eight times over. In every row the kept column lists names and the dropped column lists relations or quantities, and the eight document types have nothing else in common — a floor plan and a checkbox form share no vocabulary, no producer and no failure story. Eight independent cases agreeing on one boundary is the closest thing here to a law: extraction preserves what is written down as a symbol and loses what is expressed as an arrangement of symbols.

The flow diagram row deserves a second look, because it is the purest case. A diagram of six boxes and seven arrows extracts as six labels. The arrows carry the entire meaning — which step follows which, where the loop is, which branch is conditional — and arrows have no text, so they leave no residue at all. There is no signal in the output that anything is missing, because absence of arrows looks identical to a page that never had any.

The arrow count does more work there than it first appears. Six boxes in a single unbranching line need five arrows to connect them, one fewer than the number of boxes, because the first box starts the chain rather than being pointed at. Seven are drawn, so two cannot belong to any linear order — and two extra edges over a chain of six is exactly what one loop and one conditional branch cost. Extraction returns the labels in reading order, which can at best re-encode the five chain arrows and carries neither of the two that make the diagram a process rather than a list. The box count, meanwhile, is conserved perfectly: six went in and six came out. Any completeness check that counts labels reports a flawless parse of a page whose meaning has gone.

Scanned tables fail in a subtler and more dangerous way, because they lose nothing and corrupt everything. A three-column table of segment, quarter and margin extracts every one of its ninety cells. What it loses is which cell belongs to which row, and a text extractor that reads by sorted glyph position will happily emit EMEA Q2 24% APAC Q2 31% as one run when the row spacing is tight or the scan is a degree off square. Downstream, that string supports the claim that EMEA's margin was 31% just as readily as the true one. The chart above produced an answer with no evidence behind it; a mis-assembled table produces an answer with evidence that positively supports the wrong number, which is harder to catch and worse when it lands.

Ninety cells across three columns is thirty rows, and the conservation is exact: every cell on the page is in the string, so counts of cells, of tokens and of characters all come through unchanged. What has gone is the row membership of each cell, and row membership was never a character — it was the vertical alignment of three glyph runs. A checksum over the extracted content is therefore exactly conserved while the property that decides whether the answer is true is destroyed, which means any validation built on comparing counts before and after extraction certifies this page as perfect.

The instruments that stay constant#

Nothing in production flags this. The failure has no observable signature at run time, which is why it survives to the support queue rather than being caught in staging. The retrieval score is high and correct. The generation call returns normally, reporting stop_reason: end_turn. The answer is short, fluent, and formatted like every correct answer the system produces. If the system emits a citation, that citation is to a page a reviewer would agree is the right page — because it is. Every guardrail that watches for refusals, empty responses, low-confidence hedging or missing citations sees a healthy request.

Set those signals side by side and the pattern is the one the similarity score already showed. Each is computed from something that does not include the plotted values: the score from two strings, the stop reason from the decoder's own termination, the fluency from the model's language modelling, the citation from a document identifier, the guardrails from the shape of the response. Move evidential sufficiency from complete to zero and every one of them is unchanged, because none ever took it as an input. The belief that a citation check catches this is the expensive one, and it is tempting for a good reason: citation checking genuinely does catch fabricated sources, which is the failure everybody arrives expecting. A citation names a location. It says nothing about what is at that location, and here the location is right.

The one signal that does exist is a negative one, and it needs to be measured deliberately rather than watched for: the gap between retrieval quality and answer quality, sliced by where the evidence lives. A system whose recall is 0.94 and correctness is 0.61 is telling you that a third of the time it finds the right material and cannot use it, and the only way to find out which third is to label questions by the modality that holds their answer. That labelling is not expensive — a few hundred questions is enough to see the shape — and without it the aggregate correctness number simply reads as "the model is a bit unreliable", which points every subsequent investigation at the wrong stage.

Three fixes that operate on the wrong representation#

The natural first response to a wrong answer from a chart page is to look at the extraction quality, and that response leads nowhere. Character error rate on page 14 is essentially zero: the deck is born-digital, the text layer is embedded, every glyph came through the font encoding table intact. There is no upstream improvement available. A perfect extractor extracts what is there, and the plotted values were never characters — they are polyline vertices in a vector drawing, or anti-aliased pixels in a raster, depending on how the chart was produced. Chasing 99.8% accuracy to 99.9% moves nothing, because the missing evidence is not in the 0.2%.

Why that knob is inert generalises to every other quality metric on the extraction stage. A character error rate is a ratio whose denominator is the characters that exist to be got right. The plotted values, never having been characters, are not in the denominator, so they are not in the numerator either whatever the extractor does; the term cancels out of the measurement entirely. That is how a character error rate of zero and an answer rate of zero can hold at once without contradiction. They count disjoint things, and improving the first cannot touch the second at any value it takes.

It is equally not a chunking problem. Enlarging the chunk to the full page, or to the page plus its neighbours, adds more prose about churn and still adds zero plotted values. Chunk-size tuning changes how much context surrounds the evidence; it cannot conjure evidence that the representation never captured. Teams often spend a sprint on chunk boundaries here because chunking is the knob they are used to turning, and the failure is loud enough to demand action while being quiet enough to misdiagnose.

The third tempting fix is captions. Topic 9.4 covered running a vision model over each page and writing out what it sees, and a caption is a genuine improvement over raw OCR — it is text that at least knows a chart is present. What a caption does not do is preserve the values. "A line chart showing monthly churn rising from roughly 2% to roughly 7% over eighteen months, with SMB consistently above Enterprise" is a good caption, and it still cannot answer the question, because "first crossed 5%" needs a specific crossing point that the caption compressed away. Captions are lossy by construction: their job is to summarise, and summarising a chart means discarding most of its data points. You can push a captioner toward transcribing values, and on dense charts it starts inventing them, which converts a silent gap into a confidently wrong number stored permanently in your index.

What all three fixes share is that they operate on text. The evidence is not text. Any pipeline whose final representation of a page is a string has already discarded the answer before retrieval runs, and no amount of tuning downstream of that point recovers it.

The rule has a boundary, and drawing it stops this becoming an argument against text indexes. Material whose meaning is carried by the symbols themselves comes through extraction with nothing lost: body prose, policy text, code, correspondence, and tables whose cell geometry survives because the file format records it rather than because a reader infers it from spacing. On a corpus of that kind the gap between relevance and sufficiency is close to nothing, which is why the aggregate correctness figure above sits as high as it does. The failure belongs to a population of pages rather than to a technology, and a system that cannot say what share of its corpus falls in that population cannot say how much of its answering it should trust.

For the pages that do fail this way, the index has to hold something that preserves the plotted values, the arrows, the cell geometry and the checkbox state — which in practice means retrieving over the page image itself rather than over a description of it. A relation between glyphs cannot be stored as a glyph, and every fix that stays inside the string works on the wrong side of that boundary.

That conclusion has an obvious-looking corollary that is wrong, and the next chapter is about why. If a page image contains everything the extracted text contains plus the figures and the layout, then indexing pages as images looks like a strict upgrade over indexing them as text, and the text pipeline looks like something you can retire. It is not a strict upgrade. The failure it introduces is quieter than the one it fixes, which is a bad trade to make without knowing you are making it.