Retrieving the image is wasted unless the model is handed the pixels

A retrieval system that has learned to rank page images still has to hand those images to something that can look at them, and those are two different pieces of software. A retriever's job ends when it emits identifiers: a document, a page number, a score. Turning that list into the bytes of an API request is a prompt builder's job. Nothing in a ranked list says whether what reaches the model downstream will be pixels or characters, so the choice belongs entirely to whatever assembles the request — and in a system that began text-only, that component was written when characters were the only option there was.

Retrieval-augmented generation, usually shortened to RAG, answers a question by searching a body of documents for the passages most likely to bear on it and pasting those passages into a language model's prompt as evidence. A text-only version of that arrangement stores one string per unit, and its prompt builder has exactly one operation available to it: look up an identifier's stored string, and concatenate. Hand that same builder a page identifier from an image retriever and it performs that same operation, because fetching a stored string is the only thing it has ever been asked to do. What goes out is a text request, and what answers it is a model reading characters.

What a prompt builder does with a retrieved page#

The step from "the retriever returned the right page" to "the model saw the right page" is the one worth walking across slowly, because it is four links long and every one of them is skippable.

An index stores, for each unit, an identifier and whatever payload was written beside it at build time. A text-only build wrote one payload per unit, and that payload is a string. Adding an image retriever adds a ranking over those units — a new way of deciding which identifiers come back first — and adds no payload at all, because rendering pages and storing renders is a separate job that lives in the indexing service. So resolving an identifier still finds exactly one thing to send, and that thing is the string. The model therefore receives text, and it receives it while every log in the system records that the correct page was retrieved at rank one. The retriever was not ignored. It was obeyed exactly, and the obedience was carried out in the wrong medium.

It is worth separating what is forced here from what was chosen, because the two get argued about as though they were the same kind of thing. That a chart's values exist only as marks on a page is a property of the document: nobody decided it and no configuration changes it. That a prompt carries a page's stored string rather than its render is a decision, made implicitly, by whoever wrote the resolve step. Render resolution, image slot count, encoding format and whether a citation carries coordinates are decisions too. A decision has a knob and a property does not, which is why the rest of this chapter is about the second kind.

Why a retrieval dashboard cannot see this#

This is a common shape for the bug because the retrieval change and the prompt change live in different services owned by different people, and the retrieval change is the one that shows up in dashboards. Recall on the figure slice jumps from 0.22 to 0.79. Answer correctness on that slice moves from 0.19 to 0.21, which reads as noise and gets attributed to model variance for a fortnight.

The reason the dashboard is silent is structural rather than statistical, and it is worth stating in the form that makes it obvious. Recall@k — the share of questions for which a relevant unit appeared anywhere in the top k results — is computed from the retriever's ranked list and the relevance labels, and nothing else. A prompt builder runs after that list exists and consumes it. The two share no step. So recall@k is invariant under every change a prompt builder can make, including sending the wrong medium, including sending the wrong field, including sending nothing at all. A green recall number licenses exactly one claim: the right unit was ranked highly. It licenses no claim whatsoever about what left the building.

That invariance is the shape of the stumble here, and the stumble is a belief that a defence works. Retrieval metrics feel like a defence because they are the metrics that moved when the visual index landed, and a metric that moves is a metric that appears to be watching. What makes it dangerous is that it then holds still while the property deciding the answer varies freely underneath it. Read the two movements as a gap rather than as two levels and the bug is in the numbers already: on one slice, over one change, one instrument moved by more than half its range and the other moved by two points. Two instruments disagreeing that violently about a single change are not both measuring it.

What the request looks like when the pixels actually go#

The generation step has to send the image itself as a content block alongside the text. Here is the shape, using the Anthropic Messages API, with two retrieved page images and three text chunks in one prompt:

import anthropic, base64, pathlib

client = anthropic.Anthropic()

def image_block(png_path: str) -> dict:
    return {
        "type": "image",
        "source": {
            "type": "base64",
            "media_type": "image/png",
            "data": base64.standard_b64encode(
                pathlib.Path(png_path).read_bytes()
            ).decode(),
        },
    }

content = [
    {"type": "text", "text": "Evidence 1 — deck_q3_2024.pdf page 14, rendered at 2048px:"},
    image_block("renders/deck_q3_2024/p014@2048.png"),
    {"type": "text", "text": "Evidence 2 — deck_q3_2024.pdf page 15, rendered at 2048px:"},
    image_block("renders/deck_q3_2024/p015@2048.png"),
    {"type": "text", "text": "Evidence 3 — churn_memo.md, chunk 7:\n" + chunk_7},
    {"type": "text", "text": "Evidence 4 — board_minutes_2023-11.md, chunk 2:\n" + chunk_2},
    {"type": "text", "text": "Evidence 5 — churn_definitions.md, chunk 1:\n" + chunk_1},
    {"type": "text", "text": "Question: When did monthly churn first cross 5%?"},
]

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system=(
        "Answer only from the evidence provided. Every claim must cite an "
        "evidence number. When the evidence is an image, also give the region "
        "you read it from as a normalised bounding box [x0, y0, x1, y1] with "
        "coordinates in 0-1 relative to the image. If the evidence does not "
        "contain the answer, say so and name what is missing."
    ),
    messages=[{"role": "user", "content": content}],
)

Two details in that snippet carry most of the weight. The label text block before each image is what lets a citation refer to something — the model cannot see filenames, so an unlabelled image is an anonymous rectangle it can describe but not attribute. And the system prompt asks for a bounding box, which is the difference between a citation that says "page 14" and one that says "the region of page 14 where I read this", a distinction the rest of this chapter turns on.

The interleaving is doing a third job. Each image sits immediately after the label that names it in one ordered list, so a block's position is what binds a picture to its provenance. Collecting the images at the end and the labels at the front breaks that binding without breaking anything that raises an error.

What an image costs, and what downscaling costs#

A text chunk of 500 tokens is a rounding error in a modern context window. A page image is not. On the current high-resolution vision tier, an image is accepted up to 2,576 pixels on its long edge and consumes up to roughly 4,784 tokens — about ten times a text chunk, for one page. Older tiers capped at 1,568 pixels and roughly 1,600 tokens, and the extra fidelity is exactly what makes small type legible, so the cost increase is buying something real.

The temptation is to downscale until the images fit whatever budget you had before, and that is the move that silently reinstates chapter 1's failure in a new place. Below are measurements from rendering 200 pages of financial filings at a range of resolutions and asking the model to transcribe three targets from each: 9-point body text, 6-point chart axis labels, and four-digit numbers in table cells at roughly 7 points.

The last column is not a sixth measurement. Take any row's token count, multiply by five dollars per million input tokens, and the cost cell falls out: the 2,048 px row's three thousand and fifty tokens come to a little over a cent and a half, which is the cell as printed, and the same multiplication reproduces all five. That closes with no residual, and what it licenses is worth having. Every cost claim in this chapter is a token claim in another currency, so the table has four independent columns rather than five; a resolution the table does not list can be priced without a new experiment; and a change in the input price moves the whole column by one factor without disturbing any of the legibility findings.

The token column has an arithmetic of its own, and it is not the one most people expect from a column headed with a single edge length. Compare the 1,024 px row with the 2,048 px row: the long edge doubles and the token count goes from about seven hundred and sixty to about three thousand and fifty, which is four times over rather than twice. Four is what you get when both edges double, because a page is a rectangle and doubling each side quadruples the area. So tokens are charged on pixel area, and the single number in the first column is the square root of what is being billed. The same law holds at the top of the table, where the step from 2,048 px to 2,576 px is a quarter more on each edge and a little over half again in tokens. It fits loosely in the middle rows, where one resolution is not an exact multiple of another, and the header is honest about why: the column says measured, not computed, and the slack is whatever the encoder's own rounding and the page's aspect ratio contribute.

The 6-point column is the one that decides the architecture. Between 1,024 px and 2,048 px, axis-label legibility goes from one in five to nine in ten, and that column is a direct proxy for whether the system can answer "when did the line cross 5%". A team that downsamples to 1,024 px to keep prompt costs near the text-only baseline has built a pipeline that retrieves the right page, sends the pixels, and still cannot read the chart — with a cost line that now looks 4× worse than before for no measured gain.

Read the three legibility columns as gaps and the table sorts its own contents. The 9-point column is finished by the second row and contributes nothing above it; body text is not what the resolution decision is about, and a team that spot-checks legibility on paragraphs will conclude that 1,024 px is plenty. The 7-point table-digit column is most of the way there by the third row. The 6-point column is the only one still moving across the top half of the table, and it is the column that carries chart axes. The three columns are not three readings of one quantity at three difficulties. They are three different thresholds, and the architecture is decided by the smallest type in the corpus rather than by the average.

That gives the second stumble of this chapter its shape, and it is again a belief that a defence works. The defence is "we sent the pixels". It is tempting because it is true, checkable and visible in the request body: the image block is there, the token bill went up, the response came back normally. All three of those hold constant across every row of the table. What varies underneath them is the 6-point column, which swings from four in a hundred to ninety-six in a hundred, and nothing in the response distinguishes a page whose axis labels were read from one whose axis labels were a grey smear. The reply is fluent in both cases, for the same reason chapter 1's reply was fluent when the evidence was a title and a legend.

The arithmetic of a prompt that carries pages#

Prompt composition follows from the same table. At 2,048 px, six page images is 18,300 tokens; adding five text chunks and a system prompt lands around 21,000 tokens per request, which is $0.105 of input at $5 per million. A workload of 40,000 queries a month costs about $4,200 in input tokens, against roughly $600 for the text-only version of the same system. Whether that is acceptable is a product decision, and it is a decision you can only make with the number in front of you.

Each of those steps is a multiplication, and they close on each other tightly enough to be worth walking through, because a chain that closes is a chain you can re-run with your own inputs. Six images at the 2,048 px row is six times about three thousand tokens, which is the eighteen-thousand-three-hundred figure exactly — the prompt is linear in image count at a fixed resolution, so slot budgeting is arithmetic rather than an experiment. Five text chunks at the five hundred tokens quoted at the top of this section add two and a half thousand, bringing the running total to twenty thousand eight hundred. The residual between that and the quoted twenty-one thousand is a couple of hundred tokens, which is the system prompt and the question. Multiply by five dollars a million and the per-request figure is a tenth of a dollar and a half a cent, which is the $0.105 as printed; multiply that by forty thousand and the monthly figure is $4,200, again with nothing left over.

The ratio to the text-only baseline closes in the same way and says something the raw numbers do not. Six hundred dollars across the same forty thousand queries is about a cent and a half a request, which is around three thousand input tokens — five text chunks, a system prompt and a little else. So the two monthly bills stand in the same ratio as the two prompts, seven to one, and that entire ratio is the images: the text both prompts carry is a shared term that cancels out of the comparison. What this licenses is a cleaner argument than "images are expensive". Image tokens are about seven-eighths of the request, so every lever that matters is a lever on images — how many, at what resolution — and trimming text chunks to pay for pictures moves a number that was never the problem.

Latency moves too, and less predictably than cost. Encoding six full-resolution images adds meaningfully to time-to-first-token: on this workload p50 went from 0.9 s to 1.6 s and p95 — the slowest request in twenty, which is the number a latency budget is usually written against — from 1.8 s to 3.4 s, with the tail dominated by requests carrying the maximum image count. Image bytes also travel over the wire, so a 2,048 px PNG at 600 KB × 6 is 3.6 MB per request leaving your service.

Those four latency figures are more informative as a pair of ratios than as four levels. Before the change, the p95 was twice the p50. After it, the p95 is a shade over twice the p50. Both points moved by getting on for double, and the spread between them barely changed, which means the whole distribution was scaled rather than given a new tail. That is worth knowing because it settles a question teams argue about: a budget written against p95 and a budget written against p50 move by nearly the same factor here, so there is no version of this change that is cheap at the median and expensive only in the tail, and tuning aimed at the tail alone has nothing specific to bite on.

Encoding format is a small decision with a large effect on that transfer number and no effect on the token count, since tokens are computed from dimensions rather than bytes. PNG is lossless and enormous for a page of text and rules — 600 KB is typical at 2,048 px. JPEG at quality 85 brings the same page to about 180 KB, and it introduces ringing artefacts around glyph edges that show up first on exactly the small type you raised the resolution to preserve. WebP at quality 90 lands near 140 KB with noticeably cleaner edges and is the pragmatic default where the pipeline supports it. Whatever you pick, pick it once and record it, because a page re-encoded at a different quality is a different input and a defect that reproduces at quality 85 and not at 95 is unfindable if nobody wrote down which one shipped.

The independence in that first sentence is a term cancelling exactly. Dimensions decide tokens; bytes decide transfer. Moving from PNG to WebP cuts the payload to well under a quarter and leaves the token bill and the dollar bill untouched, because neither is a function of file size. So encoding is choosable on transfer time and glyph fidelity alone, with no cost model to trade against — which is unusual here, since every other knob in this chapter buys one thing by spending another.

Choosing how many images the prompt carries#

There is no principled way to send twenty page images, so the prompt has a hard slot budget and something has to decide what fills it. Three strategies are in common use, and they differ in what they optimise.

A fixed split — say three image slots and five text slots, always — is the simplest and is usually wrong at both ends. Questions answerable entirely from prose burn three image slots on pages that contribute nothing but cost, and questions whose evidence is spread across four figures get truncated at three.

A score-threshold split lets the fused ranking decide: take candidates in rank order, spend an image slot when the candidate is a page image and a text slot when it is a chunk, stop when either budget is exhausted. This adapts to the query, and it inherits whatever calibration problems the fusion has — which chapter 6 shows are considerable.

A two-stage approach sends text candidates plus low-resolution thumbnails first, lets the model name which pages it needs to read properly, and re-requests those at full resolution. It roughly halves image tokens on text-answerable queries, at the cost of a second round trip — about 1.4 s added p50 in this workload — and it fails when the thumbnail is too small for the model to tell whether the page is worth requesting, which is precisely the pages with small dense figures.

That second round trip is worth pricing against the p50 above rather than against nothing. A prompt carrying full-resolution pages already sits at the higher of the two medians quoted in the last section, and the extra hop adds nearly as much again, so a two-stage design roughly doubles the median wait in order to halve one term of the token bill on a subset of queries. There is no exchange rate between seconds and dollars here, which is why the decision belongs to product rather than to arithmetic.

The pragmatic default is the score-threshold split with a cap of four to six images and a floor of two text slots reserved regardless of scores, so a query whose evidence is genuinely textual never gets a prompt made entirely of pictures. Then measure the distribution of image slots actually used per query; if the cap is being hit on more than a few percent of traffic, the retrieval upstream is returning near-duplicate pages and the fix belongs there.

That last diagnostic reads as a statement about the prompt and is really a statement about retrieval, which is why it is worth stating explicitly. A cap binds when the ranked list keeps offering more image candidates than there are slots. A well-separated ranking runs out of plausible pages before it runs out of slots, because the evidence for one question rarely lives on six different pages. So a cap that binds often is the ranked list saying it cannot tell six pages apart, and raising the cap spends tokens on that inability instead of fixing it.

A citation to a page is not a citation to evidence#

Text RAG established a convention that a citation is a document identifier plus a span, and readers can verify a claim by looking at the span. Visual RAG breaks that convention quietly, because the natural unit — the page — is far too coarse. A citation reading "Q3 board deck, page 14" points at a page carrying a chart, two paragraphs, a footnote and a table. A reviewer checking whether the answer is grounded has to re-do the model's reading of the page to find out which part was used, which is most of the work of answering the question in the first place.

Asking the model for a normalised bounding box alongside each claim closes most of the gap. The answer comes back as structured output:

{
  "answer": "Monthly blended churn first crossed 5% in November 2023.",
  "citations": [
    {
      "evidence": 1,
      "source": "deck_q3_2024.pdf#page=14",
      "region": [0.09, 0.22, 0.94, 0.71],
      "read": "blended line crosses the 5% gridline between the Oct and Nov ticks"
    }
  ]
}

That region is enough to crop the page and show a reviewer the exact chart, with the answer beside it, in a UI that renders the crop rather than the whole page. Two caveats keep it honest. The coordinates are model output, so they are approximate — good enough to frame the right figure, not reliable to the pixel, and worth expanding by a margin before cropping. And a bounding box is a claim about where the model says it read, not a trace of where it did; a model that hallucinated a value can still emit a plausible box around the chart. The box makes verification cheap, which is a different and lesser thing than making grounding verifiable, and chapter 5 is where that distinction gets measured rather than asserted.

The second of those caveats is the third stumble in this chapter, and it has the same shape as the first two. A well-formed box feels airtight: the coordinates lie inside the unit interval, they enclose a region, and the region contains a chart. Every one of those checks passes whether the value was read or invented, because a model that knows a chart occupies the middle of a page can put a box around the middle of a page without consulting it. The property that would separate the two cases — whether the number came out of those pixels — leaves no trace in the output. What the box does buy is real and narrower: it turns a reviewer's job from re-reading a page into glancing at a crop, and a cheap check that gets run catches more than an expensive one that does not.

There is a second path worth knowing about: sending the source PDF as a document block with citations enabled, which returns page-level citation objects generated by the provider rather than by your prompt. That gives you provenance you did not have to ask for, at page granularity, and it does not let you control which pages are sent — the whole document goes. For a five-page memo that is the better tool. For retrieval over 400,000 pages it is not applicable, which is why the pattern above sends selected renders instead.

Pages the resolution table does not describe#

Two page geometries deserve special handling before they reach the resolution table at all. A landscape page rendered to a fixed long-edge budget gets less vertical resolution than a portrait page at the same setting, so a wide chart on an A4-landscape slide can be less legible than the same chart on a portrait page despite both being "2,048 px". Constraining the short edge instead, or budgeting on total pixel area, keeps legibility consistent across orientations. And a two-column page carries two independent reading streams that a single image forces the model to disentangle visually; if the layout analysis you already run for indexing can tell you which column a figure sits in, cropping to that column both halves the token cost and removes the disentangling work.

The area law explains why the landscape case is a real defect rather than a rounding error. Billing follows area, and a long-edge cap fixes only one factor of that area, so two pages billed at the same headline resolution can differ in total pixels by whatever their aspect ratios differ by. A wide page spends its allowance on width it does not need and receives less height, and height is what a glyph's stroke thickness is made of. Budgeting on area instead makes the two pages cost the same and receive the same detail per glyph, which is the quantity the 6-point column is really tracking.

The column crop pays twice for the same reason. Half the page area is half the tokens, and the halving is exact because area is what is billed, while the detail per glyph is unchanged because the crop is rendered at the same scale rather than the same long edge. That distinction is what makes cropping a saving rather than a downscale in disguise. The dependency is the thing to be honest about: it only works where layout analysis already runs and is already trusted, and a column boundary detected wrongly cuts a figure in half rather than framing it.

The two-stage pipeline's failure has a similar geometry to it. A thumbnail is a downscale, so it sits at the bottom of the resolution table, where the 6-point column reads four in a hundred. Asking a model to judge from a thumbnail whether a page is worth reading properly is asking it to make that judgement from the one row of the table where charts are illegible — which is fine for a page of prose, whose 9-point column is already respectable at that size, and useless for exactly the small dense figures the two-stage design was meant to save money on.

Every choice in this chapter took the retrieved unit for granted: the retriever returned a page, so the prompt carried a page, and the citation pointed inside a page after the fact. That ordering is backwards. The unit you embed at index time determines what the retriever can return, what a prompt slot costs, and what a citation can point at without the model having to guess coordinates. A whole page is one option among several, and it is the one that dilutes a small figure into the rest of the page — which is the next thing to decide deliberately rather than inherit.

Three claims hold the chapter together, and one habit connects them. A retrieval improvement is not a generation improvement, and the metric that certified the first is structurally blind to the second. Sending pixels is not the same as sending legible pixels, and the resolution that separates them is set by the smallest type on the page rather than by the budget. A citation that resolves is not a citation that can be checked. In all three cases the fault stays invisible because the instrument nearest to hand holds still while the property that matters moves, so the habit to carry forward is to ask, of any green number, which steps it shares with the failure it is meant to catch.