Multimodal RAG

Read these first

• VLM-based parsing
• Late-interaction and multi-vector retrieval

What you will know

• Retrieve over images and charts, not just text

  A page image may contain a plotted trend, legend, or spatial relationship that OCR and captions omit.

• Match a screenshot to a question

  A useful test must distinguish visual similarity from evidence that actually answers the query.

• Run an evaluation set against text-only, visual-only, and hybrid retrieval

  Compare recall, answer correctness, citation grounding, latency, and cost so a richer index earns its complexity.

• Combine text and visual retrieval in one result set

  Scores from sparse text, dense text, and image models are not naturally calibrated, so naive merging can bury the only relevant modality.

• Trace and reproduce a failed answer from source page to retrieved evidence

  Most defects come from cropping, OCR, layout loss, or rank fusion, and generation logs alone rarely identify which stage discarded the answer.

Chapters

1. 1

   An OCR'd chart is topically right and evidentially empty

   A user asks when churn first crossed 5%. Retrieval returns exactly the right page, and the answer is still wrong: the OCR text holds the title, axis labels and legend, but none of the plotted values, so the model reads around the chart and infers a number. This chapter separates topical relevance from evidential sufficiency and names the document types that fail this way without a trace.

   Read chapter
2. 2

   A page embedding retrieves what a page looks like, not what it says

   Indexing every page as an image looks like a strict upgrade — the image contains everything the text contains, plus layout and figures. It is not. A dense page embedding encodes appearance, so a query for a specific part number retrieves pages that look like parts tables rather than the page holding that string. You will leave able to say where each index wins and which failure hides better in production.

   Read chapter
3. 3

   Retrieving the image is wasted unless the model is handed the pixels

   Fixing retrieval fixes half of chapter 1: if the generator still receives OCR text, the answer is wrong for exactly the reason already diagnosed. So the generation step has its own constraints — the model has to be vision-capable, a page image costs far more tokens than a text chunk, downscaling to fit the budget is what erases axis labels and small table cells, and only a handful of page images fit in one prompt beside the text candidates. Decide how many images the prompt carries, at what resolution, and how a citation points back at a region of a page rather than at the page as a whole.

   Read chapter
4. 4

   Page, figure or table: the indexed unit decides what a citation can point at

   What you embed decides what you can cite. A whole-page embedding dilutes a small figure into the rest of the page, a tight crop loses the caption and axis labels that make it interpretable, and a table indexed as an image cannot be filtered on its own values. Choose the unit deliberately, and record the region a retrieved result came from so a citation can point at it.

   Read chapter
5. 5

   Top-1 on a screenshot test rewards resemblance, not evidence

   Ninety-one percent top-1 on a screenshot-and-question set can prove nothing about the product. Returning the right kind of chart for the wrong quarter counts as a hit, because top-1 rewards visual resemblance rather than evidence that answers the query. Here you build a test set whose distractors look alike and differ only in the fact being asked for, and score answerability rather than similarity.

   Read chapter
6. 6

   Min-max fusion is decided by your weights, not by the evidence

   Chart questions fail even though the image candidate was retrieved: min-max normalising each score list forces the top of every list to 1.0 whether or not it holds the answer, so two agreeing text retrievers bury the lone image result. Rank-based fusion and cross-modal reranking fix it, at a stated cost. A single global weight cannot express which modality holds the answer for a given query. Assembling the merged set is its own work: candidate pools drawn at different k from three retrievers, deduplicating a text chunk and a page image that are the same page, and a reranker that has to put a paragraph and a crop on one scale.

   Read chapter
7. 7

   The visual-only arm is a control, not a shippable candidate

   Text-only 68%, hybrid 74%, visual-only 41% — the middle number alone cannot say whether the six points came from evidence in the images or from ensembling two retrievers, and those imply different projects. This chapter designs the three-arm comparison, says what each arm isolates, and fixes the numbers to report: recall, answer correctness, citation grounding, latency and cost. Score by query slice as well as by arm, labelling each question with the modality that actually holds its evidence, so a fusion change can be shown not to regress the text-answerable majority. And when the visual arm moves p95 from 400ms to 1.9s, that number is a decision: gate it behind a cheap classifier, cache the embeddings, index only pages with figures, or accept it for this workload.

   Read chapter
8. 8

   Walk the trace forward from the source page, because the prompt log lies last

   A wrong answer about a figure, and three owners blaming each other for a week. Resolving it in an hour needs artifacts that exist before the incident: the source page, parser output, the indexed unit, retrieval candidates, fused ranks, the assembled prompt. Walk it forward, because evidence flattened or mis-cropped upstream arrives in the prompt as ordinary text. And 'raise k from 10 to 30' is not a fix.

   Read chapter