Multimodal embeddings and cross-modal retrieval

Read these first

• Vision-language capability beyond document parsing
• Late-interaction and multi-vector retrieval

What you will know

• Retrieve images from a text query

  A working demo should return relevant visual results for concrete descriptions, not merely images with matching metadata.

• Run matched text and image queries through two embedding models and compare nearest-neighbor agreement

  Training data, preprocessing, and normalization can cause models with compatible vector dimensions to produce materially different rankings.

• Show cross-modal retrieval failing on an abstract query

  Queries about intent, symbolism, or functionality often lack visual cues that the embedding model can reliably ground.

• Combine visual and text retrieval in one result set

  Different score distributions and candidate populations make naive interleaving produce unstable or biased rankings.

• Measure recall, latency, and per-query cost against a text-only baseline at the target corpus size

  Image encoding and vector storage can dominate the budget, while batching and precomputation change the observed tradeoffs.

Chapters

1. 1

   One shared space holds two clouds, not one

   Contrastive training pulls matched image and caption pairs together, but it never makes the two modalities interchangeable: their embeddings still occupy distinct regions of the space. Put document chunks and product photos in one index and a text query's nearest neighbours come back almost entirely text, no matter how you tune a similarity threshold. Understanding that geometry is what stops you building the mixed index in the first place.

   Read chapter
2. 2

   Matching on pixels rather than on the caption beside them

   A text-to-image demo looks impressive long before it works, because filenames, alt text and captions leak the answer into whatever you indexed. Build the retriever, then build the evaluation that can embarrass it: queries written independently of the captions, relevance judged from the picture alone. Concrete descriptive queries are where cross-modal retrieval genuinely earns its place, and this is how you show it.

   Read chapter
3. 3

   Equal dimensions do not make two embeddings compatible

   Two providers both emit 1024-dimensional vectors and neither set means anything in the other's space, because each model defines its own geometry from its own training data, preprocessing and normalisation. Run matched text and image queries through both and compare nearest-neighbour agreement; the rankings diverge materially. The operational consequence is that changing model is not a config edit, it is re-embedding the entire corpus.

   Read chapter
4. 4

   Queries that name a use cannot be grounded in pixels

   'A diagram that would work as the hero image for a post about security debt' describes an intended use, not a visible property, so there is nothing in the image for the encoder to match against. The same limit explains why 200,000 architecture diagrams are the wrong corpus for cross-modal search: the answer to 'which service talks to the payments queue' is text drawn inside the picture, better served by OCR and a text index.

   Read chapter
5. 5

   Describing an image turns cross-modal search into text search

   The alternative this topic keeps benchmarking against is a text index over descriptions, and when human captions do not exist you generate them with a VLM. That is a per-image inference bill to set beside the $400 of embeddings, and the index inherits the description model's blind spots: whatever a description omits is unfindable, whatever it invents is retrievable. A description worth indexing names the visible content, transcribes the text drawn inside the picture, and states the object relations a question might ask about. It wins on queries phrased as questions and on anything answerable from text in the image, and loses on fine visual similarity no caption bothers to record.

   Read chapter
6. 6

   Fusing two retrievers without inventing comparable scores

   Image cosine scores sit between 0.18 and 0.31 while BM25 runs from 4 to 60, and min-max normalising each over the candidates it returned forces both top results to 1.0 whether they are excellent or useless. The scale is per-query and unanchored, so the average is meaningless. Rank-based fusion, calibration against held-out relevance judgements, or a staged retrieve-then-rerank design all give you something that survives a bad query.

   Read chapter
7. 7

   Recall measured against the price of four million encodes

   0.63 recall@10 from cross-modal against 0.71 from caption text search is not a verdict, it is a prompt for per-query win/loss analysis: if they fail on the same queries, the second system buys nothing. Price the rest at target corpus size — the offline encode-and-index pass, which batching makes far cheaper and which repeats in full on every embedding model change; the query-side cost of encoding each query, and a query image too when the input is visual; the 16GB the vectors add to the store; and the latency your ANN parameters buy or spend at whatever recall you settled on. Taking the maximum score across two retrievers unions their false positives as well.

   Read chapter