One shared space holds two clouds, not one

A home-appliance retailer had 180,000 help-centre chunks already searchable and 240,000 product photographs sitting unindexed in object storage. The plan took an afternoon to write: run both through the same multimodal embedding model, put the 1,024-dimension vectors in one collection, and let a single query serve both — a customer typing "wall bracket for the 55 inch panel" would get the mounting instructions and a picture of the bracket in the same result list. The encoder handled images and text, the vectors had the same width, the distance metric was cosine for both. Nothing about the plan looked like it needed an experiment first.

The first query returned ten text chunks. So did the second, and the fiftieth. Across a sample of 500 production queries, images made up 0.4% of the top ten — nineteen image hits out of five thousand result slots, and eleven of those nineteen came from four queries that happened to be product names present in a caption. The engineer on the ticket did what the situation invites: treated it as a tuning problem. Raising the score threshold from 0.15 to 0.35 cut the result count from ten to four and left all four as text. Lowering it to 0.05 brought back forty results, still text-first, with images beginning to appear around rank 30 underneath every marginally relevant paragraph in the corpus. Boosting image scores by a fixed 0.25 flipped the problem the other way: images now dominated every query including "how do I claim under warranty", where no image is relevant at all.

The plan rested on a piece of machinery worth naming plainly before the failure gets explained. A multimodal embedding model is an encoder built in two halves, conventionally called towers: one half turns a passage of text into a fixed-length list of numbers, the other turns an image into a list of the same length, and the two were trained together on hundreds of millions of image-and-caption pairs so that a photograph lands near the words that were published beside it. A modality is a kind of input, so text is one modality and pixels are another, and the space both towers write into is called shared because a vector arriving there carries no marker saying which tower produced it. What that buys is cross-modal retrieval, which means embedding a query written in one modality and ranking items stored in another: the sentence "a red enamel kettle on a wooden countertop" finds the photograph of one, from an asset whose only text is a SKU in the filename.

That capability is the reason anyone builds this, and it exists because most image archives are unsearchable by their contents. The retailer's 240,000 photographs were described, if at all, by a product code, so every search over them was really a search over whatever words a human had typed years earlier. Cross-modal retrieval is what sits behind a stock library that answers "two people shaking hands in front of a whiteboard", behind the "find products that look like this" control in a shopping app, and behind screenshot search over a support archive that nobody captioned. Encoding both modalities into one 1,024-number format is what makes those queries answerable at all, and the rest of this topic assumes that part works.

The afternoon's design added one further claim on top of it, quietly, and that claim is the one that failed: since both towers emit 1,024 numbers into the same space, a single query should be able to rank text and images against each other in one ordered list. Nothing in the two claims above implies the third, and the gap between them is what the rest of this chapter is about.

The distributions are separated, not overlapping#

The reason none of those knobs worked becomes visible the moment you stop looking at rankings and look at the score distributions that produce them. Take 500 text queries, embed them, and compute cosine similarity against three populations: the text chunks they genuinely match, the images they genuinely match (judged by a human), and random images. Then do the same for image-to-image.

import numpy as np

def cos(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    a = a / np.linalg.norm(a, axis=-1, keepdims=True)
    b = b / np.linalg.norm(b, axis=-1, keepdims=True)
    return a @ b.T

q  = np.load("query_vectors.npy")        # (500, 1024)  text queries
tp = np.load("matched_chunks.npy")       # (500, 1024)  the right text chunk per query
ip = np.load("matched_images.npy")       # (500, 1024)  the right product photo per query
ir = np.load("random_images.npy")        # (500, 1024)  a random photo per query

for name, pool in [("text↔text", tp), ("text↔matched image", ip), ("text↔random image", ir)]:
    s = np.diagonal(cos(q, pool))
    print(f"{name:22s} mean {s.mean():.3f}  p5 {np.percentile(s, 5):.3f}  p95 {np.percentile(s, 95):.3f}")

# text↔text             mean 0.581  p5 0.412  p95 0.788
# text↔matched image    mean 0.274  p5 0.191  p95 0.343
# text↔random image     mean 0.131  p5 0.061  p95 0.204

The row that decides the outcome is the third one against the second. A perfectly matched image scores 0.274, which sits below the mean score of an unrelated text chunk at 0.301 and well below the fifth percentile of a correct text chunk at 0.412. With 180,000 text chunks in the collection, essentially every query has a few dozen paragraphs scoring above 0.35 for reasons of loose topical overlap, and all of them outrank the single most relevant photograph in the corpus. No threshold separates these populations, because a threshold is a single number applied to one axis and the two modalities are stacked along that axis rather than interleaved on it. A cut at 0.4 keeps only text; a cut at 0.25 keeps all the qualifying text plus the images; there is no value that keeps good images and drops mediocre text, because no such value exists in the data.

Changing the mix does not rescue it either, which is worth stating because the intuition that it might is strong. The retailer's collection was 43% images by count, and the reflex when images never surface is to suspect they are outnumbered. They are not being outvoted; every comparison is independent, and a query's score against an image does not depend on how many other images exist. Adding ten million more photographs would leave the top ten as text, because the top ten is decided by whether any text chunk clears 0.34, and with 180,000 chunks in the corpus several dozen always do. The only configuration in which images surface is one where the text side is small enough that no chunk happens to score above the image band — a corpus of two hundred help articles, say — and in that regime the behaviour flips per query depending on whether a mediocre paragraph happens to clear the bar, which is worse than losing consistently.

The same table also shows why the boost hack fails. The within-modality spread is wide — text-to-text runs from 0.41 to 0.79 at the fifth and ninety-fifth percentiles — while the cross-modal band is narrow, 0.19 to 0.34. Adding a constant shifts the image band upward as a block, so it lands on top of the text band rather than being merged into it in rank order. There is no additive or multiplicative constant that maps a narrow band onto a wide one in a way that preserves relative quality, which is a fact about the shapes of the two distributions rather than about the constant you pick.

The training objective never asked for one cloud#

Contrastive image-text training optimises a softmax over a batch. For a batch of N pairs, the loss asks that the correct image be scored higher than the other N−1 images for that text, and symmetrically that the correct text beat the other N−1 texts for that image. Every comparison the loss makes is between candidates of the same modality. Nothing in the objective ever compares a text-text similarity against a text-image similarity, so nothing in the objective is violated when the two similarities live on different scales. The model is free to place all image embeddings in one region and all text embeddings in another, provided that within each region the relative ordering is right — and that is exactly what it does, because it is the easiest solution to reach from where training starts.

Where training starts matters more than it sounds. The image tower and the text tower are separate networks with separate random initialisations, and a randomly initialised deep network does not spread its outputs uniformly over the unit sphere; it concentrates them in a narrow cone whose direction is a property of the weights. Two independently initialised towers produce two cones pointing in different directions, and the angle between those cones is large before a single gradient step is taken. Contrastive training then shrinks distances within pairs but does not have enough pressure to rotate one entire cone into the other, so the initial separation survives to the end of training. This has a name in the literature — the modality gap — and the 2022 paper "Mind the Gap" showed it holds across architectures and datasets, and that you can shift one modality's embeddings bodily toward the other without much affecting retrieval quality, which is the clearest evidence that the gap carries no useful information.

You can verify the geometry on your own vectors in a minute. Fit a logistic regression to predict "is this a text embedding or an image embedding" from the raw vector, on a held-out split. On the corpus above it reaches 100.0% accuracy with a margin so wide that a single direction accounts for it: project every vector onto the difference between the two modality means and the two populations separate completely, with the nearest text vector still 0.19 away from the nearest image vector along that axis. A shared space in which a linear probe can perfectly identify the modality is not one cloud with two labels; it is two clouds that a shared metric happens to be defined over.

What the gap is not#

It is not a defect in any particular provider's model, and switching vendors does not remove it. Every model trained with a symmetric contrastive objective on paired data has the same property, and the ones that report reduced gaps have usually added an explicit alignment term or a post-hoc centring step rather than changed the fundamental picture.

It is also not the claim that cross-modal retrieval fails. Within an image-only index, a text query ranks images perfectly usefully, because every comparison in that index is text-against-image and the shared offset cancels out of the ordering. The 0.274-versus-0.131 gap between matched and random images is a strong, usable signal — it is the same 2× separation you would be delighted to see in any retrieval system. What breaks is only the comparison across the two populations, which is precisely the comparison a mixed index forces you to make on every query.

Nor is it the multi-vector scoring problem from late-interaction retrieval. There the difficulty was that pooling a sequence into one point discards per-term evidence, and MaxSim recovers it; here the per-token detail is not the issue and adding vectors per item does not help, because the offset applies to every vector the image tower produces. Two different pathologies, two different fixes, and reaching for the multi-vector fix here spends storage without touching the cause.

Training a small projection head to pull the two clouds together is the fix that gets proposed once the geometry is understood, and it is a real technique with a real bill. It needs supervised pairs from your own domain — images with text that genuinely matches them, several tens of thousands of them — which is the caption corpus you were trying to avoid needing. It produces a new embedding space that is yours, with your own versioning obligations, and it is defeated by the same ranking problem one level up, because a projection that makes matched pairs score higher also makes unmatched pairs score higher and the two distributions move together. Teams that do this successfully are usually solving a different problem, such as aligning a domain-specific image encoder with a general text encoder, and the modality gap closing is a side effect rather than the goal.

Mean-centring each modality — subtracting the modality's own centroid from every vector before indexing — does narrow the gap, and it is worth knowing as a technique. It also changes the rankings inside each modality, because the centroid is not a neutral point when the corpus is not uniformly distributed over topics; on the appliance corpus, centring moved 22% of text queries' top-1 result. More importantly, it does not give you a calibrated joint scale. After centring, a matched image scores 0.44 and a mediocre text chunk scores 0.46, so the ordering is now nearly arbitrary rather than reliably wrong, which is harder to reason about and harder to debug.

What you build instead#

The design that follows from the geometry is one index per modality, queried independently, with an explicit merge step you write and own. Concretely that means either two collections, or one collection with a modality payload field and a filtered search per value — the second is operationally simpler and gives the same guarantee, since a filter restricts the comparison population to a single modality.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://localhost:6333")
qvec = embed_text("wall bracket for the 55 inch panel")

def search(modality: str, limit: int):
    return client.query_points(
        collection_name="catalogue",
        query=qvec,
        limit=limit,
        query_filter=models.Filter(
            must=[models.FieldCondition(key="modality", match=models.MatchValue(value=modality))]
        ),
    ).points

text_hits  = search("text",  limit=20)   # scores 0.41–0.79, comparable to each other
image_hits = search("image", limit=20)   # scores 0.19–0.34, comparable to each other

One caveat comes with the single-collection form. A payload filter on an HNSW graph does not restrict the graph, it restricts what the traversal is allowed to return, so a low-selectivity filter makes the search walk further to fill limit and can silently reduce recall against exact search — on this collection, filtering to the 43% that are images cost about two points of recall at a fixed ef_search compared with a dedicated image-only graph. Vector stores expose different remedies (Qdrant builds additional payload-aware links when you declare an index on the field; others recommend separate collections outright), and the general rule is that a filter which selects a stable, roughly-equal share of the corpus is fine while a filter selecting 2% is not. Two collections avoid the question entirely at the cost of two schemas to keep in step.

Two searches on the same collection cost roughly twice one search, which on a 420,000-vector HNSW index with ef_search=100 measured 11 ms against 6 ms at p95 — a real cost, and a small one next to returning nothing useful. What the two calls buy is two lists whose scores are internally meaningful, which is the precondition for any sensible merge. It is also the point at which the hard question moves rather than disappears: text_hits and image_hits now carry scores on two incomparable scales, and deciding what a combined result set looks like is a design problem in its own right that takes a chapter of its own to settle.

Before that, though, there is a prior question about the image list, and it is easy to skip because the list looks so convincing. Ranking within an image-only index does work, and a demo built on it will impress a room. Whether it works because the encoder is reading the pixels, or because the pipeline quietly indexed the filename and the caption alongside them, is not something the result list will tell you.