Queries that name a use cannot be grounded in pixels

The content team's search log for a Thursday afternoon, three consecutive queries from the same session, with the rank-1 result each returned:

14:02  "a diagram that would work as the hero image for a post about security debt"
       → screenshot of a Grafana dashboard, 8 panels, green
14:04  "something that conveys accumulated technical risk"
       → photograph of a server rack, cables tidy
14:07  "abstract image, feeling of debt piling up"
       → stock photo of coins stacked on a wooden table

The third one is arguably a hit, and it is a hit for the wrong reason: the word "debt" is doing all of the work, and the retriever has matched it to a picture of money. Nine minutes later the same user typed "padlock circuit board" and got eleven usable candidates in one shot. Same corpus, same model, same person, and the difference in outcome is total.

The encoder scores a description against a depiction#

An image embedding is a summary of what a camera or a renderer put in the frame. A text embedding, in a jointly trained space, is positioned near the images that a description of that content would sit near. The similarity the model computes is therefore an agreement score between a description and a depiction, and it can only be high when the query names something a depiction can contain.

"A padlock on a circuit board" names visible objects and their spatial relation. Every word of it corresponds to something a pixel array can carry evidence for, so the model can be right or wrong about it and the score means something.

"A diagram that would work as the hero image for a post about security debt" names a relation between the image and three things that are not in the image: a post that has not been written, a publishing context with its own house style, and a reader whose attention the image is supposed to catch. Two competent art directors would return different sets and both be right. There is no property of the pixels that the query is asserting, so there is nothing for the encoder to be right or wrong about, and the score it produces is a function of incidental lexical overlap — which is why "debt" retrieved coins.

That distinction is measurable, including on the ground truth itself. Splitting 80 blind-written queries into 40 that name visible content and 40 that name intent or fitness-for-purpose, and having three judges grade the pooled top ten for each, gives two very different pictures. The right-hand column of the table reports Fleiss κ, which measures how often those three judges gave a result the same grade, discounted for the agreement they would have reached by guessing: 1.0 is unanimous, around 0.7 is the level a well-written annotation guideline produces, and 0 means the judges are no more consistent with each other than chance.

The last row's κ of 0.31 is the more important number of the two on that line. Recall of 0.12 says the retriever is failing; κ of 0.31 says the judges barely agree with each other about what a correct answer would have been. For a large fraction of intent-shaped queries there is no fact of the matter to retrieve, and a system that returned the "right" answer could not be distinguished from one that returned a plausible one. Chasing that row with a better model is chasing a target that is not there.

The named-entity row is the interesting middle case, because it is grounded in pixels in principle and grounded in the training set in practice. A logo is a visible object with a fixed appearance, so nothing about the query is ill-posed; whether the model can retrieve it depends entirely on whether that logo appeared often enough in the pretraining corpus for the encoder to have learned it as a distinct thing. Well-known marks retrieve well, and the class average of 0.38 is dragged down by two kinds of miss — brands too small to have been on the web at scale, and brands whose current mark postdates the model's training data, which retrieves the old logo confidently. That failure is temporal rather than structural: it gets worse with every month the checkpoint ages, and it is invisible until someone notices the results are a rebrand behind.

Why the text encoder learned descriptions and not intentions#

The training signal explains the shape of the failure. Image-text pretraining corpora are built by scraping images together with the text found beside them: alt attributes, figure captions, filenames, surrounding sentences. That text is overwhelmingly descriptive, because describing what is shown is what alt text and captions are for. A few hundred million pairs of "picture of X" plus "X" teach a model the manifold of descriptions extremely well.

Sentences of the form "this would work well as the hero image for a post about security debt" essentially never appear next to an image on the web, because that is a judgement someone makes privately while choosing an image, not a caption they publish alongside it. The text tower has therefore never seen a training example of that shape, and when you hand it one at query time it does the only thing it can: encodes it as though it were a description, dominated by whichever content words look most caption-like. "Security", "debt", "post", "hero" get encoded as though the image contains those things.

This also explains why prompt engineering the query helps a little and then stops. Prefixing "a photo of" — an old CLIP trick — reliably adds a point or two on descriptive queries because it moves the text closer to the caption distribution. It does nothing for intent queries, because the problem is not the phrasing of the intent but that an intent has no depiction.

Two hundred thousand diagrams is the wrong corpus, twice over#

A platform team with 200,000 engineering diagrams — architecture drawings, sequence diagrams, dashboard screenshots — wants to ask "which service talks to the payments queue". The proposal is to embed every diagram and cosine-search the question. It is a reasonable-sounding plan that fails for two independent reasons, either of which alone would sink it.

The first is that the answer is text drawn inside the picture. A query about which service talks to a queue is answered by reading two box labels and following an arrow, which is a symbolic operation over rendered text, and the row above puts text-in-image retrieval at 0.11 recall. The second is resolution, and it is worth doing the arithmetic because it is unforgiving.

def label_height_after_preprocess(src_w: int, src_h: int, label_px: int, target: int) -> float:
    """Height in pixels of a text label after the encoder's resize."""
    scale = target / max(src_w, src_h)
    return label_px * scale

# A 1600x1200 architecture diagram with 11px service labels
for target in (224, 336, 384, 768):
    h = label_height_after_preprocess(1600, 1200, 11, target)
    print(f"resize to {target}px  ->  label height {h:.1f}px")

# resize to 224px  ->  label height 1.5px
# resize to 336px  ->  label height 2.3px
# resize to 384px  ->  label height 2.6px
# resize to 768px  ->  label height 5.3px

Legible text needs roughly 8 to 10 pixels of cap height. At the 224-pixel input that most contrastive image encoders use, an 11-pixel label becomes 1.5 pixels — not degraded, gone. Even a 768-pixel encoder leaves 5.3 pixels, which is below the threshold at which character shapes survive. The information the query asks about is destroyed by the preprocessing step, before the network sees anything.

Suppose it survived. The encoder still pools the whole image into one 1,024-float vector, and an architecture diagram routinely contains sixty labelled nodes and eighty edges. One vector cannot encode sixty names and their pairwise relations in a way that a query naming two of them can select on; the pooling that made the diagram a point in space is the same operation that erased its structure. Two diagrams with identical layout and completely different service names sit close together, because layout is what survives pooling.

The pipeline that answers the question is not a mystery. Run OCR over each diagram, keep the extracted text with its bounding boxes, index that text with BM25 or a text embedding model, and answer "which service talks to the payments queue" by finding diagrams whose extracted text contains both terms — then, if you want the relation and not just co-occurrence, hand the top few diagrams to a vision-language model with the question attached. Routing between those paths is cheaper to build than it sounds, because the classes are lexically distinctive and the cost of a wrong route is a worse result rather than an error. A few hundred labelled queries — which the blind evaluation already produced as a by-product — are enough to fit a logistic regression over character n-grams that separates the five classes at about 0.85 accuracy, and the confusions concentrate between the two grounded classes, where either route works. The alternative of asking a small language model to classify each query costs a few milliseconds and a fraction of a cent, generalises better to phrasings the training set never saw, and is worth the dependency only if the query distribution moves quickly. Either way the router should log its decision alongside the query, because the distribution of classes in production traffic turns out to be the number that decides whether a second retriever is worth owning at all.

Cross-modal embeddings still have a job in that corpus, but it is a narrow one: "a sequence diagram with a swimlane per service" or "a dark-themed dashboard with red panels" are layout queries, they are grounded in pixels, and they retrieve well. Route by query class rather than choosing a single retriever for the corpus.

What this limit is not#

It is not a claim that models cannot reason about intent. Hand a vision-language model the security-debt question together with a candidate image and it will give you a considered answer, with reasons, and a second opinion if you ask — that capability is real and it is what the previous topic was about. The limit here is specifically about precomputation. An embedding is computed once, before any query exists, and must therefore encode the image's properties independent of what will be asked. Fitness for an unstated purpose is not a property of the image; it is a property of the pair (image, purpose), and there are unboundedly many purposes, so no fixed-size vector computed in advance can carry them.

That framing points at where the capability does fit: at query time, over a small candidate set. Judging 200,000 images with a VLM per query is 200,000 model calls, which at 1,100 input tokens each and $5 per million input tokens is about $1,100 per query and hours of latency. Judging the top 40 candidates from a cheap retriever is 40 calls, roughly $0.25 and a few seconds, and it can answer intent-shaped questions that no encoder can. The retrieve-then-reason shape is not a workaround; it is the only shape in which per-query reasoning is affordable, and it means the encoder's job is recall over a grounded query, not judgement.

It is also not a limitation that fine-tuning removes, though it deforms under pressure in an instructive way. Collect several thousand examples of the form (intent phrase, image an editor chose for it) and you can train the text tower to place "hero image for security debt" near the images your editors historically picked for that phrase. What you have built at that point is a classifier over the intents present in your training data, with all the properties of a classifier: it generalises poorly to an intent nobody has expressed before, it encodes your editors' past preferences as ground truth, and it needs new labels whenever the house style changes. That is a tag taxonomy with extra steps and a GPU bill, and it is worth building only when the intent vocabulary is genuinely small and stable — a stock library with forty editorial themes, not an open-ended search box.

Nor does the limit say metadata search is obsolete for these queries. "Hero image for a post about security debt" is a question a human editor answered once, when they tagged an image abstract / conceptual / risk, and a tag index answers it instantly and consistently. Where the tags exist, use them.

Where they do not exist — which is the usual case, and the case in the diagram corpus and in most four-million-image archives — the alternative that keeps coming up is to write the descriptions yourself, at scale, with a model. That converts a visual retrieval problem into a text retrieval problem, and it inherits every property of the model that wrote the descriptions.