Matching on pixels rather than on the caption beside them

The demo that got the project funded ran for four minutes. Someone typed "a red enamel kettle on a wooden countertop" and the top three results were three red enamel kettles on wooden countertops. "Person in a yellow raincoat, seen from behind" returned six plausible hits. "Cracked concrete with moss growing in it" returned exactly that. The room was convinced, the ticket was approved, and the retriever went into the sprint as a solved component with an integration task attached.

What nobody in the room could see was the ingestion job. It read each asset's metadata row, built a string from the filename, the alt text and the marketing caption, and embedded that — because an early version of the loader had been written against a text-only endpoint and the branch that called the image endpoint was behind a config flag that defaulted to off. The vectors in the "image index" were text vectors describing images. The demo queries had been written by the same person who had read the caption spreadsheet the week before. Every query was a paraphrase of a caption, and the system was doing text-to-text retrieval on 40,000 marketing captions, which is a task it is extremely good at.

The flag was found six weeks later, when someone flipped it on to fix an unrelated bug and noticed that retrieval quality on the demo queries went down.

The leak has three doors and they are all open by default#

Metadata leakage into an image retrieval evaluation happens in three independent places, and closing one does not close the others.

The first is the index itself, as above: the vectors are derived from text that describes the image rather than from the image. This is the crudest form and the easiest to detect, and it survives code review because the call site reads embed(asset) and the resolution of asset is three files away.

The second is the query set. Queries written by someone who has seen the captions are paraphrases of captions whether or not the writer intends them to be, because a caption is what a human produces when asked to describe a picture in words, and so is a query. If the caption reads "vintage red enamel kettle on butcher block" then "a red enamel kettle on a wooden countertop" is not an independent test of anything; it is the caption with two synonyms substituted. A retriever that never looks at a pixel will score well on it as long as some text derived from the caption is in the index.

The third is the relevance judgement. If the person deciding whether result #4 is relevant can see the filename kettle-red-enamel-kitchen-03.jpg, they will use it, and they will mark as relevant an image whose visible content is a kettle in a box on a warehouse shelf. Judgements made with metadata visible measure metadata agreement, not visual relevance. The size of that effect is easy to underestimate. Running the same forty borderline results past two judge pools — one seeing the review interface as built, filenames included, one seeing only the cropped image — the filename pool marked 18 of 40 relevant and the blind pool marked 2, and the disagreement was concentrated entirely on images whose filename contained a query term and whose content did not. A judging interface is part of the measurement instrument, and an instrument that shows the answer key produces numbers that describe the key.

An evaluation built to embarrass the retriever#

The fix is procedural rather than technical: construct the query set and the judgements behind a barrier that the captions cannot cross. Two disjoint groups of people, one artefact passed between them, and the artefact is pixels.

The stratification matters as much as the barrier. Sampling 300 images uniformly from a product catalogue gives you 240 images of the two categories that dominate the catalogue, and an evaluation that reports one number over that sample tells you about those two categories. Stratify by whatever axis your queries actually vary along — category, image type, whether the picture contains rendered text — and report per-stratum, which costs nothing extra and prevents a strong average from hiding a stratum where the system returns noise.

Writers should produce more than one query per image, in deliberately different registers: one that names visible objects and their arrangement, one that names a style or mood, one phrased as a question. That variation is what lets you say later which kinds of query the retriever handles, which is a far more useful output than a single recall number and is the input to every routing decision downstream.

Judgements come back on a graded scale rather than binary, because "this is a different red kettle" and "this is a saucepan" are not the same failure and a binary label throws away the distinction. Two points for an image that satisfies the query, one for a near miss that a user would find reasonable, zero otherwise. Judge the top ten from every system in a single pooled, shuffled list so no judge knows which retriever produced which image; without pooling, the judge learns the systems' habits within twenty images and starts scoring the system rather than the result.

Two judged systems at ten results each is 6,000 judgements before deduplication and about 3,400 after, which is roughly three days of one annotator's time and the single largest cost in the exercise. It is also the reason to decide the size deliberately rather than by feel. Recall@10 measured on 900 queries has a standard error of about √(0.66 × 0.34 / 900) = 0.016, so a 95% interval of roughly ±0.03 — enough to distinguish 0.66 from 0.79, not enough to distinguish it from 0.69. Fewer than about 400 queries and the interval grows past ±0.05, at which point the evaluation cannot resolve any difference smaller than a redesign, and every comparison you run against it will be a coin flip described in decimal places.

Judging only the pooled top ten also builds in an assumption worth stating out loud: anything no system surfaced counts as irrelevant. That makes the absolute recall figure an overestimate — there are relevant images in the corpus that nobody judged because nobody retrieved them — while keeping comparisons between pooled systems fair, since every system contributed to the pool on equal terms. It also means a system added later, after the pool was judged, is measured unfairly: its novel finds score zero because they were never labelled. Re-pool and judge the increment whenever a new retriever enters the comparison, and budget for that rather than discovering it when the new system mysteriously underperforms.

The ablation that names the mechanism#

With the barrier in place, run the same query set against three indexes built over the same 300 images, and run it twice — once with the blind queries, once with queries generated by paraphrasing each image's caption. The second query set is not there to measure quality; it is there as a positive control that will light up if a leak exists.

variants = {
    "image":   embed_images(paths),                       # pixels only
    "caption": embed_texts([captions[p] for p in paths]), # caption text, same model
    "both":    normalize(embed_images(paths) + embed_texts([captions[p] for p in paths])),
}

for name, vectors in variants.items():
    index = build_flat_index(vectors)
    for qset_name, qset in [("blind", blind_queries), ("caption-derived", paraphrased_captions)]:
        hits = [index.search(embed_text(q.text), k=10) for q in qset]
        print(f"{name:8s} {qset_name:16s} recall@10={recall_at_k(hits, qset):.2f}")

Read the columns rather than the rows. The image index scores about the same on both query sets — 0.66 and 0.64 — because the queries' provenance is irrelevant to a system that only ever sees pixels. The caption index nearly triples between the two, 0.31 to 0.88, and that ratio is the leak made numerical: almost all of its apparent quality on the demo came from the queries being derived from the same text it indexed. The third row is the one that catches teams who think they are being thorough by indexing everything; summing the two vectors gives a system that is worse than pixels alone on honest queries and looks excellent on leaky ones, which is the exact combination that ships and then underperforms.

If your image row moves materially between the two columns, you have a leak somewhere in the index. On the appliance corpus that check found the config flag in eleven minutes: 0.64 against 0.88 with the flag off, 0.66 against 0.64 with it on.

Once the ratio is known to be flat, it is worth keeping it that way automatically. The whole check reduces to two numbers and a subtraction, over a 60-image subset that runs in under a minute, so it belongs in CI as an assertion rather than in a notebook someone reruns after an incident: fail the build if the image index's recall on caption-derived queries exceeds its recall on blind queries by more than a few points. What that catches is not the original bug — that one is fixed — but its many descendants, since every later change to the ingestion path is an opportunity to reintroduce a text field into the image pipeline, and none of them announce themselves.

Where the pixels genuinely win#

The blind column is also where the shape of the capability shows up. Broken down by the register the writer used, the image index's recall@10 on those 900 queries was 0.79 for queries naming visible objects and their arrangement, 0.54 for queries naming style or mood, and 0.22 for queries phrased as questions about the depicted subject. The system is strong exactly where the query describes something a camera recorded: "two people shaking hands in front of a whiteboard covered in sticky notes" retrieves that; "a stack of white ceramic plates with a chipped rim on the top one" retrieves that, including the chip.

That is not a small capability, and it is one no metadata index has. The chipped rim is in nobody's caption. Neither is "seen from behind", "shot at night", "hand-held, slightly blurred", or "the logo is partly occluded by the person's shoulder". Human captions record what someone thought was worth recording, which is a small and biased subset of what is in the frame, and the cases where cross-modal retrieval justifies its cost are the cases where the query asks about the rest of the frame.

What this evaluation does not establish#

It does not tell you the retriever is good enough to ship. Recall@10 of 0.66 on 300 stratified images is a statement about a 300-image haystack; the same system searching four million images faces a different problem, because the number of plausible-but-wrong near neighbours grows with the corpus while the number of correct answers does not. Expect the figure to fall, and measure at target corpus size before believing any of it.

It does not measure ranking quality either. Recall@10 treats a correct answer at rank 1 and rank 10 identically, and for a visual search surface where the user sees a grid, that distinction is most of the experience. Add nDCG@10 over the graded labels once the leak question is settled; keep recall as the diagnostic and let nDCG be the number you optimise.

And it is not an argument that metadata is contamination. A search index over human-written captions is a legitimate, often excellent system — on those same 900 blind queries it still reached 0.31, which is real, and on question-shaped queries it beat the image index outright. The error is not using captions; it is running an evaluation that cannot tell the two systems apart and then attributing the caption index's score to the pixels. Once the barrier exists, both systems can be measured honestly and compared on purpose, which is a comparison worth having and one this topic returns to.

The immediate consequence of having an honest evaluation is that you can finally use it for something, and the first thing every team uses it for is model selection: run two providers through the same 900 queries and pick the winner. That comparison surfaces a property of embedding spaces that the API docs state plainly and almost nobody reads as an operational constraint — that two models emitting vectors of the same width are not emitting vectors of the same kind.