Recall measured against the price of four million encodes

The numbers on the slide were 0.71 and 0.63. Caption text search, built and running, recall@10 of 0.71 on the blind evaluation. Cross-modal retrieval, prototyped over the same 300-image sample, 0.63, with an estimated $400 to index four million images and 16 GB added to the vector store. The engineer who built the prototype argued that cross-modal is where the field is going and that an eight-point gap on a 300-image sample is not a reason to throw away the direction. Someone else offered the compromise that always gets offered: ship both, take the maximum score across the two retrievers, get the best of each.

Nothing on that slide is sufficient to make the decision, and the compromise is the worst of the three options on the table.

The two numbers hide four populations#

A pair of aggregate recall figures tells you nothing about whether the two systems fail together. Two retrievers at 0.71 and 0.63 could be nearly the same system with a slight quality difference, or two systems with almost disjoint competence. The union ceiling — the best any combination could reach — differs by a factor of two between those cases, and it takes one join to find out which you have.

def winloss(gold: dict[str, set[str]], a: dict[str, list[str]], b: dict[str, list[str]], k=10):
    q = {"both_hit": [], "a_only": [], "b_only": [], "both_miss": []}
    for qid, relevant in gold.items():
        ha = bool(relevant & set(a[qid][:k]))
        hb = bool(relevant & set(b[qid][:k]))
        q[("both_hit" if hb else "a_only") if ha else ("b_only" if hb else "both_miss")].append(qid)
    return q

cells = winloss(gold, caption_hits, crossmodal_hits)
n = sum(len(v) for v in cells.values())
for name, qs in cells.items():
    print(f"{name:10s} {len(qs):4d}  {len(qs)/n:5.1%}")

# both_hit    238  59.5%
# a_only       46  11.5%     caption search finds it, cross-modal does not
# b_only       31   7.8%     cross-modal finds it, caption search does not
# both_miss    85  21.2%

Caption search covers 284 of 400 queries, cross-modal covers 269, and the two together cover 315. The union ceiling is 0.79, so a perfect combiner could add at most 0.08 recall over the caption baseline — and 0.08 is the ceiling, not the expectation, because no combiner is perfect and every one of them also changes what appears in the ten slots that were already correct.

The headline gap deserves an interval before it is argued over at all. A paired bootstrap over the 400 queries — resampling queries rather than judgements, since both systems answered the same ones — puts the 0.08 difference at a 95% interval of [+0.02, +0.13]. That is a real advantage for the caption index rather than noise, which is worth establishing explicitly, because the same procedure on the 300-image prototype sample gave [−0.04, +0.19] and the slide was quoting a difference the data could not support at the size it was measured.

The largest cell in that table is also the one nobody discusses in the meeting. Eighty-five queries, 21% of the set, are answered by neither system, and no fusion, calibration or reranking strategy touches them because nothing nominates a correct result to rank. Reading those 85 by hand took an hour and split them into three groups: 44 whose correct answer is not in the corpus at all, which is a catalogue-coverage problem and not a retrieval one; 27 that are the intent-shaped queries from chapter 4, which need a reranker with reasoning rather than a better index; and 14 genuine retrieval misses. The largest available quality win in this system is therefore not a retriever at all, and the win/loss table is what makes that visible — which is the strongest argument for running it before, rather than after, the build-or-not decision.

That is what makes the max-score compromise a mistake rather than a hedge. Taking the maximum across two retrievers unions their false positives as well as their true ones: on this evaluation, fusing by max lifted recall@10 from 0.71 to 0.76 and dropped precision@10 from 0.42 to 0.33, because every confidently wrong result from either system now enters the list at its own confidence. The mechanism is the one from the previous chapter wearing different clothes — a maximum over uncalibrated scores is a per-query comparison between two scales that were never made comparable, and since cross-modal cosines occupy a narrow band, any normalisation that widens them makes cross-modal win ties it has not earned. Max fusion is min-max fusion with the averaging step removed.

Which queries, and how many of them are real#

The 31 queries only cross-modal answers are the entire case for building it, so they deserve to be read individually rather than counted. Classified by the scheme from chapter 4: 22 are fine visual detail or capture conditions — the chipped rim, the off-centre crop, "shot at night"; 6 name visible content that the generated description happened to omit; 3 are cases where the caption index returned the right item at rank 11 and the difference is noise.

The next step is the one that decides the question, and it is not an improvement to the retriever. Sample 400 queries from the production log by frequency, classify them with the same scheme, and ask what share of real traffic falls in the class where cross-modal wins. On the appliance store, fine-visual-detail and style queries were 4.1% of traffic. Multiplying through, the traffic-weighted recall gain available from adding cross-modal is 0.041 × (0.44 − 0.19) ≈ 0.010 — one point of recall on one query in twenty-five, in exchange for a new index, a new pipeline, a re-embedding obligation on every model change, and a fusion layer to maintain.

That is the analysis, and it points at "not yet". It also names its own reversal conditions precisely, which a bare recall comparison never does. If the same measurement on a stock-photography search, where style and capture-condition queries are 30% of traffic, the arithmetic gives 0.075 and the answer flips. If the product is about to ship photo-upload search, the query distribution changes to one where the query itself is an image, a case the caption index cannot serve at all and where cross-modal is the only option — and the evaluation has to be redone against the traffic you will have rather than the traffic you have, because today's log contains no image queries by construction.

Pricing it at four million rather than at three hundred#

The prototype's $400 estimate is one line of a cost model with five, and the other four are where the surprises live.

The offline encode-and-index pass is the line everyone quotes. Four million images at $0.00007 per image is $280 through a provider API. Self-hosting the same encoder on a single L4 changes the shape of the bill entirely and is worth measuring because the throughput depends almost completely on batching: 38 images per second at batch size 1, 210 per second at batch size 32, on the same GPU. That is 29 hours against 5.3 hours, or $21 against $4 of compute at $0.71 per hour — a 5.5× difference produced by one parameter, and the reason a naive per-image loop makes self-hosting look uncompetitive with an API whose price already assumes batched inference. The real self-hosting cost is the engineering around it, not the GPU hours.

The re-encode obligation is the line that most deserves attention in the plan, because it is the only one that repeats in full. Every embedding-model change re-runs the whole pass — the entire $280 and the entire 5.3 hours — and, during the dual-index window from chapter 3, holds both copies simultaneously. That is not an argument against changing models; it is an argument for budgeting model changes as recurring events rather than one-time migrations, and for making the pipeline re-runnable from a cold start rather than a script someone ran once from a laptop.

The query side is small per query and not zero. A text query encode is about 8 ms and $0.00002; at 400,000 queries a day that is $8 a day, or $2,900 a year — comparable to ten full re-encodes of the corpus, which is a ratio worth knowing before optimising the wrong one. When the input is an image, the query-side encode is a full image encode: 45 ms and $0.00007, and it cannot be cached the way a repeated text query can.

Storage is 4,000,000 × 1,024 × 4 bytes = 16.4 GB at float32, 8.2 GB at float16, and about 4.1 GB with int8 quantisation, plus roughly 0.6 GB for an HNSW graph at m=32. What matters operationally is rarely the gigabytes but the step they push you across: 16.4 GB of vectors plus the graph plus the process does not fit alongside everything else on a 16 GB node, and the decision is really about moving to a 32 GB instance or adopting quantisation and accepting its recall cost.

Latency is bought and sold with the ANN parameters, and its effect on quality compounds with everything above.

The 0.63 recall figure on the slide was measured against exact search over 300 images. Shipping at ef_search=40 multiplies it: 0.63 × 0.91 = 0.573, and the gap against the caption baseline widens from 0.08 to 0.14 for reasons that have nothing to do with the model. Quote the recall you will actually serve, at the index parameters you will actually run, at the corpus size you will actually hold — three qualifiers that between them moved this system's headline number by more than any modelling decision in the topic.

The sixth line has no unit price and is usually the largest. A second retriever is a second ingestion pipeline with its own failure modes, a second index to keep in sync with catalogue deletions, a fusion layer with its own metrics, an evaluation that now has to be re-pooled whenever either side changes, and a page in the runbook that someone has to be able to read at three in the morning. Estimating that at a few days of build and an ongoing fraction of one person is not false precision; it is the difference between a $280 decision and a $40,000-a-year one, and it is the line that most often flips a marginal case. It also argues for building the second retriever behind the first rather than beside it, since a flag-gated path serving 4% of traffic carries a fraction of that surface.

The shape of the decision, and what would change it#

The call on this corpus is to ship the caption index as the primary retriever, keep the cross-modal prototype behind a flag routed to the query classes where it wins, and write down the three conditions that would promote it: the fine-visual-detail share of traffic exceeding roughly 15%, the launch of image-input search, or a model release that moves the per-class numbers by more than the paired-bootstrap interval on the blind evaluation. Re-run the per-class measurement quarterly; it costs an afternoon once the query set exists.

The colleague's argument is not wrong, and it deserves a better answer than a table. Cross-modal retrieval is improving quickly, the training recipes are public, and a system built around a caption index has an assumption baked into it that a better encoder would invalidate. What the cost model contributes is the observation that waiting is unusually cheap here: the option to adopt cross-modal later costs one encode pass, $280 and an afternoon, and nothing about building the caption index forecloses it. When re-running the decision is that cheap, "not yet, and here is what would change it" is a stronger position than either shipping on a hunch or refusing on a single number.

What that leaves is a small set of things to run, in an order that has emerged over this topic. Build the blind query set first, because every number afterwards depends on it and it is the only artefact here that cannot be regenerated in an afternoon. Run the index ablation to prove you are matching pixels and not metadata. Measure per query class rather than in aggregate, so the result is a routing decision rather than a verdict. Keep each retriever's index single-modality and single-model, with the model identity on the row. Fuse by rank until you have judgements, then by calibration. And price the whole thing at target corpus size with the ANN parameters you will serve, so that the recall on the slide is the recall the user gets.