The document prefix is baked into the index at build time

A retrieval system that uses an asymmetric embedding model — one trained to place a question next to the passage answering it, and told which role it is encoding by a short literal string on the front of the input — applies two such strings in two entirely different places. The query prefix is prepended inside the search handler, microseconds before the encoder runs, on text that arrived in this request and will not exist afterwards. The document prefix is prepended inside an ingestion job, and the vector it produces is written to a vector store and read back thousands of times over the following months.

Once a document has been embedded, the prefix that was in front of it does not exist anywhere except as its influence on 1024 floats. There is no field holding it, nothing to update, and no way to remove it from a vector after the fact. Changing it means producing new vectors, which means running the encoder over the corpus again. Everything operational about prefixes — how long a fix takes, whether it can be rolled back, whether the defect can be partial — follows from that one difference, and the rest of this chapter is that difference worked out in hours and in dollars.

Where each string is evaluated#

The two paths meet only at the comparison, and by then the document side has already been collapsed into stored numbers. That is the whole of the asymmetry, and everything operational about prefixes follows from it.

The last row is the one that shapes how you should think about the rest of this topic, and it is worth deriving rather than accepting. Every search request in a service runs the same handler, compiled from the same source, so at any instant there is exactly one query formatting in effect and it applies to all traffic. A document, by contrast, was written by whatever code was live on the day it was ingested, and a corpus that has been accumulating for two years contains rows from many such days. So a query-side defect is total and uniform, which makes it loud in any aggregate measurement, and a document-side defect is scoped to whatever was ingested while the bad code was live — one connector, one backfill, or one week of writes — and a corpus-wide metric will barely register it.

What a reindex actually costs#

Concrete numbers are worth having, because the decision between "fix it now" and "fix it in the next sprint" usually turns on them and estimates made in a meeting tend to be off by an order of magnitude in either direction. The figures below are for a corpus of 4.2 million chunks averaging about 180 tokens.

Start with the encoder's workload, because everything else is priced against it. Four point two million chunks at an average of 180 tokens each is a little over 756 million tokens, which is the product of those two numbers and the only quantity the inference bill depends on. Running multilingual-e5-large on a single A10G at fp16, meaning half-precision floats, and batch size 64, with inputs padded to 256 tokens, throughput lands around 450 to 500 chunks per second. Dividing the corpus by that rate gives something over eight thousand seconds, so one GPU takes about 2.4 hours of wall clock. Eight workers bring that under twenty minutes, at which point the bottleneck moves to the vector store's write path rather than the encoder.

The money follows from the same token count, and it is worth checking that it does. Through a hosted embedding API at $0.02 per million tokens, 756 million tokens is 756 times two hundredths of a dollar, which is about $15 in inference. At the $0.13 per million that larger hosted models charge, the same count comes to closer to $98. Two prices, one token count, and both land where the chapter says they land — which is the point of doing the arithmetic out loud, because it means the 756 million is a real quantity you can re-derive for your own corpus rather than a number someone reported. Neither figure is what makes a reindex a four-day event.

What makes it a four-day event is the index rebuild and the cutover. Building an HNSW graph over 4.2 million 1024-dimensional vectors with m=16 and ef_construction=200, the second of which sets how many candidate neighbours the builder examines for each vector it inserts, takes on the order of 40 to 70 minutes on a well-provisioned instance and is single-threaded in more implementations than you would hope. Storage doubles during the migration, and the size of the doubling is arithmetic you can do from the shape of the data: 4.2 million rows, 1024 components each, four bytes per float32 component, which multiplies out to about 17 GB of raw vectors, and you need the old copy and the new copy at once. And the actual delay is usually organisational: someone has to decide whether search quality is allowed to be inconsistent while the migration runs.

That decision has two defensible answers and one that is not.

Updating vectors in place, row by row, keeps one index and one copy of the data, and it leaves the corpus in a mixed state for the duration. Half the documents are encoded with the corrected prefix and half with the broken one, and since a query is scored against all of them together, the ranking during that window is drawn from two incompatible geometries. Results are worse than either the before or the after state, rather than somewhere between them, and the reason is that a ranking is a comparison. A list sorted by a score that means one thing for half its candidates and something else for the other half is not a partially correct ordering; it is an ordering produced by a rule that does not exist. For a batch job that runs overnight and finishes before traffic returns, that is often acceptable. For a 2.4-hour job during business hours it is not.

Building a second index alongside the live one and cutting over atomically, which is the blue-green deployment pattern applied to stored vectors, costs the extra 17 GB and gives you a clean switch plus an instant rollback, because the old index is still sitting there. This is the version most teams should reach for, and it is why the vector store's collection or namespace abstraction is worth using even when you only ever expect one collection. The OpenSearch write-up Asymmetric model support: optimizing semantic search for queries and documents describes the same separation from the search-engine side: the document encoder and the query encoder are configured as distinct pieces of the pipeline precisely because one of them runs at ingest and the other at search.

The answer that is not defensible is re-embedding only the documents you suspect are affected without a record of which ones those are. Selecting them requires knowing which ingestion runs used which code, and if you had that record you would already know the scope of the problem.

That record is cheap to create and almost nobody creates it before they need it. A format_version integer in each row's payload — the metadata a vector store keeps beside the vector and returns with it — bumped whenever the formatting rules change, turns "which documents are affected" from an archaeology exercise into a WHERE clause. It also makes a partial reindex safe to resume after a crash, since the rows already converted are marked, and it lets a migration run at whatever rate the encoder budget allows rather than as one long transaction. The cost is one small integer column across 4.2 million rows and a discipline about incrementing it.

-- rows still carrying the pre-fix formatting
SELECT id, source, ingested_at FROM chunks WHERE format_version < 3 LIMIT 10000;

Whether a mixed-version index is acceptable while that query drains depends on how the two formattings compare, and the answer is generally the one derived above: they are worse together than either alone. Scores from the old geometry and scores from the new one are being sorted into a single ranked list, and since both produce cosines in a similar numeric range there is no way for the search layer to tell them apart or to reweight them. If you must serve during the migration, restrict queries to one format_version at a time with a filter, accept the reduced corpus, and flip the filter at the end.

Recovering the formatting from an index you did not build#

Because the document prefix is frozen rather than lost, you can read it back out. The property that makes this work is determinism: a fixed model at a fixed revision, run at fp32, maps a given string to exactly one vector, every time. So the stored vector is a fingerprint of the string that produced it, and you can identify that string by producing candidates and checking which fingerprint matches.

Take a document whose original text you still have in the source store, embed that text under each candidate formatting, and compare against the vector already sitting in the index. Call this the identity probe, since it recovers the formatting by finding which candidate string reproduces a vector that is already stored. The correct candidate comes back at cosine 0.9999 or better and the wrong ones land well below.

import numpy as np

CANDIDATES = {
    "no prefix": "",
    "passage": "passage: ",
    "query": "query: ",
    "no space": "passage:",
}

def identify_document_prefix(model, chunk_text: str, stored_vector: np.ndarray) -> None:
    stored = stored_vector / np.linalg.norm(stored_vector)
    for name, prefix in CANDIDATES.items():
        v = model.encode(prefix + chunk_text, normalize_embeddings=True)
        print(f"{name:>12}  cosine={float(np.dot(v, stored)):.4f}")

# chunk 8f21c4, the "Rotating credentials without downtime" runbook:
#      no prefix  cosine=0.9631
#        passage  cosine=0.9702
#          query  cosine=0.9999   <- this is what was indexed
#       no space  cosine=0.9688

Read that output as gaps rather than as levels, because the levels are what mislead. The three wrong candidates sit within about seven thousandths of each other, which is to say they are indistinguishable — no ordering among them carries information. The correct candidate sits about three hundredths above the best of them, which is roughly four times the entire spread of the wrong group. A single measurement whose winner clears the runner-up by several times the noise among the losers is a measurement you can act on from one sample, and that is the property the probe is exploiting.

Two further things about that output are worth noticing. The wrong candidates are still very high in absolute terms, which is a compact demonstration of why this failure hides — a 0.96 cosine between two encodings of the same text is a small perturbation, and yet it is enough to reorder a ranked list of near-neighbours that are themselves separated by hundredths. And the probe needs no logging, no instrumentation, and no access to the code that built the index, which makes it the right first move on a system you inherited or on an index built by a job that no longer exists.

Run it over thirty chunks rather than one, and sample them from different ingestion dates and different sources. A single chunk tells you what that chunk was built with. Thirty chunks spread across the corpus tell you whether the whole index shares one formatting, which is a different and more important question, and it is the question the last row of the table above says you should expect to have.

Which side to test first#

When recall is bad and nothing is erroring, the query side is the cheaper hypothesis to eliminate and the document side is the more likely place for the defect to be partial. Testing the query side costs a single request: encode one labelled query under each candidate formatting, run each resulting vector against the untouched index, and see which one puts the known-relevant chunk highest. No data changes, no deploy, and the whole experiment fits in a REPL.

The document side needs the probe above, and if it turns out to be wrong you are committed to a reindex. That order — cheap reversible test first, expensive irreversible one second — is not a claim about which is more often at fault. It is a claim about what you can learn per unit of effort, and it happens to also protect you from the most common self-inflicted wound in this area, which is reindexing to "make sure" the documents are right and thereby destroying the evidence about what they used to be.

There is a further reason the query side deserves attention beyond its cheapness. Because it is evaluated per request, it can drift from the index without anyone touching the ingestion code at all. A model version bump, a framework upgrade that starts adding its own prefix, or a refactor that moves the constant into a config file with a different default all change the query formatting while the index sits unchanged, and the resulting mismatch looks exactly like a bad model. The durable defence is to write the formatting into the index's own metadata at build time and assert against it on service startup.

{
  "collection": "docs_v7",
  "built_at": "2026-03-04T02:11:09Z",
  "model": "intfloat/multilingual-e5-large",
  "revision": "ba46b73",
  "query_prefix": "query: ",
  "document_prefix": "passage: ",
  "normalize": true,
  "metric": "cosine",
  "chunks": 4213877
}

A service that reads this manifest at boot and refuses to start when its own QUERY_PREFIX disagrees turns a silent quality regression into a failed deploy, which is the trade you want. It costs about twenty lines and it only works for the mismatch it checks — it says nothing about whether the manifest itself is correct, or whether the code between the constant and the encoder does something else to the string on the way.

Whether a query-side change can compensate#

The question that comes up at this point, and deserves a plain answer, is whether a document-side mistake can be compensated for at query time — whether you can adjust the query formatting, or apply some correction to the query vector, so that it meets the badly-formatted documents where they are.

It cannot, in general, and the reason is a property of the encoder rather than a limitation of anyone's tooling. If the difference between passage: X and query: X were a fixed vector — the same displacement whatever X happened to be — then subtracting that vector would undo the mistake exactly, and the whole of this chapter would be unnecessary. The encoder is a deep non-linear function, so the difference depends on X: the prefix tokens interact with the document's own tokens through every attention layer, and two documents on different subjects are displaced differently. You can measure the mean difference vector over a sample and find that subtracting it recovers a couple of points, which is worse than the prefix fix by a wide margin and leaves you maintaining a correction term that has no defined behaviour on documents you have not sampled.

The one query-side change that genuinely helps is matching the documents' formatting exactly — putting query: on the query when the documents were built with query: — which restores a consistent geometry at the cost of running the model in a role it was not trained for on both sides. That is a stopgap for the hours before a reindex completes, and it is worth being precise about what it buys: consistency, not correctness. Both sides now sit in the question region, so the comparison is at least between comparable objects, and the finer question-to-answer signal from chapter 1 is still absent because neither side is playing the answer role. It is a resting state for an afternoon and not for a quarter.

The manifest has a matching limit worth naming, since it is the guard most likely to be trusted past its scope. It compares a constant in your code against a constant in a file, and both of those could have been written by the same mistaken belief on the same afternoon. It is a check for drift between two recorded intentions rather than a check on what the encoder received, and a service whose manifest and constant agree perfectly can still be handing the model something neither of them describes.

That last gap is where this gets uncomfortable, because the constant being right and the model receiving the right input are two different facts, and a surprising amount of machinery sits between them.