Retrieval encoders expect a different instruction prefix on each side

An embedding model turns a piece of text into a fixed-length list of numbers, arranged so that texts about the same thing come out pointing in similar directions. Retrieval systems use one at two moments: offline, every document is cut into passage-sized chunks and each chunk is pushed through the model once, with the resulting vector stored; at query time the user's text is pushed through and the store returns the chunks whose vectors lie nearest. Several of the best open models add a requirement to that picture which is easy to miss and expensive to get wrong. They expect a short fixed string — a prefix, sometimes called an instruction — pasted onto the front of the text before encoding, and they expect a different string depending on whether the text is a query or a stored document.

The strings look like decoration and are not. They are the mechanism by which one set of weights does two different jobs, and a system that omits them does not fail loudly. It returns results, the results are plausible, the similarity scores look normal, and the correct answers quietly slide down the list. The cost is usually discovered by accident, during a model comparison, and read as evidence about the model rather than about the harness that called it. That confusion is the reason this matters beyond tidiness: a model called incorrectly and a model genuinely unsuited to your documents produce the same symptom, and only one of the two is fixed by choosing a different model.

Why the model needs to be told which side it is on#

Nothing in a bi-encoder — the arrangement in which the query and the document are embedded separately, never seeing each other, and compared only through their two vectors — tells the model whether the text it is reading is a question someone typed or a paragraph sitting in an index. Both sides usually run through the same weights. That is what makes the architecture cheap, since you encode a whole document collection once offline and the query once at serving time, and it is also what creates the problem: identical weights, identical code path, identical everything, for two inputs that are not the same kind of object at all.

The two roles are genuinely different, and it is worth being concrete about how. A query and the passage that answers it are not paraphrases of each other; they are complementary. One is short, often ungrammatical, and states a lack. The other is long, well-formed, and states a fact. The training objective has to teach the encoder to map an eight-token question and a four-hundred-token passage into the same neighbourhood, while keeping two four-hundred-token passages about the same topic apart when only one of them actually answers. Those two demands pull in opposite directions if the encoder has no way of knowing which kind of text it is holding.

The older solution to exactly this problem was to use two encoders. Dense Passage Retrieval, which established the modern shape of this architecture in 2020, trained a separate BERT for queries and a separate BERT for passages — BERT being the transformer encoder these retrieval models are still mostly descended from — with no shared weights, so the role was encoded in which network you called rather than in the text. That works. It also doubles the parameters you have to train, store and serve, and it makes the model useless for anything other than the one asymmetric task it was built for.

What the prefix does to the vector#

The trick that several model families settled on instead is to make the role explicit in the input text. During contrastive training — the procedure that pushes a query's vector towards the passage that answers it and away from passages that do not — every query in every batch carried the literal token sequence query: in front of it, and every passage carried passage: . The model therefore had a reliable signal in the first few tokens of every example it ever saw, and it learned two related mappings conditioned on that signal. The prefix is a switch that selects between them.

That much is the usual summary, and it skips the step that makes the consequences predictable. The step is what the prefix physically is to the model. It is not metadata travelling alongside the text, and it is not a flag the encoder reads and then discards. It is tokens, prepended to the sequence, occupying positions the model attends over like any other. The vector that comes out is produced by pooling — averaging the model's per-token outputs, or taking the output at one designated position — across that whole sequence, prefix included. An encoder attends in both directions across the sequence, so a marker sitting at position zero is visible to every later position while their representations are being computed. A handful of tokens at the front can therefore condition the summary of the entire passage.

The causal chain from there is short and each link is checkable. Every training example the loss was ever computed on had one of those two openings attached. So the region of the output space that unprefixed text maps into was never assigned a target by the objective — no gradient ever pushed an unprefixed query towards its answer, because no unprefixed query was ever presented. So a raw query at inference time is encoded under whichever mapping the unprefixed input most resembles, which for short text is closer to the passage side, and the query vector lands roughly where a very short document would land. So cosine similarity between that vector and the stored passages measures something like topical overlap between a fragment and a paragraph, rather than the question-answers-passage relation the model was trained to express. The ranking then degrades in the way that looks exactly like a mediocre model: mostly plausible results, the right answer often at rank 6 instead of rank 1.

One link in that chain is a design decision rather than a requirement, and separating the two is what stops the rule being memorised as folklore. That the marker is the literal string query: rather than any other short sequence is arbitrary — the provider picked it, and a different provider picked search_query: for the same job. That the asymmetry is carried in text at all is also a choice: Dense Passage Retrieval carried it in weights, and one hosted provider carries it in an API parameter. What is not a choice is that whatever convention was used during training must be reproduced at inference. Which string it is was free to choose; using the one that was actually chosen is not.

The prefix approach keeps one tower and spends a handful of tokens instead, which is what lets a single checkpoint — one saved set of weights, identified by a name and a revision — serve retrieval, clustering and classification. That is why the strings look like throwaway boilerplate while doing the work that a whole second network used to do. The intuition worth holding onto is that the prefix is not an instruction the model understands. It is the part of the architecture that used to be a second encoder, relocated into the input string.

Why "it still returns results" proves nothing#

The belief that quietly costs the most here is not a mistake about the strings. It is a mistake about what counts as evidence that the strings are right. Two versions of it are common, and both are tempting for good reasons.

The first is that because the same model is loaded on both sides, the two sides are being treated the same, so there is nothing to get out of step. The premise is true and the conclusion does not follow. The weights are shared; the input is not. Asymmetry lives in the text handed to the encoder, not in which encoder is called, so a single shared model object is exactly the configuration in which the two sides can diverge without any code looking different.

The second is worse, because it feels like verification. The pipeline is run, ten neighbours come back, the top result is roughly on topic, and the conclusion drawn is that the prefixes must be working. A nearest-neighbour search cannot return nothing. It sorts every stored vector by similarity to whatever vector you handed it and returns the highest few, and it will do that for a query vector produced with the right prefix, with the wrong prefix, or with no prefix at all. The check returns the same answer in all three conditions. A defence that is constant while the fault varies is not a defence.

Score inspection fails for a related reason and is worth understanding rather than merely avoiding. When embeddings are normalised to unit length, the cosine between two of them is bounded and, in practice, real text vectors occupy a fairly narrow cone rather than spreading over the whole space. Anything vaguely on topic therefore scores inside a band a few hundredths wide. A displacement that moves a query vector out of its trained region can easily be smaller than that band, so the number on the screen barely moves. Rank does not have this problem, because rank is not a threshold on one score — it is a comparison against every other vector in the store, and inside a densely packed band a small displacement reorders a great many neighbours. That is the whole reason the verification below is written in terms of the rank of a known-correct chunk rather than its score.

Which string each family expects#

The idea generalised beyond two fixed strings. Instruction-tuned encoders such as e5-mistral-7b-instruct and the gte-Qwen2 family take a task description on the query side — Instruct: Given a support question, retrieve documents that answer it\nQuery: ... — and leave the document side raw, which lets one set of weights serve retrieval, clustering and classification from the same stored document vectors. Nomic's models use search_query: and search_document: for retrieval and offer clustering: and classification: for other uses of the same encoder. Cohere moved the same mechanism out of the text and into the API, where input_type="search_query" and input_type="search_document" do the switching for you. OpenAI's text-embedding-3 family takes no prefix at all and was trained without one.

Read down that table and the practical hazard is obvious: there is no convention. Two models from the same year, both at the top of the same leaderboard, disagree about whether the document side takes a prefix, and one of them treats adding a prefix as a defect. Read it as a record of independent decisions rather than as a lookup table and the shape is clearer still — every row is one provider's answer to the same question, and the answers differ on the string, on which side takes it, and on whether it is text at all. A harness that hardcodes query: and reuses it across candidates handicaps the ones that do not want it and handicaps the ones that want something else, and it does so quietly.

Where the strings are actually written down#

Three sources are authoritative and they should be read in this order. The model card — the README that authors publish beside the weights on a model hub, describing how the model was trained and how it expects to be called — is first, because it is where the authors state the requirement, usually in the first hundred lines and often with a bold warning. It will also tell you the cases where the prefix should be dropped, which for BGE is symmetric tasks like deduplication where both inputs are the same kind of text. The reference implementation in the card's usage snippet is second, and it is more reliable than the prose because it shows exactly how the string is concatenated — whether there is a trailing space, whether the model expects the prefix before or after any template, whether normalisation is applied afterwards.

The third source is the library's own defaults, which are machine-readable and easy to miss. Models packaged for sentence-transformers ship a config_sentence_transformers.json containing a prompts map, and the library will apply the right string if you name it:

import json
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/e5-base-v2")
print(json.dumps(model.prompts, indent=2))
# {
#   "query": "query: ",
#   "passage": "passage: "
# }

queries = ["container missed cutoff rotterdam"]
chunks = ["Where a container is not delivered before the terminal cut-off ..."]

q = model.encode(queries, prompt_name="query", normalize_embeddings=True)
d = model.encode(chunks, prompt_name="passage", normalize_embeddings=True)

That map is the least error-prone place to read the strings from, because it is the same object the library uses, and a model whose prompts map is empty is telling you it wants none. What it will not do is apply them for you: model.encode(queries) with no prompt_name runs the raw text, silently, with no warning, and that call is what most harnesses contain. The default is the unprefixed path, which is the one the training never covered.

For hosted models the equivalent is a request parameter, and the same asymmetry applies at the two call sites:

# index time, once per chunk
co.embed(texts=chunks, model="embed-english-v3.0",
         input_type="search_document")

# query time, once per search
co.embed(texts=[user_query], model="embed-english-v3.0",
         input_type="search_query")

Getting input_type wrong here is worse than getting a text prefix wrong, because there is no string in your code to grep for. The two call sites are usually in different services written months apart, and a copy-paste of the indexing call into the search path produces a system that embeds queries as documents forever without raising anything.

Verifying with one query and one document#

The check that settles it takes a minute and needs no golden set — no pre-judged collection of queries with their known-correct answers. Pick a query whose correct answer you know, embed it both ways against the same stored document vectors, and look at where the known-correct chunk lands.

import numpy as np

CHUNKS = load_chunks()                    # list[str], your corpus
GOLD = 4127                               # index of the chunk you know is right
QUERY = "demurrage charges after free time expires"

docs = model.encode(CHUNKS, prompt_name="passage", normalize_embeddings=True)

for name, prompt in [("no prefix", None), ("query:", "query"), ("passage:", "passage")]:
    q = model.encode([QUERY], prompt_name=prompt, normalize_embeddings=True)[0]
    order = np.argsort(-(docs @ q))
    rank = int(np.where(order == GOLD)[0][0]) + 1
    print(f"{name:10s} rank={rank:4d}  score={float(docs[GOLD] @ q):.3f}")

# no prefix  rank=  31  score=0.781
# query:     rank=   2  score=0.844
# passage:   rank=  18  score=0.803

Three things in that output are worth reading carefully. The first is the size of the correction: the right prefix moves the gold chunk from rank 31 to rank 2. No amount of tuning ef_search, which sets how many candidates an approximate index visits before it answers, or k, which sets how many results come back, would have recovered that, since neither of them changes where the query vector sits.

The second is the wrong prefix, and it is the most informative row because of where it falls rather than because of its absolute value. Read the three ranks as the gaps between them. Going from no prefix to the wrong prefix closes a little under half the distance to the correct answer's final position; going from the wrong prefix to the right one closes the larger remainder. That ordering is what the mechanism predicts: passage: on a query is better than nothing, because it at least puts the text into a region the training did visit, and it is still bad, because it is the wrong region. Neither half of that sentence would be predictable from a rule learned as "always add the prefix".

The third is that the raw cosine scores barely move, sitting between 0.78 and 0.84 in all three conditions, which is why an eyeball check of "does the similarity look reasonable" catches none of this. The whole spread across the three rows is a few hundredths, while the ranks they correspond to span twenty-nine positions. Absolute cosine values from a normalised encoder cluster in a narrow band for anything vaguely on topic, and the ranking is where the damage shows.

Auditing every candidate rather than one#

One condition belongs on that corrected result before it is quoted anywhere. A harness passing raw strings was passing them for every candidate it ran, so a model whose own convention puts an instruction on the query side and nothing on the document side was also being measured without the half it wants. Its score is therefore a floor rather than a reading, and the corrected number for the other candidate is only comparable against a rerun produced under that model's own stated convention.

The general form of the rule is that a prefix audit has to be per model, not per harness. Fixing one side of a two-model comparison and declaring a winner reproduces the original error with the sign flipped, and it is harder to catch the second time because the fix that was applied looks like diligence. The finished audit is one where each candidate was called the way its own card says to call it, and no two candidates were assumed to want the same thing.

When the model card does not say#

Documentation is sometimes genuinely silent. This happens most with fine-tunes — copies of a published model trained further on someone else's data and released under a new name — put out by people who assumed the base model's conventions were obvious. There the check above becomes the way to discover the answer rather than to confirm it.

Take twenty queries with known-correct documents, embed a sample of your stored documents once under each candidate document-side convention, and evaluate the four or six combinations of query prefix and document prefix that the base model's family plausibly uses. Mean reciprocal rank — the average, across those queries, of one divided by the position of the first correct document — across twenty queries separates the right combination from the wrong ones by a wide enough margin that no statistics are needed. The correct pairing typically sits ten to twenty points above the rest, and combinations that put the same prefix on both sides land in the middle, for the reason the rank experiment above showed: a matched pair of markers is at least inside trained territory, and it still collapses the asymmetry the training built. That experiment takes under an hour on a small sample and it is the only reliable answer when the card does not give one.

The string is a marker rather than an instruction#

Prefixes look like prompt engineering and behave nothing like it. The string is a token sequence the model saw millions of times during training, so it functions as a learned marker rather than as an instruction the model comprehends and acts on. Rewriting BGE's Represent this sentence for searching relevant passages: into something clearer, shorter or more specific to your domain reliably loses ground, because the rewritten string is out of distribution — unlike anything the encoder saw while it was being trained — in exactly the way the missing string was. There is no creative upside here and the correct move is to copy the string, including its trailing space, from the model card.

Instruction-tuned encoders are the exception that shows the shape of the rule. Their query-side task description is genuinely written by you and genuinely can be tuned, because they were trained across many synthetic task descriptions specifically so that unseen ones would work. Even there the gains from wording are small — a point or two of nDCG in published ablations, the experiments authors run to isolate the effect of one input at a time — and the gains from having any task description at all rather than none are large.

Two other things prefixes will not do. They will not fix a model that is wrong for your document collection: the keyword-shaped query problem from the previous chapter, where a benchmark asks well-formed natural-language questions while real users type fragments, is a property of the training mix, and prefixing a question-tuned model correctly still leaves it question-tuned. They also carry a cost you should notice before applying one to every stored chunk. BGE's instruction is nine tokens, and multiplying that by a collection of forty million chunks gives the 360 million billable tokens you would pay to embed it, plus nine tokens stolen from every chunk's sequence budget. That arithmetic is one reason several families put the longer string on the query side only, where it is paid once per search rather than once per document.

Where two services drift apart#

The place this most often goes wrong in a running system is not the model card but the boundary between two services. The indexing pipeline is a batch job owned by one team and the search endpoint is a request handler owned by another, and the prefix is a string constant in each of them. Nothing connects the two constants, no test exercises them together, and a refactor that changes one will not fail any check that looks at the other.

The cheapest structural fix is to stop treating the prefix as configuration and make the encoder a single object with two methods, encode_query and encode_document, published as a library that both services import, so that the strings live in one place and cannot drift apart. The reason that works is worth stating rather than assuming: it converts a consistency requirement that no test can see into one that cannot be expressed incorrectly, because there is no longer a second copy to disagree with the first. Where that is impractical, asserting the expected prefix pair against the model identifier at startup catches the same class of mistake at deploy time.

There is a durability problem here too. The prefix used at index time is baked into every stored vector, so changing it later, or discovering it was wrong after a full reindex, is not a code fix. Every vector was produced from a sequence that began with that string, and there is no operation on a stored vector that removes a token from the text it came from. It is the same corpus-wide operation as changing the model, for the same reason, and that reason is worked through by a later chapter in this topic, on why changing the embedding model invalidates every stored vector.

An encoder that wants a prefix is not being fussy. It is telling you that part of its architecture lives in the input string rather than in its weights, and that the string is the only thing distinguishing a question from an answer at the moment it reads them. Omit it and the model is not being asked a harder question. It is being asked a question in a form the training never covered, and its answer is defined only in the sense that any vector has neighbours.

Nothing in the stack will tell you which case you are in. Neighbours come back, scores look ordinary, and the only instrument that varies with the fault is the position of a document you already know is correct. That instrument costs one query and one known answer to build, and it is the check to run before a comparison between two models is allowed to mean anything at all. What else has to be held still for such a comparison to be readable — the document collection, the chunking, the query set, the judgments, the cutoff and the index type — and how many queries the set needs before a difference is real, is what the next chapter takes up.