A query naming nothing the index contains cannot be retrieved

Between what a person types and what the retriever is given, there is a stage that can rewrite the text, and that stage is where this chapter works. Query transformation takes the user's string and produces one or more different strings to embed and search with: by shortening it, by resolving a pronoun against the previous turn, by restating it several ways, by splitting it into parts, or by replacing it with a passage that reads like the answer. Nothing downstream changes — same embedding model, same index, same generator — and only the text that reaches the index is different.

The stage exists because an index is matched against rather than asked. A passage comes back when its vector sits near the query's vector, so a question that never uses a corpus's words — a corpus being the whole set of documents that were embedded and stored — or that names its own subject only in a previous turn, produces a vector with nothing to sit near, and the answer stays in the index unreachable. In practice you meet query transformation as a component in front of retrieval in almost every product where a human types the question rather than a benchmark supplying it: a support console, an internal assistant, the chat box on a docs site. What makes it worth a topic rather than a paragraph is that "the question was unretrievable" is four distinct conditions wearing one label, and the repairs are not interchangeable.

Four ways a question arrives unretrievable#

Four real failures make the distinctions concrete. They are worth reading together, because each of them went through the same pipeline — the same embedder, the same 12,400-chunk index over runbooks, product docs and release notes, the same top-10 — and in a weekly dashboard they collapse into one bar labelled recall failure.

A  "hey quick one — for the 2023 filing, did we NOT have to restate Q3 revenue?"
B  "does that one cover weekends?"
C  "how do I rotate the signing key"
D  "which regions are still on the old rotation policy and what does migrating each one cost"

Every one of these returned ten chunks and none of the ten answered the question. The evidence exists in the corpus for all four: there is a passage stating that Q3 2023 needed no restatement, a page describing weekend coverage for the enterprise escalation tier, a runbook for rotating the JWS signing certificate, and a finance sheet giving per-region migration costs. The retriever is working the way it was built in topic 9.2, and the index contains the answers. What fails is upstream of all of it, in the text that gets embedded.

Why the text should matter so much is worth deriving rather than assuming, because the whole topic rests on it. The query vector is a summary of the tokens in the query, and that phrase can be unpacked into a mechanism. An embedding model reads a string as tokens and returns one vector for the whole string, produced by combining the per-token representations into a single point. That combination is an average of a sort, which has two consequences. The first is that a concept absent from the tokens is absent from the vector, since there is nothing in the input carrying it. The second is that a concept present in few tokens contributes little to the average, because each token's pull on the result is limited by how much of the string it occupies. So anything that changes which tokens are there, or what proportion of the text they occupy, changes where the query lands — and those are two different levers, which is what gives four defects rather than one.

Query A carries conversational filler around a constraint. The tokens that determine relevance — 2023, Q3, restate, and the negation in did we NOT — are present, but they share the string with an opener that contributes nothing about the subject. Filler is the mildest of the four; dense retrievers, meaning the ones that match a query to a passage by the distance between their embedding vectors rather than by shared words, tolerate a fair amount of it, and A's actual retrieval was not catastrophic. Its top-10 held four restatement passages, two of them from the right year. The reason filler matters is that it is the defect people are most eager to repair, and the repair is where the damage happens.

Query B has an unbound referent. Two turns earlier the assistant described a standard tier and an enterprise tier, and that one points at the second. The word "enterprise" appears nowhere in the query text, which means it appears nowhere in the query vector, which means there is nothing for the enterprise coverage passage to match on beyond weekends. The retriever returned weekend-coverage text from four different products, which is a faithful answer to the question that was actually asked. Note what the retriever did not do: it did not fail. It found the nearest passages to a vector built from one content word, and there is no sense in which it could have known which of four products was meant.

Query C has a vocabulary gap. It is a single, clear, well-formed intent. The runbooks call the operation certificate rollover and the artefact the JWS signing certificate; nobody who wrote them has ever typed "signing key". In the index, signing key as a phrase appears in three chunks, all of them changelog entries about an unrelated deprecation, while certificate rollover appears in forty-one. The correct runbook chunk scored 0.71 against the query; the tenth result scored 0.78. The document is seven hundredths of a cosine away and might as well be in another corpus.

That gap is small enough to be worth reading carefully, because its size is the argument. Nothing needs to move very far for the right chunk to surface — it has to overtake one score, not climb from nowhere — and that is why a repair which changes only the query's wording can work at all. It also sets the bar a repair has to clear: any transformation that raises the correct chunk by less than the distance to tenth place buys nothing, since the top-10 cut is a threshold rather than a gradient. A change that lifts the runbook chunk by a few thousandths is indistinguishable in the results from no change at all.

Query D is compound: two intents in one sentence, one asking which regions are on the old policy and one asking what migration costs. One query produces one vector, and that vector has to represent both intents at once. In practice it does not split the difference evenly — it lands near whichever intent contributes more mass. Here the rotation-policy vocabulary occupies most of the sentence, so the vector sits in policy space and all ten results are about rotation policy. Retrieved separately, the phrase regional migration cost scores 0.81 against the finance sheet; inside the full compound query, that same sheet scores 0.62 and never enters the top-50.

Those two scores for one document are the most informative pair of numbers in the chapter, because they measure the dilution rather than assert it. The finance sheet did not move and the index did not change; only the string being embedded did, and the sheet's score fell by roughly a quarter of its standalone value. Meanwhile the rotation-policy passages held their scores well enough to fill all ten slots, which means the same act of combining two intents cost the two intents very different amounts. A uniform effect — a query that got vaguer and pulled everything down equally — cannot produce that. The mass argument can: the side with more tokens keeps its position and the side with fewer is pushed away, and the size of the push is set by the share of the string each side occupies.

The fourth column names the repairs this topic builds, and they are worth having in plain terms before any of them is defended. A rewrite sends the query through a model with an instruction and uses whatever comes back, either to shorten it or to put a missing entity into it. Expansion, also called multi-query, asks a model for several phrasings of the same question, retrieves for each of them, and merges the results into one pool. HyDE, which stands for hypothetical document embeddings, has a model write the answer it expects to exist, throws the text away, and embeds that fake passage in place of the question. Decomposition splits a question that is really two questions into parts that can be retrieved separately and joined afterwards.

The column that does the most work is the third one. B is missing a fact that exists outside the query and can only be supplied from the conversation. C is missing words that exist outside the query and can only be supplied from the corpus, or from a model's knowledge of how such documents are usually worded. A and D are missing nothing at all; their content is complete and the problem is how it is packaged.

That column sorts the four defects into two classes with no residual, and the sort is exhaustive by construction rather than by luck. Either the material needed to reach the right passage is inside the query string or it is not, and there is no third option, so every defect lands in exactly one class. The repairs sort the same way. A contextual rewrite and the expansion family both bring in text from somewhere the query is not — the conversation, or the model's sense of how documents are worded — while filler-stripping and decomposition only rearrange what the user already typed. Reading the two sorts against each other, the fourth column pairs each defect with a repair of its own class and never crosses over. That is what licenses the rule stated plainly: repairs that supply material cannot fix packaging problems, and repairs that repackage cannot supply missing material.

Why the wrong repair leaves the query exactly as broken#

Take C and D, since they are the pair most often confused. Both look identical in the log: ten results, zero relevant, one clean-sounding user question. The tempting response is that both are recall failures, so both get multi-query — five paraphrases each, retrieve for all of them, merge.

For C that works, because paraphrasing is a way of sampling the space of phrasings for one intent, and one of the samples has a decent chance of using the documents' words. A model asked to rewrite "how do I rotate the signing key" five ways will produce something containing "certificate", "rollover" or "renew the signing certificate" with high probability, and that leg retrieves the runbook. The mechanism is worth naming because it explains the probability: the model has read a great deal of writing about certificates, so the phrasings it reaches for are drawn from roughly the same distribution the runbook author was drawing from. Sampling five times from that distribution is a real chance of collision, and each additional sample is another chance.

For D it cannot work, and the reason is structural rather than probabilistic. Every paraphrase of a two-intent question is still a two-intent question, because a paraphrase preserves meaning by construction. Each variant still contains both the policy terms and the cost terms, each variant still has the policy terms dominating, and so each of the five legs returns approximately the same policy passages. The union of five sets that are nearly identical is one set that is nearly identical, and there are no cost documents anywhere in it to merge. Running D five times is running D once at five times the price.

The stumble to name here is not an arithmetic slip; it is a belief that a defence works. A reader who has just met multi-query will confidently write down that it is a general-purpose recall repair, and the sentence is tempting for a good reason — multi-query does raise recall on a large fraction of real traffic, and a dashboard will show it doing so. What makes the belief false is that its coverage is constant while the thing it must discriminate varies. Multi-query samples phrasings, and a query's phrasing is exactly what is wrong with C and exactly what is not wrong with D. The same defence, applied to both, returns the same kind of answer for the query it can help and the query it cannot, so nothing in the output tells you which case you were in.

The inverse substitution fails just as cleanly. Decomposing C splits a single intent into fragments — "how do I rotate" and "the signing key" — neither of which is a question anybody's document answers, and both of which carry the same wrong vocabulary that broke the original. A decomposer that is well-behaved returns C unchanged, which is the best possible outcome and still leaves you where you started, having paid for a generation call.

The order in that diagram is not arbitrary, and each edge earns its position by making the next test well-defined. Referent binding comes first because until the query names its subject, no other test means anything — you cannot judge whether B's vocabulary matches the corpus when the query contains one content word. The compound test comes before the vocabulary test because a compound query has two vocabularies to evaluate and evaluating them jointly gives you a blurred answer for both. Run in that sequence, each question is asked of a query for which it has an answer, which is why the diagram reads as a decision procedure and not as a ranked list of things to try.

Which defects you can see before retrieval runs#

Three of the four are detectable from the query text and the conversation, at accuracy good enough to route on. The fourth needs the index.

An unbound referent is a demonstrative, pronoun or definite description whose antecedent is not in the query. that one, the second one, it, the same plan, the other tier — a shallow parse plus a check that no candidate antecedent appears in the query itself gets most of them, and the false-positive cost is low because a contextual rewrite of a query that did not need one is close to a no-op. That last clause is what makes the detector worth shipping at its accuracy rather than at a better one: the cost of a false positive and the cost of a false negative are wildly asymmetric here, so the threshold should sit far towards catching everything.

A compound query is two coordinated question predicates, or two clauses with different expected answer types. In D, the first clause expects a list of regions and the second expects a number per region, which is a stronger signal than the presence of and on its own — plenty of single-intent queries contain conjunctions ("latency and throughput on the ingest path" is one intent about one subsystem). Asking a small model for the count of distinct answer types is more reliable than any regex here, and it is cheap enough to run inline. This is a design decision rather than a requirement: what the system needs is a signal that separates one intent from two, and counting expected answer types is one way of getting it that happens to be robust to conjunctions. A different signal would satisfy the same requirement.

Filler is detectable and barely worth detecting, since the classifier is a list of openers and the payoff is small.

The vocabulary gap is the one you cannot see in the query. Nothing about "how do I rotate the signing key" is malformed; it is a good sentence, it is unambiguous, and a human support engineer would answer it immediately. Its defect is a property of the relationship between the query and the corpus, so the only pre-retrieval signal available is corpus-side statistics: take the content terms, look up their document frequency in the index — the count of chunks in which that word appears at all, which the index already keeps for its keyword side — and flag the query when its distinguishing terms are rare or missing.

def diagnose(query: str, history: list[str], stats: IndexStats) -> set[str]:
    """Pre-retrieval defect flags. `stats` is built from the live index."""
    defects: set[str] = set()

    if unbound_referents(query):                 # text-only
        defects.add("referent")
    if distinct_answer_types(query) > 1:         # text-only, small model
        defects.add("compound")
    if FILLER_OPENERS.match(query.strip()):      # text-only
        defects.add("filler")

    # Not text-only: needs the index.
    rare = [t for t in content_terms(query) if stats.doc_frequency(t) <= 2]
    if rare:
        defects.add("vocabulary")

    return defects

That last check has a real failure mode worth knowing before you rely on it. Document frequency is a lexical measure, meaning it counts the exact word forms present, and dense retrieval is not lexical, so a query using a rare surface form whose meaning is well covered by the embedding space will be flagged when it did not need to be. The reverse is more dangerous: a query whose terms are all common in the corpus can still miss, because the terms are common in the wrong sense. key has a document frequency of 2,900 in this index, almost all of it about API keys and partition keys, and none of that helps query C. Document frequency tells you about presence, and relevance is about sense, so the signal is a prior worth routing on rather than a verdict.

Run the function on query C and the point stops being a caveat and becomes a result. unbound_referents finds nothing, since C names its own subject. distinct_answer_types returns one, since C asks for a procedure and nothing else. The opener does not match, since C has no opener. And the rare-term list comes back empty, because rotate, signing and key are all ordinary words with ordinary counts in a body of operational documentation — key alone appears in thousands of chunks. So diagnose returns an empty set for the one query in this chapter that is definitely broken.

That result is worth stating as a finding rather than as a shrug, because it was reached by a route sharing no step with the argument above it. The prose case for C's undetectability was semantic: presence is not sense. The function's case is mechanical: every branch is a text-level or count-level test, and C passes all of them. Two independent routes landing on the same verdict is what licenses treating the vocabulary gap as a genuinely different kind of defect rather than as a detector that needs tuning. There is a real residual, and it should be named: diagnose returning an empty set is also what a healthy query looks like, so the function cannot distinguish "nothing wrong" from "wrong in the one way I cannot see". Routing on its output means routing a broken C down the healthy path.

As an analogy — and it is an analogy, so it should be held loosely — document frequency is a catalogue that records which books contain a word without recording which meaning of it. Two books both listing "bank" tell you nothing about whether either is about rivers. The place the analogy breaks is worth naming, because that break is where a misconception forms: a catalogue is at least exact about presence, and a dense retriever never consults presence at all. It matches on a vector that has already blended the word into its neighbours, so a query can score well against a chunk containing none of its words and badly against one containing all of them. The catalogue picture makes document frequency look like a weak version of what the retriever does, when it is in fact a measurement of something else.

Retrieval failures that are not in the query#

Everything above assumes the failure is in the query. That assumption is doing an enormous amount of work, and for the four queries in the review queue it happens to be true, because someone opened the corpus and confirmed by hand that each answer exists.

Retrieval fails for at least three other reasons that no rewrite touches. The document may not be in the index at all, because an ingest job skipped a file type or a PDF came back as three pages of ligature noise. The passage may be split so that the answer and the entity that identifies it landed in different chunks, in which case the chunk holding the answer is genuinely not relevant to any phrasing of the question. Or the passage may have ranked eleventh, which is a ranking problem that a better query would only accidentally repair.

Those failures wear the same costume in the log as the four above: ten results, nothing useful, an annoyed user. Guessing which kind you have from a single example is a coin flip, and the four defects here are only worth the engineering they imply if they account for a meaningful share of what is actually failing. The eleventh-place case deserves particular suspicion, because it is the one a transformation can appear to fix. Any change to the query reshuffles the tail of the ranking a little, so a passage sitting just outside the cut will sometimes cross it, and the log will record a success that generalises to nothing.

The diagnosis therefore has a ceiling, and it is lower than it first reads. It sorts query-side defects into four kinds and routes each to a repair, on the condition that the failure is query-side at all, and it offers no test of that condition. Establishing it by hand works for four queries and does not work for four thousand.

Everything a transformation can do is decided before this chapter's diagnosis runs, by whether the question is missing material or merely packaged badly, and by whether the answer was reachable in the first place. The four defects are worth separating because their repairs do not substitute for each other: a paraphrase cannot split an intent, a split cannot supply a word, and neither can name an entity the user only ever said two turns ago. What the diagnosis cannot do is tell you it applies.

Before writing any transformation, then, you need a way to separate a bad question from a missing document, and it needs to work over a population rather than one query at a time.