A hypothetical answer supplies vocabulary and invents specifics

HyDE — hypothetical document embeddings — has a model write the answer it expects to exist, throws the answer away, and embeds that fake passage in place of the question. It exists because of a mismatch in form. A question is an interrogative written in the user's words; the passage that answers it is a procedure written in the documentation's words, and an encoder asked to place those two near each other is being asked to bridge a difference in phrasing on top of a difference in vocabulary. Writing something document-shaped and searching with that removes one of the two gaps before retrieval ever runs.

The measurements below come from a 300-question eval over an internal support corpus. Against that set the step moves recall@10 from 0.55 to 0.68, end-to-end answer accuracy from 71% to 74%, and latency up by 380 ms. Two terms are needed to read the rest. Recall@10 is the share of questions whose known-correct chunk appears in the ten retrieved. A version-sensitive question is one whose correct answer differs between releases of the product, so that a passage about the right subject and the wrong release is a wrong answer rather than a near miss.

What is actually being embedded#

The step generates an answer to the question, throws away the answer, and embeds it in place of the question.

HYDE = """Write a short passage from the product documentation that
answers this question. Write it as documentation, not as a reply."""

def hyde_search(question: str, index, k: int = 10) -> list[Chunk]:
    hypothetical = llm(HYDE, question, temperature=0.7, max_tokens=180)
    return index.search_by_text(hypothetical, k=k)

For how do I rotate the signing key, the model produces something like this:

Signing certificate rollover is performed from the Security section of
the admin console or through the CLI. Run `cc-admin cert rotate
--service auth --grace 24h` to issue a new signing certificate while
keeping the previous one valid for the grace period. In release 3.1 and
later the rotation is applied without restarting the auth service; on
earlier releases you must restart each replica after the new
certificate is distributed. Update the `signing.key.rotation_grace`
value in the service configuration if the default 24-hour window is
too short.

Almost none of that is verified. The CLI flags may not exist, the config key name is a guess at a plausible naming convention, and the claim about release 3.1 is invented outright. The passage is never shown to a user and never used as an answer; only its vector reaches the index.

What makes it work is that the vector lands in the region of the space where documentation lives. The query "how do I rotate the signing key" is an interrogative in the user's vocabulary; the target chunk is an imperative procedure using the corpus's vocabulary. Embedding a question and a document and expecting them to be close asks the encoder to bridge a form gap on top of a semantic one, and the original HyDE result — Precise Zero-Shot Dense Retrieval without Relevance Labels — is that generating a fake document and comparing document to document beats comparing question to document, without any labelled relevance data. The hypothetical contributed rollover, certificate, grace period and admin console, which is exactly the vocabulary chapter 1 established was missing, and the correct runbook moved from rank 14 to rank 2.

Worth walking is why the step is allowed to work at all, since it looks like it should not. Nothing checked whether the passage is true, and the retrieval result improved anyway. The reason is that an embedding of a passage is not a record of the passage's claims; it is a summary of the words in it and the way they are arranged. Two documents about certificate rollover land near each other whether or not either is accurate, because accuracy is not one of the things the encoder measures. So the truth of the hypothetical is irrelevant to the part that helps — and, by the same argument, irrelevant to the part that hurts. Every token in the passage steers the vector with equal standing, and the vector carries no marker saying which tokens were known and which were guessed.

That is an analogy away from being intuitive, and the analogy is worth stating along with where it fails. A hypothetical passage is a police sketch of the document you are looking for: right about the type, unreliable about the particulars, and useful precisely because a type is what you are searching on. Where it breaks is that everyone reading a sketch knows it is a sketch. A vector arrives at the index with no such understanding attached, so the invented particulars are matched against real ones with full confidence, and the retrieval step has no way to weight them down.

The same sentence that helped also steered#

The words that raised recall did not arrive alone. In release 3.1 and later is in the embedded text with the same weight as everything else, and the index contains version-tagged pages for 3.0, 3.1, 4.0, 4.1 and 4.2, all of which describe certificate rotation in nearly identical prose. The one distinguishing signal between those five near-duplicates is the version string, and the query vector now carries one — an invented one.

The damage is concentrated for a structural reason worth stating in causal steps. Five pages describe the same procedure in nearly the same words, so on every dimension except one they are indistinguishable to the encoder. A vector that is a good match for one of them is therefore a good match for all five, and whatever separates them has to come from the single dimension on which they differ. That dimension is the version string. So a query carrying a version string picks one of the five almost deterministically — which is exactly why the correct version, when the user supplies it, is such a powerful signal, and exactly why a wrong one is such a powerful error. The invention is not competing with a weak signal; it is occupying the position where the only strong signal lives.

Measured on the same eval, the number of retrieved passages that are on-topic but tagged to a different product version than the question implies rose from 0.8 per query to 2.6 per query. Recall going up and wrong-version retrieval going up are the same event seen twice: the generated passage supplies detail, some of the detail is vocabulary the corpus shares, and some of it is specifics the corpus discriminates on.

The end-to-end number showed almost none of this because it was an average over an eval where most questions do not depend on a version. Splitting the 300:

Seven points of gain on 256 questions and twenty points of loss on 44 nets out to plus three, and plus three is what the pull request said. The eval had the information in it the whole time and no one had a reason to slice it, because there was no hypothesis to slice on until support engineers supplied one.

Check that table both ways before drawing anything from it. Weighting the two slice accuracies by their sizes reproduces the overall baseline figure, and doing the same to the two treatment accuracies reproduces the overall treatment figure. Both land on the printed value with nothing left over. That is the two-slice model closing, and what it licenses is a precise statement about what went wrong: nothing was mismeasured. The aggregate was a correct average of two movements in opposite directions, so the defect was in the reporting granularity and not in the instrument. No amount of re-running the eval would have surfaced it.

Read the rows as gaps rather than as levels and there is a second finding, which is the one that explains the mechanism. The version-sensitive slice did not start out as the hard slice — it started seven points above the insensitive one and finished twenty points below it. Questions that depend on a version are usually questions that state a version, and a stated version is the single strongest discriminating token available in a corpus with five near-identical copies of every page. Those questions were easy for exactly the reason that made them fragile. Replacing the query with a hypothetical overwrites the user's version string with the model's, so the intervention does its most damage where the original query was doing its best work.

Temperature 0 removes the variance, not the invention#

The proposal after the incident is to set the hypothetical's temperature to 0 so it stops making things up. Temperature is the decoding setting that controls how much randomness there is in the choice of each next token, with higher values letting the model pick words it rates as less likely and 0 pinning it to the single most likely word every time. Determinism and grounding are different properties, and this changes only the first.

At temperature 0.7 the model samples; across five runs of the same question it will say release 3.1 twice, release 4.0 once, and omit the version claim twice. At temperature 0 it emits the single highest-probability continuation, and it still has to complete the sentence it started — the prompt asked for a passage of documentation and a passage of documentation states versions. What it produces is the modal invention, which is whichever release dominated the material the model saw. It will say release 3.1 every time.

The effect on the metrics is small and the effect on the failure is large. Wrong-version citations fall from 11% of version-sensitive questions to 9%, and the 9% becomes a fixed set: the same questions, failing the same way, on every run, forever. Nothing is intermittent any more, which sounds like an improvement and is what makes the failure permanent rather than self-correcting. A user who re-asks gets the identical wrong page, and any retry, resample or self-consistency check on top of the pipeline returns the same answer with the same confidence.

That last clause is the belief this section exists to break, and it is worth stating as a general property rather than as a fact about this system. Resampling defends against errors that vary between samples, by looking for the answer the samples agree on. Under a deterministic decode the samples cannot disagree, so the defence returns the same verdict for a question the model got right and a question the model invented an answer to. Its output is constant while the thing it must discriminate varies, and a check with that shape reports confidence rather than measuring it. Determinism did not make the pipeline more trustworthy; it removed the only signal that any trust check downstream was reading.

Temperature 0 also destroys the mitigation that does work. If you generate several hypotheticals and fuse their retrieval runs, the idiosyncratic inventions differ between samples — one says 3.1, one says 4.0, one names a config key the others do not — while the shared vocabulary is the same in all of them. Fusion promotes what the samples agree on and demotes what one of them made up alone, which is precisely the separation you want. At temperature 0 every sample is identical, so fusing them costs five times as much and changes nothing.

def hyde_fused(question: str, index, n: int = 5, k: int = 10) -> list[Chunk]:
    docs = [llm(HYDE_GENERIC, question, temperature=0.8, max_tokens=180)
            for _ in range(n)]
    runs = [index.search_by_text(d, k=k) for d in docs]
    runs.append(index.search(question, k=k))     # keep the original query
    return rrf(runs, top=k)                      # fusion from chapter 5

Two details in that function matter more than the fan-out. Including the original query as one of the fused runs anchors the result to what the user actually asked, which limits how far an entirely off-base hypothetical can drag the pool. And HYDE_GENERIC is a modified prompt that forbids the specifics:

HYDE_GENERIC = """Write a short passage from the product documentation
that answers this question. Write it as documentation, not as a reply.

Do not state version numbers, release names, dates, exact command
flags, or configuration key names. Describe the procedure and the
concepts in general terms. Use the terminology the documentation would
use."""

That instruction is a deliberate trade: the passage keeps the part that raises recall and gives up the part that steers. Compliance is imperfect, in the region of 85% on this corpus, so it reduces the problem rather than removing it.

The last row is the only one that solves the reported problem rather than reducing it, and it is not a query transformation at all. If the question is version-sensitive and the asking customer's version is known from the session, the retrieval call carries a hard filter on the version field and no invented string can select a passage the filter excludes. An invented specific can only steer retrieval along a dimension where retrieval is soft; make that dimension a filter and the invention becomes inert.

Separating the requirement from the design decisions makes that row easier to argue for. The requirement is that the embedded text should use the corpus's vocabulary without acquiring particulars the system cannot vouch for. Generating a whole passage is a design decision, and a generous one — it satisfies the first half of the requirement by satisfying much more than it. Forbidding specifics in the prompt is a second, and it addresses the second half with a request rather than a guarantee, which is why its compliance rate has to be quoted. Fusing several samples is a third, and it converts the guarantee into a statistical one. The metadata filter is a fourth, and it is the only one that removes the failure mode from the space of possible outcomes rather than making it less likely, because it acts on the retrieval call instead of on the text.

What to have measured before shipping#

Recall@10 and aggregate accuracy were both measured and both moved in the right direction, so the missing measurement is not more of the same. Two things would have caught this.

The first is a metadata-agreement metric over retrieved passages: for each retrieved chunk, does its product, version and date metadata match what the question implies? It needs no relevance labels, only the metadata already on the chunks, and it is the direct measurement of the failure — 0.8 wrong-version passages per query rising to 2.6 is a threefold regression sitting in plain sight two weeks before anyone filed a ticket.

The second is slicing the eval on the dimensions the corpus discriminates on. A corpus with five versioned copies of the same page has version as a discriminating dimension by construction, and any change to how queries are formed should be reported per-slice on it. The 44 version-sensitive questions existed in the eval before the change; nothing had to be collected.

Reading the generated hypotheticals is the third thing, and it is not a metric so much as a habit. Twenty of them, on questions from the eval, takes fifteen minutes, and the In release 3.1 and later clause is visible in the first three.

One residual is worth naming rather than smoothing over, because it bounds how much of the story these measurements tell. The share of version-sensitive questions carrying a wrong-version citation is smaller than the share that lost accuracy, so wrong-version citation is a lower bound on the damage rather than a full account of it. Some of the lost questions were answered from a wrong-version passage that was never cited, and some were lost because the correct passage was pushed out of the window by the near-duplicates. The metadata-agreement metric sees all of those, which is a further reason to prefer it over counting citations.

Where a hypothetical answer is the wrong tool#

HyDE closes a gap between question form and document form. When the corpus is already question-shaped, that gap does not exist and closing it makes things worse. On a corpus of 400k support tickets, where each ticket opens with a customer describing a problem in their own words, the user's raw question is already the same kind of object as the target document, and question-to-question similarity is the strongest signal available. Replacing the query with an answer-shaped passage of documentation moves the vector away from the tickets and toward the resolution notes, and on that corpus HyDE measured 0.03 below the raw query while costing 380 ms.

The size of the gain also depends on how well the corpus was written, which is easy to miss when reading the original results. HyDE is bridging a gap between how users ask and how documents state, and that gap is widest where the documentation is written for people who already know the vocabulary — internal runbooks, engineering design notes, anything with a house dialect. Where the documentation opens each page with a plainly worded statement of the problem it solves, the gap is already half closed by the author, and the measured HyDE gain on such corpora is a few points rather than thirteen. The same is true as retrieval embedders improve at asymmetric question-to-passage matching, which is what they are increasingly trained for; a technique that compensates for an encoder's weakness pays less as the encoder gets stronger.

Cost is the other input to the decision and it is dominated by one line. A 180-token generation per query is more expensive than the entire retrieval path, and the fused variant multiplies it by five. On a service handling 40,000 queries a day, five hypotheticals per query is 36 million generated tokens a day whose only purpose is to be embedded and discarded. That figure is the product of three numbers already on the table — the daily query volume, the number of samples, and the generation length — and it closes exactly, which is worth knowing because each of the three is a lever somebody will reach for. That is affordable and it is not incidental, and it is the reason the fused version belongs on a routed slice rather than on all traffic.

It also does nothing for the compound query. Asking for a hypothetical answer to which regions are still on the old rotation policy and what does migrating each one cost produces one passage that gestures at both halves and elaborates the half with more room to elaborate, and its vector has the same problem the original query had. The same is true of the five-variant fan-out from chapter 5, for the same structural reason.

Both techniques take a single intent and improve how it is expressed, and neither has any mechanism for noticing that there are two intents present, still less for retrieving one of them and then using the result to construct the other.