A correct rewrite with wrong passages is a ranking failure

A conversational assistant answering from retrieved documents runs four stages between a user's message and the reply, and each stage leaves an artefact behind. Chat state assembles the messages the request will use. Contextualization rewrites the latest message into a query that stands on its own. Retrieval turns that query into a set of passages. Generation writes an answer from those passages and cites them. A report that the answer was wrong indicts all four at once, and diagnosis is the work of deciding which one to spend the week on.

That decision is worth making carefully because a wrong answer arriving inside a conversation invites a conversational explanation. The belief underneath is that a symptom observed in a conversational context is evidence about the conversational machinery, and it is tempting because that machinery is the newest part of the system and the only part whose output a person can read. It is wrong because the conversation is equally present in the world where the assistant answers correctly, so its presence discriminates nothing.

Four stages, and the artefact each one leaves#

One report of this kind said the assistant had answered a Dublin question with Frankfurt's numbers, and the trace attached to it was complete for once.

[state]    messages=4  (user, asst, user)  window=8  no truncation
           t1 user "Which regions support customer-managed keys?"
           t2 asst "Frankfurt and Dublin both support customer-managed keys."
           t3 user "what's the uptime SLA on the second one?"

[rewrite]  query   = "What is the uptime SLA for the Dublin region?"
           resolved= [{"surface":"the second one","antecedent":"Dublin","turn":2}]
           dropped = ["customer-managed keys"]
           latency = 287 ms

[retrieve] top_k=5
           1  0.681  docs/sla/regions#eu-central-1   "Frankfurt — monthly uptime"
           2  0.679  docs/sla/regions#eu-central-1a  "Frankfurt — exclusions"
           3  0.678  faq/sla#0003                    "Frankfurt SLA credits"
           4  0.677  docs/sla/regions#eu-west-1      "Dublin — monthly uptime"
           5  0.671  docs/sla/regions#eu-central-1b  "Frankfurt — measurement"

[answer]   "The Frankfurt region is covered by a 99.95% monthly uptime SLA,
            measured over calendar months. [docs/sla/regions#eu-central-1]"

Three of the four stages clear on the evidence in front of you. State holds both prior turns in the right order with no truncation, so nothing the rewriter needed was withheld. The rewrite resolves the ordinal to the right region, keeps the uptime-SLA intent, and drops the spent key-management qualifier, which is precisely the target chapter 2 defined. Generation used what it was handed and cited it accurately; an answer about Frankfurt is the correct answer to a passage set consisting of Frankfurt pages, and the generator had no basis for suspecting the passages were wrong.

Each of those verdicts rests on a different field, and each field settles less than it appears to. The [state] block reports how many messages were assembled and whether any were dropped, so it can show that the window did not cut the antecedent away and it cannot show that the messages are this user's; here the turns it prints are the turns in the report, so that question does not arise. The resolved array records which surface expression was mapped to which antecedent and in which turn, which is what makes the rewrite checkable without re-reading the transcript.

Generation clears by an argument with a condition in it, and the condition is what makes the verdict trustworthy. Given a passage set in which every document describes Frankfurt, an answer about Frankfurt is faithful, and faithfulness is the only property a generator can be held to; no channel exists through which it could learn the passages are about the wrong region. The condition is that the answer agrees with the passages, and an answer that disagrees with them is the one verdict no upstream artefact explains — the case the next chapter takes, alongside a fault in chat state.

Two instruments that share no step#

The decisive test for the rewrite takes thirty seconds and is worth running rather than reasoning about. Take the string What is the uptime SLA for the Dublin region?, send it through retrieval with no conversation attached, and see what comes back. It returns the same five Frankfurt-heavy passages. That result separates the two hypotheses cleanly: if the standalone query had retrieved the Dublin page at rank 1, something about the conversational path was corrupting the query between the rewriter and the index, and since it does not, the query is fine and the index is ranking it this way for reasons that have nothing to do with the conversation.

The strength of that argument comes from what the two runs have in common and what they do not. The conversational run assembles state, calls the rewriter, embeds the result and searches. The standalone run takes a string somebody typed, embeds it and searches. They share the embedding model, the index and the documents in it, and above embed they share no step at all, because a string typed into a box has no chat state and no rewriter behind it. Two procedures with no upstream step in common landed on the same five identifiers, in the same order, with the same scores.

What that agreement licenses is narrow and worth stating exactly. The contribution of every stage the two runs do not share is measured by the difference between their outputs, and the difference is nothing. This is not a claim that state and contextualization were inspected and found correct; the standalone run is blind to both by construction, and that blindness is what makes it useful, since it runs the conversational stages at zero and shows what the index does without them. Read the other way the test is weaker, since disagreement would license only "something between the transcript and the index changed the outcome" without naming which stage.

There is one thing neither run can settle, and it is the thing a reader is most likely to assume it settled. Both runs are given the same string, so a rewrite that resolved the ordinal to Frankfurt would produce Frankfurt passages in the conversational run and the identical Frankfurt passages in the standalone run, and the replay would report agreement while the fault sat squarely in the rewriter. The replay holds the query fixed, and no test that starts from the query can see a wrong query.

That gap is what separates the two causes this chapter is about. A rewriting failure means the string reaching the index is not the question that was asked, so the index answers a different question correctly; a ranking failure means the string is the question and the index fails to put the answer at the top of what it returns. Downstream the two are indistinguishable, since both produce plausible passages, a faithful answer, a citation to a real document, and no error anywhere in the request. Separating them takes two checks, and this trace carries both: the resolved array names the second one as Dublin from the turn where Dublin appears, which settles by reading that the rewrite is the question asked, and the replay rules out corruption of that query on its way to the index. A reader who remembers only the replay will confidently indict ranking on traces where the rewrite is the whole problem.

What the score column is telling you#

The five scores span 0.671 to 0.681, ten thousandths from top to bottom, with the correct passage sitting fourth inside that band. This is not the flat neighbourhood from chapter 1, where a contentless query had no topical centre; every one of these passages is genuinely about regional uptime SLAs, which is exactly what was asked for. The query landed in the right neighbourhood and then could not distinguish the members of it.

The reason is visible in the documents. The regional SLA pages are generated from one template, so docs/sla/regions#eu-central-1 and docs/sla/regions#eu-west-1 differ by a proper noun, a region code and two numbers across about 180 words of identical legal prose. Cosine similarity between those two chunk vectors is 0.971. A dense encoder is trained to place text near other text about the same subject, and by that objective these two chunks are about the same subject; the token that distinguishes them is a low-frequency proper noun contributing a few thousandths of angular separation, while the boilerplate contributes almost everything. The 0.004 that separates rank 1 from rank 4 is then decided by incidental things — the Frankfurt chunk is 40 words longer, its heading repeats "monthly uptime", and the FAQ page at rank 3 mentions SLA credits, which the query does not ask about and which happens to sit close to SLA language.

Those two figures measure different things, and conflating them is the easiest mistake available here. The 0.971 is a property of the document pair: the two chunks compared against each other, with no query involved, saying the encoder places them almost on top of one another. The 0.004 is a property of the ranking: the difference between each chunk's similarity to this query. A pair sitting about three hundredths apart can still be separated by only a few thousandths on a given query, and wherever "the cosine gap between the two template chunks" appears below on a figure of that smaller size, it is the query-side quantity that is meant.

The step from "the boilerplate contributes almost everything" to "incidental things decide it" is an assertion so far, and a short piece of algebra settles every case of this kind rather than this one. Scoring is a dot product between the query vector and a document vector, and with vectors normalised to unit length as cosine scoring does, the gap between two documents' scores is the query's dot product with the difference of their two vectors. Whatever the two documents hold in common subtracts away exactly in that difference, however much of the text it accounts for, so the ordering between two near-duplicates is decided entirely by the query's overlap with what distinguishes them. The shared legal prose does not merely count for less; it cancels, and it cancels whatever the query is.

What survives in that difference is short, which is what the 0.971 reports, and it is not made only of what you would hope. The region token is in there, and so is every incidental way the two chunks differ — one being 40 words longer, one repeating a phrase in its heading — and a query about monthly uptime SLAs overlaps a repeated "monthly uptime" heading more than it overlaps a rare place name. Reading a piece of prose as a direction in a vector is an approximation rather than a fact about encoders, since embedding models are not additive and no component is labelled with the words that produced it. The cancellation is exact algebra; what dominates the remainder is an inference, and the header experiment below is what puts it to a test.

The stumble this section exists to prevent is reading 0.681 as a confident result. A score of that size means the retrieved text is about the subject of the query, which it is, and carries nothing about whether the right member of that subject was chosen. What decides an answer is the margin between the top passages rather than their level, so a dashboard reporting mean similarity is reporting the quantity that stays high in exactly this failure.

Region names are the canonical case and not the only one. Version numbers, plan tiers, model identifiers, error codes and customer names all share the property of being rare tokens that carry the entire meaning of a query while contributing almost nothing to a dense vector. Any corpus built from templates has this problem, and documentation corpora are mostly built from templates.

The two fixes that address the wrong stage#

The first proposal that usually arrives is to add a line to the rewrite prompt telling the model to pay close attention to the region. The rewrite already names the region correctly, so the best available outcome of that change is that nothing happens; the realistic outcome is that the prompt grows a clause that fires on every turn, including the turns where no region is involved, and the rewriter starts inserting regions into queries that had none. It also costs a prompt revision, a re-run of the follow-up set, and a deployment, spent on the one stage the trace exonerated.

The second proposal is to enrich the rewrite with more conversation context, producing something like uptime SLA for the Dublin region, which supports customer-managed encryption keys, on the theory that a retriever given more to match on will do better. Chapter 2 measured that exact string: it moves the gold passage from rank 1 to rank 8 by dragging the query into the key-management neighbourhood. Applied here it would take a query that ranks the gold at 4 and rank it lower, while producing a rewrite that reads as more thorough in the log. Both proposals share a structure worth recognising, which is that they treat a conversational symptom as evidence of a conversational cause, when the trace has already localised the fault to a stage where the conversation plays no part.

The two are refuted by different kinds of evidence, and the difference is worth keeping visible. The first falls to an argument: the rewrite already contains the region, so no state of the world exists in which adding emphasis to the instruction improves the string that reaches the index. The second falls to a measurement taken on that exact sentence, which is rarer and stronger. More context helps a query missing its entity and harms a query that already has one, because a query is one point in the space and the added words are a second subject pulling that point toward themselves.

Four retrieval-side repairs, priced#

The latency column spans two orders of magnitude while the second column barely varies, and reading the table down that column picks the wrong row. What separates these four is where the cost lands rather than how large it is: the first three are paid on every query forever, and the fourth is paid once, at ingest, by a team that is not the team getting the bug reports. The bottom row costs nothing at query time and requires re-embedding an entire corpus, which is why it is both the cheapest durable repair here and the one that is routinely skipped.

Hybrid retrieval works here because BM25 treats Dublin as a rare term with a high IDF weight, so the one document that contains it three times outranks the near-duplicate that contains it zero times, and fusing that ranking with the dense one lifts eu-west-1 to the top.

That works by inverting the problem, and naming the inversion is what lets you predict where hybrid retrieval helps. Inverse document frequency weights a term by how few documents contain it, so a phrase appearing on every page of a template scores near zero lexically. The shared boilerplate that dominates a dense vector is exactly the text a lexical scorer discounts to nothing, and the rare proper noun a dense vector nearly ignores is exactly the term it weights most. The two methods disagree about which part of a document carries its meaning, which is why fusing them recovers this case and why it costs something on a paraphrased query, where the user's words appear nowhere in the correct document and the lexical half contributes noise.

dense  = index.search(embed(q), top_k=50)
sparse = bm25.search(q, top_k=50)
fused  = reciprocal_rank_fusion([dense, sparse], k=60)[:5]
# rank of docs/sla/regions#eu-west-1 : dense 4 → fused 1

Fusing by rank rather than by score is what makes that safe to write, since a cosine similarity and a BM25 score are numbers on unrelated scales and combining them directly would mean calibrating one against the other and recalibrating whenever either side moved.

The metadata filter is the sharpest instrument and the most brittle. Because the rewriter already returns JSON, it can return the region as a field rather than leaving it to be parsed back out of the query string, which removes an entire class of regex failures.

rw = contextualize(history, latest)      # {"query": ..., "filters": {"region": "eu-west-1"}}
flt = {"region": rw.filters["region"]} if rw.filters.get("region") else None
hits = index.search(embed(rw.query), top_k=5, filter=flt)

The conditional is load-bearing. A filter that defaults to something when no region was identified turns a ranking problem into an empty result set, and the failure mode of an over-eager filter — zero passages, or passages from one arbitrary region — is worse than the ranking failure it replaced. Filters also need the field to exist on every chunk, which means the fix is only available if ingest wrote it, and retrofitting it is a reindex.

Reranking is the general-purpose answer and the expensive one. A cross-encoder reads the query and the passage together rather than comparing two independently computed vectors, so the mismatch between Dublin in the query and Frankfurt in the passage is directly observable to it, and on the near-duplicate template cases it put the correct region at rank 1 in 84% of trials. The 180 ms is real and lands on top of the 320 ms already spent on the rewrite, which is how a conversational assistant ends up with half a second of latency before the first token.

Those two figures account for that half second between them with nothing left over, so a team asked to cut the delay before the first token has exactly two places to look and no mystery third term. The reranker's share cannot be optimised away, for the same reason it is accurate: a passage compared as an independent vector was encoded before the query existed, while a cross-encoder lets every token of the query attend to every token of the passage, and that comparison can be neither precomputed nor reused across queries.

The fourth row is the cheapest durable fix and the one that gets skipped because it lives in the ingest pipeline rather than the query path. Prepending the document title and region to every chunk before embedding — so the eu-west-1 chunk begins Dublin (eu-west-1) — Regional SLA — rather than with the shared legal preamble — raises the cosine gap between the two template chunks from 0.004 to 0.031 and moves the gold to rank 1 on this query with no runtime cost at all. It requires a full reindex and it only helps where the distinguishing token can be lifted into the header, which is why it complements rather than replaces the others.

That measurement is also the test of the inference offered earlier. If the ranking between two template chunks is decided by the query's overlap with their difference, moving the region name into a short, prominent position should enlarge the part of that difference a region-bearing query aligns with, and the gap grows by nearly eightfold while nothing else about either document changes.

Choosing between them comes down to how much of the corpus is templated and how much latency budget the rewrite already consumed. A documentation set with many near-duplicate pages gets most of the available benefit from the header change plus hybrid retrieval, for 15 ms and one reindex, and a reranker on top of that buys a further few points at three times the latency. A corpus where the distinguishing attribute is genuinely structured — region, version, tenant, environment — is better served by the filter, because a filter is exact where a ranking signal is statistical, and the rewriter is already producing the structured field for free.

Where each repair stops working#

The failure column of that table holds four kinds of failure rather than four severities of one. Hybrid retrieval degrades gradually, on queries whose vocabulary the corpus does not share; the reranker fails absolutely outside its pool, since it reorders 100 candidates and can do nothing for a document ranked below them; the filter fails visibly, returning nothing; and the header change fails silently, wherever the distinguishing detail lives in a table cell halfway down a page.

The eval from chapter 3 sees this class of failure only if the harness records where the gold passage landed rather than whether it was in the top k. A stratum where gold sits at rank 4 to 20 on most failing items is a ranking problem and is fixed in the index; a stratum where gold sits beyond rank 1,000 is a query-construction problem and is fixed upstream. Those two distributions are as far apart as measurements get, they are produced by the same failing score, and one extra field in the score record is what separates them.

The two regimes barely overlap, for the same geometric reason as the score band. A query that still names its subject lands among documents about that subject and the gold is one of them, so it comes back a few ranks down however badly that neighbourhood is internally ordered. A query that has lost its subject — the contentless follow-up of chapter 1 is the extreme — lands somewhere else entirely, and its gold is thousands of ranks away because it is not near the query at all rather than being near it and mis-sorted. Rank is therefore a third check, independent of the replay and of reading the resolved array, and it works across a stratum where those two work on one trace.

Reports that arrive on turn three#

The thing this trace establishes is a habit rather than a fix. A wrong answer arriving in a conversational context invites an investigation of the conversational machinery, and a substantial share of those reports are ordinary retrieval failures that happened to be reported by a user who was on their third turn. Had the same user typed What is the uptime SLA for the Dublin region? as their first message, they would have received the same wrong answer, and nobody would have suspected the rewriter.

Running the rewrite standalone is what tells the two apart, and it is cheap enough to be the first thing done on every report of this kind. If the standalone query retrieves correctly, the fault is in how the query was constructed and the earlier chapters apply. If it retrieves the same wrong passages, contextualization is not involved and the work is in the index, the ranking, or the corpus.

A correct rewrite followed by a wrong answer is therefore a verdict rather than a puzzle. It is reached by measuring the conversation's contribution as the difference between a run that has one and a run that does not, and confirmed by reading the rewrite against the transcript so that a wrong query cannot hide inside an agreement between two runs given the same wrong string.

The trace here was unusually cooperative, in that all four stages were recorded and only one was wrong. The two verdicts that this case does not illustrate — a fault in chat state, and a fault in generation — present with different evidence, and one of them produces a rewrite that is wrong for a reason no amount of work on the rewriter will fix.