A follow-up turn must be rewritten into a standalone query

A follow-up turn is a message that only makes sense after the one before it. "Does that one cover weekends?" is a complete English sentence and an incomplete question, because the thing being asked about is named nowhere in it. A retriever is handed a string and nothing else, so a follow-up turn embedded as written carries no signal about its own subject, and the passages it matches are whatever else in the index happens to be about the words that survived.

What is needed instead is a standalone query: one that would retrieve correctly if it arrived cold, with no conversation attached. Producing one is a different operation from the shortening rewrite of chapter 3. That rewrite had a complete question in front of it and had to decide what to remove. This one has an incomplete question in front of it and has to decide what to add, from a source outside the query, and adding is the harder direction because there is more than one thing it could add and nothing in the query says which.

The rewrite that puts the subject back#

Three turns from a support assistant on the Cloud Connect docs make the gap concrete.

user  what escalation options do we have on cloud connect
asst  Two tiers. Standard escalation routes to the on-call queue with a
      4-hour response target. Enterprise escalation adds a named
      responder and a 1-hour target.
user  does that one cover weekends?

Retrieval for the third turn returned ten chunks: weekend on-call rotation for the data platform, a support-hours table for the billing product, an SRE handbook page on weekend paging, the holiday coverage policy, and six more of the same shape. Every one is about weekend coverage, none is about Cloud Connect escalation tiers, and the generator produced a confident paragraph about weekend on-call that was true of a different product.

The query that was embedded has two content words, cover and weekends. The word "enterprise" is not in it, "escalation" is not in it, "Cloud Connect" is not in it. Nothing in that vector distinguishes the passage the user needs from a hundred others, so retrieval returned the hundred sorted by how much they are about weekends. This is not a ranking problem or a vocabulary problem; the query text does not contain its own subject.

Binding a referent is, as an analogy, resolving a relative path against a working directory: the same string names different files depending on where you are standing, and the string alone cannot tell you which. The analogy is worth stating because it makes the rewriter's job legible, and it is worth breaking immediately, because the break is where this chapter's real subject lives. A filesystem given a relative path that resolves to nothing raises an error. A query bound to the wrong entity resolves to a document that exists, is well written and is genuinely about what the query now says, so nothing anywhere raises anything.

STANDALONE = """Rewrite the user's latest message as a standalone search query.

Replace pronouns and demonstratives ("that one", "it", "the second") with
the entity they refer to, taken from the conversation. Carry forward the
product, version and other scoping details that the user is still talking
about. If the latest message introduces a new subject, do not carry
anything forward.

Return JSON: {"query": str, "resolved": {referent: antecedent},
"ambiguous": bool}"""

For the transcript above it returns:

{
  "query": "Does Enterprise escalation on Cloud Connect cover weekends?",
  "resolved": {"that one": "Enterprise escalation tier"},
  "ambiguous": false
}

That query retrieves the Cloud Connect escalation policy page at rank 1, because the page contains all four of the terms that now appear in the query. The whole intervention is one generation call of about 60 output tokens.

Why rank 1 rather than merely better is worth a step, because it is the mechanism the rest of the chapter trades against. The rewritten query names a product, a tier, an operation and a time scope. A page that is about exactly that combination contains all four, and no other page in the index contains all four, since every other weekend-coverage page belongs to a different product and every other Cloud Connect page is about something other than weekends. So the correct page is the only document with a high value on every axis the query cares about, while its competitors each score on one axis and not the others. Adding a term does not nudge a ranking; it removes from contention every document that lacks the term.

The history buffer feeds the rewriter and nothing else. It does not go to the index, and it goes to the generator only as dialogue context. Keeping those paths separate is what stops the next problem, which is that the amount of history the rewriter reads is a real parameter with a real optimum.

How much of the conversation to read#

The temptation is to pass the whole conversation, on the grounds that more context cannot hurt. It can, and the shape of the harm is specific: the model carries forward a topic the user has abandoned. Measured over 180 labelled follow-up turns from real sessions, where the label is whether the rewrite named the correct entity, the window is a curve rather than a monotone improvement. The last column is added p50, the extra latency at the median request, which is the honest way to price a step that runs on every follow-up rather than occasionally.

Correct binding climbs from one turn to three because the antecedent is often two turns back rather than one, particularly when the assistant's answer was long and the user's acknowledgement sat between. It falls again past three turns for the reason in the over-binding column: with a whole session in view, a model asked to make a query standalone will attach scope from wherever it can find it. A user who spent eight turns on Cloud Connect and then asks "what's the SLA on status page incidents?" gets rewritten to "what's the SLA on status page incidents for Cloud Connect Enterprise", which retrieves the wrong page with total confidence.

The two middle columns are worth reading as a pair rather than one at a time, because the gap between how fast they move says something neither says alone. Going from the three-turn window to the whole session, over-binding rises by twice as much as correct binding falls. If every over-binding destroyed a correct binding, the two would have to move by the same amount in opposite directions. They do not, so about half the extra over-bindings left the correct entity named anyway — the rewrite carried forward the right entity and some abandoned scope alongside it. That rules out the tidy model in which the two columns are complements of each other, and it is a conclusion available from the table alone, without inspecting a single rewrite.

It also settles a question about instrumentation that would otherwise be a matter of taste. Because the columns move independently, no single number derived from them carries both. An end-to-end accuracy figure sums a correct binding that came with baggage and a correct binding that did not into the same bucket, so a window change that trades one for the other shows up as no change at all. The only way to see the trade is to measure the two properties separately, and the table is what makes that a demonstrated requirement rather than a preference.

Over-binding is the failure the instruction in the prompt is aimed at, and prompt wording alone gets it from 0.19 to about 0.11 at the same window. Shrinking the window is what gets it to 0.05, because the model cannot carry forward what it cannot see. That pair of figures is easy to misread: the second one matches a cell in the table, and it is not that cell. The table's 0.11 belongs to a six-turn window with the original wording, while this 0.11 belongs to a whole-session window with better wording — two different configurations that happen to land on the same rate. Reading the second as the first would suggest that prompt wording and window size are the same lever, when the point being made is that they are different ones with comparable effects.

The rolling summary line is the version most systems end up at: a 40-token running description of the session's subject, refreshed every few turns, plus the two most recent turns verbatim, which preserves the antecedents that matter without keeping eight turns of abandoned topics live.

That line also contains a small anomaly worth resolving rather than skipping, because resolving it tells you where its cost sits. It feeds the rewriter fewer tokens than the three-turn window and yet costs more at the median. Fewer input tokens cannot take longer in the same call, so the extra time is not in the rewriter call at all — it is the summary refresh, which is a second generation step that runs every few turns and whose cost lands on whichever request happens to trigger it. The measured median is therefore the rewrite plus an amortised share of a separate call, and anyone reading the column as the price of one model call will underestimate what the row costs at the tail.

Binding to the wrong antecedent is worse than not binding at all#

The unrewritten query fails loudly. Ten weekend-coverage passages about four products produce an answer that a support engineer reads once and reports, because it is visibly about the wrong thing.

A mis-bound rewrite fails silently. Suppose the rewriter binds that one to the Standard tier rather than Enterprise — a plausible error, since Standard is mentioned first and the sentence describing it is shorter. The query becomes "Does Standard escalation on Cloud Connect cover weekends?", the index returns the Cloud Connect escalation policy page at rank 1, the generator reads the correct document, and it answers accurately about weekend coverage under Standard escalation. Every stage after the rewrite behaves perfectly. Retrieval metrics are excellent, the citation is real, the answer is internally consistent, and it is about the tier the user was not asking about.

There is no downstream check that catches this. Reranking cannot, because the passage is genuinely relevant to the query it was given. Groundedness or citation-faithfulness checks cannot, because the answer is faithful to the retrieved passage. Confidence estimates cannot, because the model has good evidence for what it said. The error was committed before retrieval and everything after it is a correct computation on a wrong input, which is why the mitigations have to sit at the rewrite itself.

The general form of that failure is worth naming, because it is the shape most quality checks in a retrieval system have. A groundedness check asks whether the answer follows from the retrieved text. Its input is the answer and the passage, and both of those are identical in the correct case and the mis-bound case — same page, same paragraph, same faithful summary of it. A check whose inputs do not differ between the two situations it is supposed to separate returns the same verdict in both, and that is a property of what it can see rather than of how strict it is. Tightening the threshold does not help. The information that would distinguish the two cases is the user's intent, and that left the system one stage earlier.

Three mitigations are worth building. The first is surfacing the binding: the resolved map in the JSON output goes into the interface as a line reading Answering about: Enterprise escalation, Cloud Connect, so the one person who can detect the error sees it before reading the answer. The second is refusing to guess when the guess is a coin flip.

def resolve(turn: str, history: list[Turn]) -> Resolution:
    out = llm_json(STANDALONE, format_history(history, window=3), turn)
    if out["ambiguous"]:
        return Resolution(ask="Do you mean Standard or Enterprise escalation?")
    if not out["resolved"] and has_referent(turn):
        return Resolution(query=turn)     # nothing bound; send it raw
    return Resolution(query=out["query"], bindings=out["resolved"])

The third is measuring the binding directly rather than through end-to-end accuracy. Two hundred follow-up turns with a hand-written correct entity each is a day of work and gives you the two columns in the table above, which move independently and get averaged into invisibility by any downstream metric. The label is cheap to produce because it is not a judgement about answer quality — it is the name of the thing the user meant, which a reader of the transcript can supply in about fifteen seconds.

Separating what is required from what was chosen keeps those three arguable. The requirement is that the query reaching the index must name its own subject, and that a wrong subject must be visible to somebody. Everything else is a decision. Surfacing the binding in the interface is one, and a system with no interface would need a different answer. Asking a clarifying question on ambiguity is another, and it trades a round trip for a guess. Falling back to the raw turn when nothing was bound is a third, and it is the conservative choice: the unrewritten query fails loudly, and a loud failure is the better of two failures here.

Not every turn needs the rewriter. On this traffic, 68% of turns are first turns or self-contained questions with no referent, and running a generation call on them buys nothing while adding 240 ms to the majority of requests. The unbound-referent check from chapter 1 is what gates it, and its false-positive behaviour is benign in a way that matters here: sending a self-contained query to the rewriter costs a call and usually returns the query nearly unchanged, whereas failing to send a follow-up costs the whole answer. Gating on a cheap and slightly over-eager detector is the right asymmetry.

Why concatenating the conversation into the query does not work#

The cheaper-looking alternative is to skip the model call and embed the last few turns together with the new question, on the theory that the missing terms are in the history and the history is now in the vector. It is one line of code and it measures worse than the raw query on this set: recall@10 of 0.58, against 0.61 for sending the follow-up alone and 0.81 for the rewrite.

The reason is that a query vector is a summary of everything in the string, and the assistant's previous answer is much longer than the user's four-word follow-up. Concatenating a 60-word answer about two escalation tiers with does that one cover weekends? produces a vector dominated by the escalation-tier description, so retrieval returns the escalation overview page the assistant already used and not the coverage detail the user is now asking for. The history contains the missing entity and also contains eight sentences of material the user has stopped caring about, and embedding cannot tell which is which. A rewrite works because it is a selection step: one entity comes forward, everything else stays behind.

Note what the ordering of those three figures rules out. Concatenation lands below the raw follow-up, not between it and the rewrite, so adding the history is worse than having no history at all. If the history were merely a diluted version of the right signal, more of it would help a little and the concatenated arm would sit above the raw one. It sits below, which means the history is not weak signal but competing signal, pulling the query toward a page the conversation has already used.

This is a formulation with a research history worth knowing about if you want stronger baselines — conversational query rewriting is a standard task with public datasets of human-written standalone rewrites, and the earlier framing of it as classical coreference resolution, the linguistics task of deciding which earlier noun phrase a pronoun points back at, is what most naive implementations reinvent. The classical framing is also why those implementations underperform. A pronoun-resolution model handles that one and it, and a large share of real follow-ups have no pronoun at all: "and for weekends?", "what about the enterprise one", "same question for Frankfurt". Those are ellipsis and bridging, where the missing material is a whole clause rather than a noun, and the repair is not substitution of a word but reconstruction of a question.

The limit of a single phrasing#

Both rewriting chapters produce exactly one query and hand it to exactly one retrieval call. Chapter 3's version removes tokens under a preservation contract; this one adds the entity the conversation established. Each is a bet that a single correct phrasing exists and that a model can find it.

For the vocabulary bucket, the largest of the four in chapter 2's partition, that bet is much weaker. There may be three or four ways the corpus expresses the concept the user asked about, no way to know from the query text which one the relevant document used, and a rewriter that commits to one phrasing commits to being wrong when it picks the other.

The response is to stop producing one query, which changes the retrieval call, the merge step, and — in a way that is easy to get wrong and easy to report favourably — the meaning of every recall number you measure afterwards.