An unscoped history welds the old entity onto a new question

Query contextualization — the step that reads the recent turns of a conversation and rewrites whatever the user just typed into a search query that stands on its own — has to decide, on every turn, how much of the earlier conversation still belongs in that query. When somebody changes subject mid-conversation, the step gets that decision wrong in one direction. The query it emits carries an entity from the subject the user has left, reads in a log as a careful piece of work, and retrieves documents for a question nobody asked.

A useful name for that surviving entity is a contaminant: a token in the rewrite that came from a stretch of conversation the user has closed. What makes contamination worth a chapter is that the rewriter producing it is not malfunctioning and its output is not detectably odd. It is doing what its instructions say, on an input that contains no fact about which turns are still in scope. Fixing it therefore means changing that input rather than the prompt, and the two obvious ways of changing it trade one failure for another in a way that a single recall number is unable to show.

The turn where the subject changes#

One conversation with a pivot in it, with the rewrite recorded beside the turn that produced it, makes the shape of the failure concrete.

turn 1  user   Which EU regions support customer-managed encryption keys?
turn 2  asst   Frankfurt and Dublin both support CMK for data at rest.
turn 3  user   What's the uptime SLA on the second one?
turn 4  asst   Dublin (eu-west-1) is covered by the 99.95% monthly uptime SLA…
turn 5  user   And the RPO for Dublin?
turn 6  asst   The documented RPO for Dublin is 15 minutes for…
turn 7  user   ok different question — is the billing export API paginated?
        rewrite: "Is the billing export API paginated in the Dublin region?"

The rewriter is following its instructions exactly. It was told to use the conversation history to resolve references and to carry forward qualifiers that still constrain the answer, and it was handed eight turns in which the token Dublin appears six times, three of them in the assistant's own confident prose. Nothing in that input distinguishes an entity the user has finished with from one they are still working on. Recency is the only signal available, and by recency Dublin is extremely present.

That verdict is worth walking rather than accepting, because each link in it is checkable and the last one is the surprising part. The rewriter's input is a list of turns with no annotation on any of them. The only facts it holds about Dublin are that the token is present, that it is recent, and that it occurs often, and both of the properties that could rank one entity above another — recency and frequency — point the same way. No third property in the input points the other way, because the fact that would settle it, that the user has finished with regions, was never written down anywhere a model can read it. Inclusion is therefore the answer the input licenses, and a rewriter that had omitted Dublin here would have been guessing correctly rather than reasoning correctly.

Salience makes it worse in a way that is easy to miss. Every time the assistant answers a question about Dublin it repeats the name, so a conversation of three user turns about a region contains six or eight mentions of it by turn seven, and the entity the user just introduced — the billing export API — has appeared once. A model asked to decide what is relevant in that window is looking at a distribution that says Dublin is what this conversation is about.

The rate is high enough to be a product problem rather than an anomaly. On the items in the follow-up set that follow a topic change — 9% of production turns by the distribution in chapter 3, which builds a 300-item set of labelled follow-ups sampled from real transcripts — retrieval recall was 0.34 against 0.86 overall, and auditing the rewrites showed that 71% of them contained at least one entity from the segment the user had left.

The prompt contributes as well, in a way that is invisible until you read it as an instruction set rather than as a description. Contextualization prompts say what to do with the history and are silent on when to ignore it, so every clause pushes in one direction: resolve references, carry constraints, use the conversation. There is no clause a model can point to that authorises leaving the history alone, and a model resolving an ambiguous case under those instructions includes rather than omits. Adding an explicit counterweight — a line saying that if the latest message introduces a new subject, prior entities must not appear in the query — is worth doing and does not solve it, since on the audited items it moved contamination from 71% to 44%. The model still has to decide whether the subject is new, from a window in which the old subject is the loudest thing present.

The reason that clause gets so far and no further is worth stating as a general one, because it recurs for every fix in this chapter. An instruction can only be executed as well as the evidence available to execute it, and the counterweight adds an instruction while leaving the evidence exactly as it was. That fixes the cases where the pivot is obvious from wording alone and leaves the cases where it is not. The requirement underneath, which the rest of this chapter is measured against, is that the rewriter's input must distinguish the turns still in scope from the turns that are not. Everything below is a design decision about how to produce that distinction, and the first of them produces it by proxy.

One window length, two opposed demands#

The immediate fix is to stop sending eight turns and send two. It is a one-line change, it is easy to justify, and its measured effect is the reason this chapter exists.

Whether any window length can be right for a given turn is settled by two distances, both counted in turns back from the message being rewritten. Write d_live for how far back the antecedent that the current turn needs sits, which is zero when the message stands on its own, and d_stale for how far back the nearest entity the user has finished with sits. A window of w turns shows the rewriter everything within w turns of the present and hides everything beyond it. It reaches the antecedent when w is at least d_live, and it keeps the stale entity out when w is smaller than d_stale.

Both conditions hold together only when d_live is smaller than d_stale, and where that holds, any w between the two values works. Where d_stale is the smaller of the two, no w exists at all: every window long enough to reach the antecedent also reaches the entity that should not be there, and every window short enough to exclude that entity also cuts off the antecedent. This settles the question for every turn rather than for the two columns measured above, and it says something the columns cannot. The two distances vary from turn to turn while w is one number chosen for a whole deployment, so choosing w is choosing which turns it will be wrong on.

The transcript above supplies both cases. At its pivot turn, d_live is zero, since the billing question stands on its own, and d_stale is one, since Dublin sits in the turn immediately above; any window shorter than a single turn is correct there, which is to say the correct window is no history at all. A conversation where somebody works through a list and asks "and the third one?" after two turns about something else has d_live of several turns and d_stale of one, and no window length lies between them. A product that contains both kinds of conversation, which is every product, has no correct value of w.

The bottom row is what ships. A two-point drop on a 300-item eval sits inside the confidence interval established in chapter 3, so the change reads as neutral, and the qualitative complaints stop. What has actually happened is that the failure moved from a category users complain about into a category they do not, because a reference reaching back four turns is normal when someone is working through a list — "and the third one?", "what about the last two?" — and when it fails the assistant answers a question that was never asked without any visible trace of the earlier topic to make the mistake obvious.

The mean moves so little for an arithmetic reason worth working out, since the same arithmetic governs every eval reported as one number. Recall over a whole set is the sum, across strata, of each stratum's recall weighted by that stratum's share of the set, so a movement inside one stratum enters the reported number shrunk by that share. Long-range references are 17% of a set stratified as chapter 3 describes, and their recall falls from 0.79 to 0.31, a fall of forty-eight points; weighted by their share, they pull the mean down by about eight points. Post-pivot follow-ups are 9% of it, and they rise from 0.34 to 0.71, thirty-seven points; weighted, they push the mean up by about three points. The two movements together predict a fall of around five points, and the fall actually reported is two.

That decomposition does not close, and the gap between five points and two is the most useful thing in the table. Several things could produce it and the figures here choose between none of them: the two strata may overlap, since a reference reaching back four turns can also be a reference across a topic change; the rest of the traffic may have moved upward, as a shorter window also removes the milder contamination ordinary follow-ups carry; or the item set's stratum shares may differ from the production shares these weights were taken from. Picking one would be inventing a finding. What the arithmetic establishes without picking is that the reported two points are the residue of larger movements in opposing directions, and that the aggregate conceals more movement than its own size suggests.

Those weights are borrowed rather than measured here, which is worth one clause of justification. The post-pivot share and the long-range share are the same figures chapter 3's reference-pattern table reports from its production sample of logged turns, and the follow-up set was stratified against that table, so the strata in the two chapters name the same populations and their recalls are comparable rather than coincidentally similar. That is what makes the weighting legitimate at all, and it is also why a set built to different quotas would need the arithmetic redone before its mean could be read.

The belief this table exists to break is that the window is a tuning parameter, and that some length is right for a given product with the work being to find it. It is tempting because every readout behaves like a tuning problem: the aggregate barely moves, one complaint category empties, and nothing in the run comes back red. What makes it wrong is that the quantity holding still is not the quantity at risk. Recall on the whole set stays within a couple of points across every window length, while the identity of the turns that fail changes completely, and a measurement that reads the same in both of those worlds is not evidence about which one you are in.

The regression is invisible for a specific reason, which is that the eval that would catch it is the eval that missed the original problem. A follow-up set sampled and stratified as chapter 3 describes contains 17% long-range references, so the drop from 0.79 to 0.31 in that stratum is a real signal available in the per-pattern breakdown, and it is not available in the mean. Teams that built the set and then reported one number get told the change was fine.

Marking scope: segments and live entities#

The window length is a proxy for the thing that actually matters, which is whether a turn is still in scope for the question being asked. Two designs attack that directly, and they fail differently enough that the choice depends on the product. Both supply the rewriter with the fact the raw transcript was missing, and they differ in what kind of object carries it.

Segmentation cuts the conversation into topical spans and shows the rewriter only the current one. The cheapest detector embeds each user turn and compares it with the centroid of the current segment, which is the average of the vectors of the turns already in that segment, starting a new segment when cosine similarity drops below a threshold; a better one asks the rewriter itself to emit a topic_shift boolean alongside the query, which costs no extra call because it rides on the JSON that already exists. A conversation that pivots at turn 7 then hands the rewriter turns 7 onward, and Dublin never enters the prompt.

The centroid version works on an assumption worth making explicit, since it is where the design's failures come from. Turns about one subject embed near each other, so the running average of a segment's turns is a reasonable stand-in for what that segment is about, and a turn that lands far from it is evidence of a change of subject. What segmentation still assumes is that scope is an interval ending at the present, exactly as the window did. It has only replaced a boundary chosen by counting with a boundary chosen from content, so it satisfies the two conditions above whenever the user's scope really is contiguous and it has nothing to offer when it is not.

Entity liveness keeps a working set instead of a span. Each named entity gets a slot recording when it was introduced, when it was last referred to, and what kind of thing it is, and the rewriter receives the live slots plus the last two turns rather than raw history.

That structure dissolves the conflict between the two distances rather than finding a better compromise between them. Liveness is a fact held per entity, so an antecedent's slot stays live however many turns of unrelated material have piled up in front of it, and an abandoned entity's slot goes stale however recently it was mentioned. The demand that d_live be reachable and the demand that d_stale be excluded stop competing for one number, because they are now answered by two independent slots. The state machine above is what makes them independent: every transition is triggered by something that happened to that entity, and none of them is triggered by the passage of the conversation as a whole.

The diagram offers two ways out of Live and they differ in kind rather than in degree, which is the part of the design most worth copying. Going stale after three turns of silence is reversible: the entity stops being shown to the rewriter and comes straight back the moment somebody names it, so being wrong about staleness costs one turn of a missing antecedent that the next mention repairs. Eviction after eight turns is terminal, and so is being superseded, since in both cases the slot is gone and there is nothing left to revive. The grading follows from that: the reversible decision is triggered by weak evidence that costs nothing to collect, which is silence, while the terminal ones need either strong evidence, which is a replacement entity of the same type arriving, or a horizon long enough that silence has stopped being ambiguous.

The typed slot is what makes the Superseded transition possible and is the part that does the real work. When the user asks about the billing export API, an entity of type service enters the conversation; the previous region slot holding Dublin is untouched, which is correct, because a question about pagination has no region in it. When they later ask about the reporting export API, the service slot is overwritten rather than accumulated, so the rewriter never sees two competing services and never produces a query mentioning both.

def build_rewriter_input(state: ConvState, latest: str) -> str:
    live = [e for e in state.entities if e.status == "live"]
    recent = state.turns[-2:]
    return (
        "LIVE ENTITIES:\n"
        + "\n".join(f"- {e.type}: {e.name} (last mentioned turn {e.last_seen})"
                    for e in live)
        + "\n\nRECENT TURNS:\n"
        + "\n".join(f"{t.role}: {t.content}" for t in recent)
        + f"\n\nLATEST: {latest}\n"
        "Use a live entity only if the latest message refers to it."
    )

Populating the slots is a small extraction problem with two workable implementations. Running an entity pass over each user turn adds a second model call and roughly 90 ms, which doubles the contextualization budget for something that is not the query. Asking the rewriter to emit the entities it saw as part of the JSON it already returns costs perhaps 40 extra output tokens and keeps the request path at one call, at the price of the slots being only as good as the rewrite that produced them. The second is what most deployments end up with, and it has a pleasant property: the slot table and the rewrite are produced together, so a contaminated rewrite is accompanied by the slot state that caused it, and the two can be read side by side in the log.

On the same 300 items, segmentation scored 0.88 overall with post-pivot recall at 0.79, and entity liveness scored 0.87 overall with post-pivot recall at 0.83 and long-range recall at 0.74. Both beat every window length, and both cost either nothing (the topic_shift flag rides along) or one cheap extraction pass per turn.

Those figures support less than they appear to. They compare each design against the window lengths on strata where both were measured, and that comparison is clean: no window length reaches either overall score while also holding post-pivot recall up. They do not compare the two designs against each other, because the long-range stratum — the one the window trade was lost on, and the one segmentation's own failure mode lands in — is reported for entity liveness and absent for segmentation. A tenth of a point between two overall scores, with the deciding stratum missing from one column, is not a weak preference for liveness; it is no evidence about the choice at all.

How a scope marker fails#

A segmenter that cuts one turn early strands the reference the user did mean, and the resulting rewrite still reads perfectly well in the log. If the boundary lands between turns 5 and 6 in a conversation where the user's turn 6 is "and the RPO for Dublin?" — the recovery point objective, or how much data a failover is allowed to lose — the rewriter receives a segment that begins with a question mentioning Dublin, which is fine; if the boundary lands between 6 and 7 in a conversation where turn 7 is "what about the RTO?", asking after the recovery time objective that pairs with it, the rewriter receives one turn, has no antecedent for the elided predicate, and emits What is the RTO? — a well-formed query with no entity in it, retrieving generic disaster-recovery pages. Tuned to a similarity threshold of 0.55, the segmenter cut early on 6% of conversations, and those items scored 0.19. The failure is quiet in the same way the short-window failure is quiet, and it is harder to reason about because the boundary depends on the encoder's opinion of two sentences.

That is the same defect as a window that is too short, arriving by a different route. The segment boundary is a value of w decided per conversation instead of per deployment, so a cut placed above the antecedent puts d_live outside the input exactly as a fixed short window does. Deciding it from content rather than from a count improves the average case and removes none of the structure, because the boundary is still a single point on a timeline and scope is still being modelled as everything after it.

Segmentation also assumes topics are contiguous, and they are not. Users interleave — ask about Dublin, ask about the export API, come back to Dublin, then ask about the API again — and a segmenter that starts a new span at every pivot produces four segments where a human would see two threads. Entity liveness handles interleaving naturally, since both slots stay live and the Stale → Live transition costs nothing, which is the main reason to prefer it in products where people compare things.

Entity liveness has its own boundary case, which is the reference to something explicitly closed. "Go back to what you said about Dublin" after eight turns on billing hits an evicted slot, and the rewriter has nothing to resolve against. Extending eviction to session lifetime rather than eight turns fixes those items and slowly reintroduces the contamination the mechanism was built to prevent, so the eviction horizon is a tuning parameter with the same shape as the window length, moved to a place where it does less damage.

The reason it does less damage there is the same reason the design works at all, and it is worth being precise rather than taking the claim on trust. A window length has to be simultaneously right about every entity in the conversation, since one number governs all of them; an eviction horizon has to be right about one entity at a time, and it is applied from that entity's own last mention rather than from the present. A conversation with one long-abandoned region and one recently discussed service gets the correct treatment of both under liveness, and cannot get the correct treatment of both under any window. The horizon is still a guess, and it is a guess made per slot, which is a strictly smaller thing to be wrong about.

Two strata the follow-up set does not contain#

The follow-up set as built scores whether the gold passage came back, and every failure discussed here is a rewrite containing something it should not. Recall alone cannot see that: a contaminated rewrite that still retrieves the right passage passes, and a clean rewrite that misses passes as a retrieval problem. The set needs items whose correct answer includes an absence, and a check that scores the rewrite directly.

{"id": "fu_0211",
 "prefix": ["…6 turns about the Dublin region…"],
 "follow_up": "ok different question — is the billing export API paginated?",
 "gold": ["docs/api/billing-export#pagination"],
 "reference_type": "post_pivot_standalone",
 "forbidden_in_rewrite": ["Dublin", "eu-west-1", "SLA", "encryption"],
 "required_in_rewrite": ["billing export"]}

The forbidden_in_rewrite field is doing something the rest of the item cannot, and the difference is worth holding on to. Every other field asserts that something must be present, and a presence check can be satisfied by accident: the retrieval score is the same whether the correct passage came back for the right reason or in spite of a contaminated query. An absence check fails on the contaminated query regardless of what retrieval then did with it, which is what makes it the only field in the item that can indict a rewrite that got lucky.

Two new strata come with it. Post-pivot standalone items, where the follow-up needs no history at all and the correct behaviour is to leave it alone, catch contamination. Long-range items, where the antecedent is deliberately four or five turns back with unrelated turns in between, catch the window-shrinking regression that otherwise ships as a two-point wobble. Reporting those two strata separately alongside the mean is what makes the trade between them a decision rather than an accident, and it is worth setting the CI gate on the strata rather than the mean for exactly that reason.

Gating on the strata rather than the mean also removes the arithmetic that hid the regression in the first place. A gate on the mean is a gate on a share-weighted sum, so it can be passed by a change that ruins one stratum and slightly helps three others, and the smaller the damaged stratum the more completely it can be ruined without moving the gate. A gate on each stratum has no weights in it and no offsetting, and a change that halves one number has to be argued for on that number.

Every repair in this chapter is an answer to the same requirement: put into the rewriter's input a fact the transcript does not contain, which is whether a turn or an entity is still in scope. A window length answers it by assuming scope ends a fixed distance back, a segmenter by assuming it ends where the subject last changed, and a slot table by tracking each entity separately and assuming nothing about the shape of the conversation. The first is the only one of the three that is guaranteed to be wrong on some turns of any product whose users both change subject and refer backwards, and it is the one that is easiest to ship and hardest to measure.

Scoping the history correctly raises the ceiling on how often the rewrite is right and does nothing about the turns where it already was. Those turns exist in every deployment, they arrive as ordinary bug reports, and the trace that accompanies them clears three of the four stages before you have finished reading it.