A single-turn benchmark makes contextualization a no-op

A retrieval benchmark is a file of questions paired with the passage ids that should come back for each one, run on a schedule and reported as a single recall number. It is the artefact a team points at when somebody asks whether search is working. Contextualization is the separate stage that turns a follow-up such as and for the enterprise plan? into a query that can be searched on its own, by reading the turns above it. Both of those things exist in most conversational assistants, and the benchmark almost always predates the stage.

What a benchmark of that shape can say about that stage is nothing, and the reason is stronger than the usual complaint that a test set is unrepresentative. An unrepresentative test set gives a biased estimate of the quantity you care about, which is still a kind of evidence. This one gives a quantity that is independent of it: the same digits come out whether the stage works, misfires, or was never deployed. That is worth deriving step by step rather than asserting, because the same derivation fixes the sign of the effect the benchmark will report once the stage is wired into it, and that sign is negative.

Where the stage returns its input unchanged#

An item in a benchmark of this kind carries a question and the ids of the passages that answer it, and nothing else. Four hundred of them in one such file all had this shape.

{"id": "q_0187",
 "question": "What is the data retention period for Mongo cluster snapshots?",
 "gold": ["runbook/mongo-retention#0001", "runbook/mongo-retention#0002"]}

The harness reads the question field, embeds it, searches, and compares the returned chunk ids against gold. Follow the chain of what determines the score for one item. Whether an item counts as a hit is fixed by the ranked list of chunk ids that came back. That list is fixed by the query vector and the index it was searched against. The query vector is fixed by the query string and the embedding model. Holding the index and the model still, as a nightly run does, the item's score is a function of one string and nothing else in the system reaches it.

Now put the stage into that chain. With contextualization in the path, the string that arrives at the embedder is no longer question but the stage's output on the pair (history, question). Every item in this file carries an empty history, since an item has no prior turns to load. Every question in it is self-sufficient, which is what makes the set a standalone benchmark in the first place. Rule 4 of the resolution prompt in chapter 2 — if the latest message already stands alone, return it unchanged — therefore fires on every item, and the stage emits the input string character for character. Identical strings give identical vectors, identical vectors give identical ranked lists, and identical ranked lists give identical hits. The equality holds item by item rather than in the aggregate, which is a stronger statement than "the scores come out close".

That invariance is the whole finding, and it is worth spelling out what it covers. Two builds differing only in their contextualizer score the same. A build with the stage deleted scores the same. A build whose stage throws on every non-empty history, and is therefore never exercised here, scores the same. The number is fixed while the property it is supposed to be watching varies across its entire range, which is the precise sense in which this test still passes with the control removed. The repair people reach for first — pointing the harness at the full conversational endpoint instead of calling the retriever directly — leaves every step of that argument standing. The stage now runs. It runs as the identity function.

What a healthy reading does license is worth being equally precise about, because the benchmark is not a bad artefact. A recall@5 of 0.91 is a real measurement of a real thing: given a well-formed standalone query, this index returns a correct passage inside its top five about nine times in ten. That covers the encoder, the chunking, the coverage of the document set and the ranking, and a CI gate at 0.88 will genuinely catch a regression in any of them. What it cannot cover is any failure whose trigger is a non-empty history, and the failures contextualization exists to prevent are exactly that set. The gate is not watching that branch of the code weakly. It is watching a branch that follow-ups never take.

The sign of the effect the benchmark can report#

Rule 4 is an instruction a model is asked to obey rather than a property of the code, and it is obeyed imperfectly. Chapter 2's measurement on a set drawn from production found 12% of already-standalone queries altered anyway, with about a third of those alterations lowering recall by substituting a paraphrase for a phrase that happened to match the indexed wording exactly. Feed those two rates back into the chain above. Items the stage leaves alone contribute exactly zero to the difference between the two builds, since their strings never change. Items the stage rewrites contribute whatever the rewrite does to retrieval, and on a set of questions that already stand alone there is no improvement available, because the question was already the best available form of itself. Those contributions are zero or negative.

The measured contribution of contextualization on this benchmark is therefore bounded above by zero and bounded below by whatever damage over-rewriting does. The expected value follows from the two rates: about one item in eight changes, about a third of those get worse, none of them can get better, so the expectation is a small negative number. That is the shape teams report. Contextualization is switched on, the nightly benchmark ticks down by half a point, and the rational response to that signal, read on its own terms, is to switch the feature off. The signal is not lying about what it measured. It is being read as an answer to a question it was never able to ask.

Read that table across the gaps between rows rather than down its columns. The first two rows differ in what runs and not in what is measured, and the argument above says their scores are equal item for item, so any comparison between them compares a measurement with itself. The only over-rewriting cell is the entire increment the second design buys, and it is a detector for damage rather than for benefit. Only the third row changes the input, and that is the load-bearing difference: it refuses to hand the system a query. The item's job is to make the system construct the query itself, from state, the way it does in production.

The shape of one item#

{"id": "fu_0042",
 "prefix": [
   {"role": "user",
    "content": "Which EU regions support customer-managed encryption keys?"},
   {"role": "assistant",
    "content": "Frankfurt and Dublin both support customer-managed keys (CMK) for data at rest."}
 ],
 "follow_up": "What's the uptime SLA on the second one?",
 "gold": ["docs/sla/regions#eu-west-1"],
 "reference_type": "ordinal",
 "gold_rewrite": "What is the uptime SLA for the Dublin (eu-west-1) region?",
 "source": "conv_9f21c4/turn_3"}

Three fields carry most of the weight. The prefix is a real conversation, loaded into chat state before the item runs, so that whatever windowing, truncation and message-ordering logic exists in production is exercised rather than bypassed. The follow_up is the string the user typed, keystrokes and all, including the missing capital and the trailing question mark, because normalising it into a tidy sentence is the first step towards making the eval easier than reality. The gold_rewrite is optional and is not scored; it exists so that a failing item can be diagnosed without a human re-reading the transcript.

Two of those are requirements and one is a design decision, and keeping them apart makes the set easier to argue about later. The prefix and the untouched follow_up are requirements: drop either and the system is handed a self-sufficient query again, which puts the stage straight back into the identity case the previous section derived. The gold_rewrite is a choice. Nothing in the scoring path reads it, and a set without it would score identically; it is carried because it converts a failing item from a number into a readable comparison, and because the ceiling check further down depends on having a hand-written standalone query for every item.

def score_item(item, k=10):
    session = chat.new_session()
    for msg in item["prefix"]:
        session.append(msg)                       # no model call, state only
    result = session.answer(item["follow_up"])    # the production path
    got = [p.chunk_id for p in result.passages[:k]]
    hit = len(set(got) & set(item["gold"])) > 0
    return {"id": item["id"], "hit": hit, "rewrite": result.rewrite.query,
            "dropped": result.rewrite.dropped, "ranks": rank_of_gold(got, item["gold"])}

The harness records the rewrite alongside the hit, which costs one field and turns a failing run into a readable diff rather than a number that went down.

Where the items come from#

The tempting way to build 300 of these is to take the standalone questions that already exist and convert each into a two-turn conversation, replacing the entity in the question with a pronoun and putting a sentence about it in the prefix. It is fast, it costs nothing, and one afternoon produces a full set. It also produces an eval that scores 0.88 on the same contextualizer that scores 0.55 on production-derived items, which was the measured gap on one system where both sets were built.

The reason for that gap is in how the items differ, and it is systematic rather than a matter of degree. A follow-up written by someone holding the answer is always recoverable, because the writer knew what the reference had to point at when they wrote it. The antecedent lands in the immediately preceding turn, since that is the turn they just wrote. The reference is a clean pronoun or ordinal, since those are the ones that come to mind. The vocabulary matches the indexed documents, since the writer is looking at them.

Both columns close. The five shares in the production column add to exactly one hundred with nothing unaccounted for, and so do the five in the synthesised column. That is worth adding up rather than assuming, because it settles a question the table would otherwise leave open. There is no sixth pattern sitting outside the classification quietly absorbing the turns that did not fit, so every referring turn in the sample fell into one of these five. Two things follow. A quota across the five patterns is a partition of the space rather than a selection from it, so a sampling scheme built on them can be made exhaustive. And any production follow-up failure worth chasing is a failure in one of these five categories, which makes a per-pattern breakdown a complete account rather than a highlights reel.

Now read the two columns as the gaps between them, and in particular as the rate at which each one falls. The synthesised column collapses: about nine items in ten in the top row, a small remainder in the second, almost nothing by the third, and exactly nothing in the bottom two. The production column declines gently and is still carrying about a quarter of its mass across the two rows where the synthesised column has already run out. The support of the synthesised distribution ends two rows before production's does, and because both columns close, that ending is real rather than an artefact of what happened to be listed.

A zero in a column that sums to a hundred is a different kind of number from a small one. A set with 7% of its items in a category supports a wide and weak estimate of performance on that category, which is still an estimate, and more items would narrow it. A set with none supports no estimate at all, and there is no sample size at which that changes, because the quantity being estimated has no observations behind it.

That is what makes the headline comparison misleading rather than merely optimistic. One contextualizer scoring 0.88 on the synthetic set and 0.55 on the production-derived one looks like an easy test beside a hard one. The two sets are drawn from different populations, and one of those populations does not contain the cases the system fails on. The synthetic score is not a generous estimate of the production score. It is an accurate estimate of a different quantity, and the arithmetic that would convert one into the other requires per-category performance on categories the synthetic set contains none of.

Sampling from real transcripts is therefore not a quality improvement on the synthetic approach; it is the only way to obtain items in those categories at all. The pipeline is a filter over logged conversations for non-first turns whose text contains a referring expression with no in-turn antecedent, stratified sampling across the five patterns above, which means filling a fixed quota for each pattern rather than keeping whatever proportions the logs happened to produce, so the rare ones are not swamped, and then human labelling of gold passages by someone who reads the whole transcript and searches the document set themselves. At roughly four minutes per item, 300 items is about twenty hours of annotation, which is two and a half working days of one person, and that number is the real reason teams reach for synthesis. It is a plain product and worth carrying as one: multiply the item count by the minutes an item takes, divide by the minutes in an hour to get the hours, and divide again by the hours in a working day to get the days. Nothing in it is an estimate except the four minutes, so the only lever on the total is how much reading an item demands, which is why a set drawn from long transcripts costs more per item than one drawn from short ones.

Sizing the set#

Size follows from what you intend to do with the number. At 300 items and a score near 0.6, the standard error of the mean is about 2.8 points. The recipe behind that figure is worth carrying, because it is the one calculation that says whether a run has told you anything. Each item lands as a hit or a miss, so a single item behaves like one weighted coin flip, and the variance of one such flip is the score multiplied by one minus the score — largest when the score sits near a half and shrinking towards either end. Averaging independent items divides that variance by the number of items, and the standard error is its square root. Multiply 0.6 by its complement, divide by the item count of 300 and take the square root, and 2.8 points is the result.

A 95% interval is about two standard errors either side of the estimate, so it spans roughly eleven points end to end and a run that moves from 0.58 to 0.61 has covered about a tenth of that width and has told you nothing. The two statements are one formula stated twice, which is worth noticing because it settles every set size rather than this one: the item count enters only under a square root, so halving the interval takes four times the items, and no plausible annotation budget buys the precision needed to read a two-point change. That is fine for the decisions this set is for, which are large ones — does contextualization work at all, does a strategy change help, did a deploy break follow-ups — and it is not fine for tuning a prompt one clause at a time.

Stratification helps here as well, since a per-pattern breakdown of 60 items each is where the interesting movement shows up: a change that lifts elided predicates from 0.31 to 0.55 while leaving pronouns flat is legible in the breakdown and invisible in the mean. Five strata of 60 items account for all 300 items in the set, the same partition the table's columns closed on, so no item is scored twice and none is left out of the breakdown. That the division comes out even is what makes the breakdown available at all: left at the proportions the logs produce, the rarest pattern would take under a tenth of the set and the breakdown would report it out of a stratum too small to read. The precision inside a stratum is worse in a way the same formula predicts: dividing by sixty rather than three hundred multiplies the variance by five, and its square root by a little over two, so a per-pattern standard error runs more than twice the whole-set figure. A twenty-four-point move like the one above clears that comfortably. A five-point move in one stratum does not, and reading it as real is how a stratified breakdown starts generating stories.

Before you conclude the retriever is at fault#

The first run of a production-derived set usually comes in low. A score of 0.42 against a standalone benchmark reading 0.91 invites the conclusion that contextualization is broken and the fix belongs in the rewriter, and there are four cheaper explanations to eliminate first.

Run the gold_rewrite field straight through retrieval, with no conversation and no contextualizer. That establishes the ceiling: if the hand-written standalone queries only reach 0.61, then the corpus or the retriever caps the whole eval at 0.61, and contextualization can be responsible for at most the 0.19 between there and 0.42. The subtraction closes exactly, and it is worth seeing why that matters. The ceiling is what the pipeline achieves when the query is already perfect, so every point of the score below it is attributable to something that is not query construction, and every point between it and the observed number is attributable to query construction or to nothing. There is no third bucket, which is why this is the check to run first. What the residual does not resolve is the split inside itself — some of the 0.19 is the rewriter and some is the state assembly feeding it — and the bound points one way only: 0.19 is a ceiling on how much blame contextualization can carry, rather than an estimate of how much it does. Teams that skip this step spend weeks tuning a rewriter against a ceiling they never measured.

Audit thirty gold labels by hand. The recurring bug is a granularity mismatch, where gold was recorded as a document path while the retriever returns chunk ids, so every item scores zero on a technicality. The second recurring bug is chunk-id drift, where the corpus was re-chunked after labelling and the recorded ids no longer name anything; on a document set with normal documentation churn, roughly 12% of gold ids stopped resolving within three months on one system that tracked it.

Check what the contextualizer actually received. The prefix in the item is what you intended to load, and the message list that reached the rewriter may be shorter, may have had system or tool messages spliced in, or may have been reordered by an async write. Logging both and diffing them takes an hour and settles the question, and chapter 7 treats the class of failures this reveals.

Measure how many items are unanswerable. If the annotator could not find a correct passage for an item and it was kept in the set with a best-guess label, the item is unscoreable and drags the mean down without indicating anything about the system. A healthy set reports its unanswerable rate explicitly rather than hiding those items among the failures.

Retrieval recall and the answer built from it#

This set measures retrieval recall on follow-up turns and nothing else. An item can pass with the gold passage at rank 9 while the generator, handed nine passages, writes an answer from the wrong one, and it can fail while the assistant produces a perfectly good answer from a passage the annotator did not think of. Both directions are failures of attribution rather than of measurement: the score is doing its job, and it is a score about a list of passages. Answer quality needs its own judgement, and conflating the two produces a metric that moves for reasons you cannot separate afterwards.

The set also decays, more quickly than the standalone benchmark does. Conversation prefixes reference product names, plan tiers and regions that get renamed; gold ids rot as documents are rewritten; the reference-type distribution shifts as the product's users change. A standalone question ages along one of those axes and a conversational item ages along all three, which is why the two sets need different re-labelling budgets rather than a shared one. Budgeting a re-labelling pass every quarter, or at least a validation pass that flags gold ids which no longer resolve, is the difference between a set that stays honest and one that quietly becomes a fixture.

Keeping the original 400-question benchmark alongside it is the right call for the same reason, and the pair is more informative than either alone. When both drop, the retriever moved. When only the follow-up set drops, the contextualization path moved, and the two require entirely different investigations. The asymmetry is what makes that reading valid: the standalone set is insensitive to the stage by construction, as the first section derived, so it works as a control that cannot be contaminated by the variable under test. The mirror-image reading is the one the pair does not support. A standalone benchmark that falls while the follow-up set holds steady says nothing good about the contextualizer; it says the retriever regression missed the passages the follow-up set happens to ask for, which two sets labelled independently over the same documents will do to each other routinely.

The invariance is the thing to carry out of this chapter. A number that stays put while the mechanism it names is switched on, switched off and removed altogether is not a weak signal about that mechanism, and treating it as a weak one is how a working feature gets reverted on the strength of a half-point dip. The repair is not a better threshold or a larger sample. It is an item shaped so that the query does not exist until the system builds it, which is the only condition under which a contextualizer can be observed doing anything at all.

With a set that can distinguish those cases, the choice between ways of getting conversational context into the query stops being a matter of taste. Two implementations that both look reasonable on a whiteboard produce different numbers on the same 300 items, and they also fail differently in ways the numbers alone do not capture.