Route each transformation to the defect it repairs

By the end of this topic there are five distinct interventions and no rule for when to use which. Filler-stripping rewrite repairs conversational noise, contextual rewrite repairs an unbound referent, multi-query fan-out and HyDE repair a vocabulary gap, and decomposition repairs a compound question. Each has been measured on the slice it addresses, and each costs somewhere between a couple of hundred milliseconds and well over a second.

Two arrangements are available. A chain runs the stages in sequence on every request, each stage taking the previous stage's output as its input. A router runs a diagnosis on every request and then takes exactly one branch, including a branch that does nothing. The two look like variations on a theme and behave nothing alike, because a chain has no way to decline and its stages compose their errors rather than their benefits.

Reading the ten that broke#

The chained pipeline went in as one change: rewrite the query, expand the rewrite into five variants, generate a hypothetical for each, fuse everything, rerank, answer. On the 250-question eval, answer accuracy went from 78% to 79%. p95 latency went from 810 ms to 2.5 s and cost per query rose 4.1x.

One point is two and a half questions out of 250. The per-question diff shows what produced it: twelve questions moved from wrong to right, ten moved from right to wrong.

Those two counts are the finding, and the aggregate is what hides them. Twenty-two questions changed answer, which is nearly a tenth of the set, and the reported movement is the small difference between two large opposing effects. A single accuracy figure returns the same value for a pipeline that moved twenty-two questions and one that moved two, so it cannot distinguish a layer with no effect from a layer with two strong effects that nearly cancel — and those call for opposite decisions. The first should be deleted; the second should be routed.

Each of the ten traces to a failure mode from an earlier chapter, and the pattern is that the chain applied every repair to every query regardless of what was wrong with it.

Four were constraint-stripping. The rewrite step removed a date or a version string, and because the rewrite is the first stage, all five variants and all five hypotheticals were generated from the shortened text. The year was not in any of the eleven query vectors that reached the index. This is the sharpest argument against chaining: a chain does not average its stages' errors, it propagates them, so an early stage's mistake is present in everything downstream and every later stage adds confidence to it.

That count of eleven is worth reconstructing, because it is the whole argument in one number. One rewritten query, five variants written from it, and one hypothetical written from each variant is one plus five plus five, and every one of the eleven descends from the same shortened string. Nothing in the fan-out reintroduces material the first stage deleted, because no later stage has access to anything except what it was handed. Fan-out therefore multiplies the corruption instead of diluting it, and fusion — whose entire mechanism is promoting what several legs agree on — promotes the corruption hardest, since all eleven legs agree about the missing year by having lost it.

The remaining six sort cleanly against the earlier chapters. Three were wrong-version retrievals from the hypothetical. Two were single-intent questions that the decomposer split into fragments. One was a follow-up turn whose referent got bound to the wrong antecedent from six turns back. Four plus three plus two plus one accounts for all ten with nothing unexplained, which matters more than it looks: a residual would mean the chain had a failure mode none of the earlier chapters described, and there is none.

The twelve that improved were, without exception, questions where the user's phrasing missed the corpus vocabulary. Every one of them would have been repaired by the vocabulary branch alone.

The eval itself explains why there were only twelve. It is 250 questions written by the team, in complete sentences, with the product name and version in them, one intent each, no conversation history. That is a set constructed to have almost none of the four defects, and transformation is a recall intervention, so it only pays on questions that are failing for recall reasons. Fifty-five questions were failing at baseline, and most of that failure was ranking, chunking and generation rather than query formulation. An eval of clean questions measures a layer aimed at broken ones at close to its floor.

The routed version costs one diagnosis on every query and one transformation on the queries that need one. It also has a property the chain does not: a query that is diagnosed as clean is untouched, which means it cannot be made worse.

That property is stronger than a measurement and it is worth being precise about why. The clean branch is the identity function — it returns the query it was given, unchanged, to the same retrieval call the baseline used. So on that branch the routed pipeline and the baseline are the same computation, and their outputs are equal by construction rather than by having come out equal on some eval. No labelled set is needed to establish it, no regression can appear there later, and a model upgrade cannot change it. The guarantee is real, and it is also narrow: it covers only the queries the diagnosis calls clean, so its value is exactly as good as the diagnosis is at not mislabelling a broken query.

The policy, with the cost attached#

Chapter 2's partition gives the shares and the earlier chapters give the effects, so the routing table is arithmetic rather than judgement.

The share column adds to the whole with nothing left over, which is what makes the rest of this section arithmetic. It holds because the branches are exclusive by construction — the code below takes one and returns — so a query is counted once and the shares can be added and subtracted freely.

Two figures in the last column look like the same measurement and are not. The referent row and the filler row both start from a recall of 0.61, and they are different sets measured in different chapters: one is the follow-up turns of chapter 4 retrieved without any rewrite, the other is the opener-only slice of chapter 3. Neither is a baseline for the other, and averaging them would be meaningless.

def retrieve(query: str, history: list[Turn], index) -> list[Chunk]:
    defects = diagnose(query, history, index.stats)     # chapter 1

    if "referent" in defects:
        query = resolve(query, history).query           # chapter 4
    if "compound" in defects:
        return execute(plan(query), index)              # chapter 7
    if "vocabulary" in defects:
        return hyde_fused(query, index)                 # chapters 5, 6
    if "filler" in defects:
        query = safe_rewrite(query)                     # chapter 3
    return index.search(query, k=10)

Referent resolution runs before the other tests rather than as an alternative to them, because a query that names nothing cannot be diagnosed for anything else. After that the branches are exclusive, and the order among them reflects which defect makes the others unmeasurable: a compound query has two vocabularies and testing them jointly gives a blurred answer for both.

The requirement being served is narrow: stop paying for repairs on queries that were never broken, without losing the repairs on the ones that were. Everything else in that function is a decision. Exclusivity is a decision, and a system could apply two branches to a query with two defects at the cost of reintroducing the chain's propagation problem on a small slice. Resolving referents first and then continuing is a decision, and it is the one place the function deliberately does chain. Returning immediately from the compound and vocabulary branches is a decision that trades completeness for the guarantee that no query passes through more than one transformation.

The cost line that people get wrong is the tail. Routing pulls the median down to almost nothing, because 81% of traffic gets a 35 ms diagnosis and its original retrieval, and it does not protect p95. The routed share here is 19%, and 19% is larger than 5%, so the slowest 5% of requests all sit inside the routed group and p95 is set by the decomposition branch. Median stays at 700 ms, p75 stays at 740 ms, p95 goes to about 2.0 s. Routing bought back the median and the average bill, and if the requirement is a p95 under a second the answer is a deadline that abandons the transformation and issues the original query when the budget is spent, not a smaller routed share.

The arithmetic under that paragraph is short and worth doing, because it generalises to any routed pipeline. A percentile is decided by whichever slice of traffic occupies that position in the sorted latencies. The routed slice is nearly a fifth of traffic and the untouched slice is the rest, so everything above the eightieth percentile is routed and the ninety-fifth is comfortably inside it. Within the routed group the decomposition branch is the slowest and occupies exactly the share that reaches the ninety-fifth percentile, so p95 is the untouched median plus the decomposition branch's added time. Adding those two gives the quoted figure to the tenth of a second, which is the model closing on itself rather than a coincidence.

The belief this is aimed at is that routing is a latency fix. It is tempting because the median improves dramatically and the median is the number on most dashboards. What makes it false is that the median cannot see the change: it reports the same value whether the routed fifth of traffic costs three hundred milliseconds or three seconds, because the median sits in the untouched majority either way. A statistic that returns an identical answer for the case it should approve and the case it should reject is not measuring the thing being decided, and reading a healthy median as evidence of a healthy tail is the specific error that gets a routed pipeline shipped into a service with a hard deadline.

What "six points for 1.7 seconds" hides#

The support-ticket proposal from chapter 2 has the same shape at a larger scale, and the framing of it as a single trade is where the decision goes wrong.

An aggregate treats the layer as if it did one thing to all traffic. Six points net is consistent with nine points of repair on underspecified questions and three points of destruction on precise ones, and it is equally consistent with six points of clean gain and no damage anywhere. Those are different systems with different users affected, and the summary statistic cannot tell them apart. The precise questions are also disproportionately the ones asked by experienced users who have learned to phrase things well, so a layer that helps the average while hurting precision is a layer that taxes the people who use the product most competently.

The framing also treats latency as the only cost, which makes the next move obvious and wrong. Running the three calls concurrently and caching rewrites keyed on the normalised query does get p95 back under a second, and it does so by making both effects permanent and cheap — the gains and the regressions — before anyone has looked at which queries moved in which direction. A cache in particular converts an intermittent bad rewrite into a bad rewrite served identically to everyone who asks that question, which is the temperature-0 dynamic from chapter 6 arriving through a different door.

The measurement that unblocks the decision is the paired per-question diff, not a better aggregate: the list of questions that were right before and wrong after, read by hand. Twenty minutes on that list tells you whether the layer is helping or is a wash with two large opposing effects, and no amount of additional aggregate reporting will.

Sizing the reachable share for the remaining 38% is the triage from chapter 2 — sample sixty failures, run the oracle probe, sort what survives against the four defects — and it is the same four hours whether the layer ships or not.

Corpora where the answer is no at any price#

Some deployments should not have this layer even if it ran in zero milliseconds for zero cost, because the harm is not in the cost line.

Traffic that is mostly exact identifiers is the clearest case. A support console where users paste order numbers, error codes, case IDs and SKUs — ERR_TLS_CERT_ALTNAME_INVALID, INV-2024-88213 — is traffic where the query is already a key. Any rewrite is a chance to mangle the token, any expansion adds five near-miss identifiers that never existed, and a hypothetical answer invents an error code that looks exactly like the real ones. There is no recall to recover, since an exact-match lookup either finds the row or the row is not there, and every transformation available is a way to make an exact match approximate.

Precision-critical corpora are the second case. In a contract or clinical corpus, retrieving the adjacent clause is worse than retrieving nothing, because a near-miss passage reads as authoritative and gets acted on. Every technique in this topic widens the candidate pool and imports material that is on-topic and not correct, and chapter 6's version-confusion incident is the mild version of that failure — the severe version is a termination clause from the wrong schedule, cited correctly, in an answer that is fluent. A system whose worst outcome is "found nothing, ask a human" should not be traded toward "found something adjacent".

Reproducibility is the third and the least often considered. If answers are auditable — a regulator, a dispute, an internal investigation asking what the system told someone on 3 March — then a non-deterministic model call between the user's words and the evidence retrieved means the same question returns different evidence on different days, and the record of what was asked no longer determines the record of what was found. Logging the transformed query alongside the original mitigates this and does not remove it, because the audit question is usually whether the system would have found the document, and the honest answer becomes "sometimes".

There is also the case where the layer is not refused so much as unmeasurable: a deployment with no labelled set and no per-question diff has no way to see the ten regressions, and shipping a transformation there means shipping an effect of unknown sign.

Running this in production#

The order of work is the order these chapters were written in, and it is deliberately not the order the layer gets built in when nobody triages first. Sample and partition the failures before writing a transformation, so the buckets are sized and the ingest and reranking fixes get priced against the query-side ones. Build one branch, for the largest bucket, and measure it at a matched candidate budget against the original query on the slice it targets. Keep the original query as a fallback path and as one leg of every fusion, so the worst case is the baseline rather than an unknown. Log the transformed query with every request, since the production signal is the diff between what the user typed and what the index saw, and it is unavailable afterwards if nobody wrote it down.

Then watch the two numbers that move independently and get averaged into invisibility: the per-slice effect, and the count of questions that were right before and wrong after.

Every argument in this topic reduces to the same one. A transformation is a repair for a specific defect, its benefit is confined to the queries that have that defect, and its cost and its damage fall on all the traffic it touches. Applying it everywhere is therefore a way of buying a narrow benefit at a wide price, and the diagnosis is what makes the price match the benefit. The pass-through branch is the part that carries the guarantee, because it is the only branch whose behaviour is provably identical to doing nothing.

Routing is the current, practical answer to a problem the field is still working on — learned routers trained on logged retrieval outcomes and adaptive retrieval loops that decide what to do after seeing the first set of results are both more capable and both need the same per-slice evidence to justify themselves. A hand-written diagnosis and five branches is the version you can debug, and it is enough to stop paying for repairs on queries that were never broken.