A rewrite that strips filler can strip the constraint too

The mildest of the four repairs is the one that shortens. A filler-stripping rewrite sends the user's message through a model with an instruction to produce a concise search query, and searches with whatever comes back. It is the transformation people reach for first, because the failure it addresses is the one that is visible without any instrumentation: users of an internal assistant type the way they type in chat, and the documents being searched were written by people who do not.

The setting for this chapter is a finance assistant over a corpus of 9,000 chunks of filings, board packs and audit memos. Two terms are needed before any number below means anything. Recall@10 is the share of a set of questions whose known-correct passage turns up somewhere in the ten chunks retrieved — it says nothing about ordering within those ten, only about whether the answer reached the generator at all. A gold passage is that known-correct chunk, marked by hand for each question and fixed independently of anything the pipeline does.

REWRITE = """Rewrite the user's message as a concise search query.
Remove conversational filler. Return only the query."""

rewritten = llm(REWRITE, user_message)   # temperature 0, ~40 tokens out
hits = index.search(rewritten, k=10)

Query A from chapter 1 goes in as hey quick one — for the 2023 filing, did we NOT have to restate Q3 revenue? and comes out as restate Q3 revenue. It is a clean, well-formed, professional-looking search query. Against the labelled set it is worse than what the user typed: recall@10 across 220 questions falls from 0.68 with the raw questions to 0.54 with rewrites. At 0.54, for 101 of the 220 the answer never reached the generator at all.

What the shortened query is now asking#

The gold passage for this question is a paragraph in the FY2023 audit memo reading, in part, "Management concluded that no restatement of previously issued quarterly figures was required for Q3; the immaterial classification difference identified in the review was corrected prospectively." Against the raw question, that passage ranks 3rd. Against restate Q3 revenue, it ranks 14th.

Everything above it in the rewritten run is a genuine match for the query that was actually sent. There is the FY2019 restatement of Q3 revenue, which is a real restatement of real Q3 revenue and therefore an excellent match. There is the FY2021 memo on the revenue recognition change, which discusses restating three quarters. There is the accounting policy page defining when restatement of quarterly revenue is required. None of these are retrieval mistakes; the retriever was asked for passages about restating Q3 revenue and it returned ten of them, ordered well.

Two things were deleted along with the word "hey". The year is a filter: 2023 reduces the candidate space from every filing period the company has ever had to one, and losing it is losing roughly a factor of six on this corpus. The negation is stranger and more important, because it does not narrow the candidate set so much as invert the intent. The user is asking for confirmation that something did not happen, and the passage that confirms it is phrased as no restatement was required. A query asking for restatement discussion ranks passages about restatements happening above a passage about one not happening, which is correct behaviour and the exact opposite of what the user needed.

The two deletions therefore do different arithmetic, and keeping them separate is what makes the fix designable. Removing the year multiplies the number of plausible candidates, because every filing period that was previously excluded is now admitted, and the correct passage's rank drops in proportion to how many of those newcomers outrank it. Removing the negation does not multiply anything. It changes which end of the ranking the correct passage sits at, since the query now describes the event whose absence was being asked about. One is a widening and one is a reversal, and a repair that only guards against widening — a rule about keeping the query narrow, say — would let the reversal through untouched.

Preserving the negation is necessary and it is not sufficient, which is worth being clear about before the fix in this chapter gets oversold. Dense embedding models encode negation weakly; did we have to restate Q3 revenue and did we not have to restate Q3 revenue land close together in most embedding spaces, because the models are trained on objectives where topical similarity dominates and a single function word rarely moves a sentence vector far. Keeping the NOT in the query recovers a little of the ranking and does not reliably put the "no restatement was required" passage above the ones describing restatements. What it does do is keep the polarity in the text that reaches the generator, which is the stage that can actually adjudicate it — a top-10 containing both the FY2023 memo and three genuine restatements is a context a model can answer from correctly, and a top-10 with the FY2023 memo missing is not. The realistic goal for polarity is getting the right passage into the window, with the generator resolving the direction.

That is a scope worth stating plainly, because it is narrower than the chapter's title suggests. Protecting the negation buys admission to the window and not correctness of the ranking inside it. The claim "keep the negation and the query works" is false; the claim "keep the negation and the passage that settles the polarity is in front of a model that can read it" is true, and only the second is defensible from what was measured here.

A useful way to think about the constraint is as a query optimiser rewriting SQL — an analogy, and one worth holding at arm's length. An optimiser may reorder joins, push predicates down, and choose a different index, and it may not drop a WHERE fiscal_year = 2023 clause because the plan is faster without it. The result set is the contract. A query rewriter has the same contract with a fuzzier notion of result set: it may change how the intent is expressed and it may not change what the intent selects for.

The analogy breaks at the word "contract", and the break is where the misconception forms. An optimiser's contract is checkable: two plans either return identical rows or they do not, and the engine knows the schema well enough to prove that dropping a predicate changes the answer. A rewriter has no such test. Its result set is a ranking, equality between rankings is not the right notion of correctness, and nothing in the system knows which documents exist that the query is now admitting. So the enforcement below is a proxy over the query's tokens rather than a proof over its results, and a reader who takes the analogy too literally will expect a guarantee the mechanism cannot give.

The token classes that carry selection#

Sorting the 220 questions by what their rewrites removed makes the pattern legible.

Only the last row is safe to remove, and it is the row the prompt was written for. Every other row is a selector, which means the rewrite is a deletion operation over a set where about a fifth of the deletable material is noise and the rest is the question.

Read the third column downwards and the rows sort into two kinds of damage rather than nine. Absolute time, named entity and numerals with units all widen: they admit candidates that were previously excluded, and the correct passage falls by however many of those candidates outrank it. Negation, comparatives, quantifier scope, modality and answer-type words all change what would count as a correct answer at all: the query after the deletion is asking a different question, and a perfect retriever answering it perfectly returns the wrong thing. A widening can be survived by retrieving more candidates; a change of question cannot be survived by anything downstream, because there is nothing wrong with what came back.

The reason the shortening instinct is so hard to argue with is that it is right about the mechanism and wrong about the arithmetic. Conversational filler does dilute a query vector; a 22-token question where 9 tokens are greeting genuinely places the vector closer to chat-like text than the same question without them. On the 63 questions in this set whose only defect was an opener, rewriting moved recall@10 from 0.61 to 0.71, which is a solid result for a fourteen-line prompt. The layer then applied the same operation to 157 questions that had constraints in them, and the losses there are what pulls the total to 0.54.

Those three figures close on the aggregate, and checking that they do is worth the two minutes it takes. The opener-only slice at its rewritten rate accounts for a known number of successes, and subtracting those from the total number of successes implied by the aggregate rate leaves what the constraint-bearing slice must have contributed. Divided by the size of that slice, it says the constraint-bearing questions came back at a rate roughly a third below where they started. Do the same subtraction on the raw run and the constraint-bearing slice started above the aggregate, not below it. A two-slice model with one rate each therefore reproduces both reported averages to within a question or two, and the residual is only rounding, since the reported rates are quoted to two decimal places over a set of 220.

What that licenses is the sentence the two averages cannot support. The rewrite did not degrade recall a little everywhere. One population went up by about a sixth of where it started and the other went down by about a third, and a single operation applied uniformly produced opposite signs on the two halves of its input. No uniform cause explains opposite signs. The cause has to be something that is present in one slice and absent from the other, which is exactly the presence or absence of a selector token — and that is a conclusion about mechanism reached from two averages and a slice size, without reading a single query.

Comparing the two on a fixed corpus#

The comparison has to hold everything constant except the query text, which means the same index, the same embedder, the same k, and the same relevance labels — a chunk id per question, marked by hand, that does not change when the pipeline does.

def compare(questions: list[Q], index, k: int = 10) -> dict:
    rows = []
    for q in questions:
        raw_hits = index.search(q.text, k=k)
        rw = llm(REWRITE, q.text)
        rw_hits = index.search(rw, k=k)
        rows.append({
            "id": q.id,
            "slice": q.slice,                       # filler_only | has_constraint
            "raw_hit": q.gold_id in ids(raw_hits),
            "rw_hit": q.gold_id in ids(rw_hits),
            "dropped": protected_tokens(q.text) - protected_tokens(rw),
            "rewrite": rw,
        })
    return {
        "recall_raw": mean(r["raw_hit"] for r in rows),
        "recall_rw": mean(r["rw_hit"] for r in rows),
        "regressions": [r for r in rows if r["raw_hit"] and not r["rw_hit"]],
        "rows": rows,
    }

The regressions list is the output that changes minds, because a paired comparison over the same questions tells you something the two averages cannot: 31 of the 220 questions were retrieved correctly before the rewrite and incorrectly after it, and 8 went the other way. Reading the 31 takes twenty minutes and every one of them has a token from the table above missing from its rewrite. Two averages, 0.68 and 0.54, tell you the layer is bad. The paired list tells you exactly which operation is doing it, which is what you need to fix rather than revert.

The pairing is doing more work than it looks. Two averages over the same set can move for two reasons that are indistinguishable in the averages: the same questions got worse, or a different set of questions got better and a larger set got worse. Because every question is run through both arms, the two counts of movers are directly observable, and their difference is the net. Knowing that the net is made of many losses and a few gains rather than a small uniform sag is what turns a revert into a repair.

The dropped field is worth computing even though it looks redundant next to the regression list. It is a check that runs without labels, on live traffic, and it turns "the rewrite hurt recall on the eval" into a signal you can watch in production where no gold chunk ids exist.

Making the rewrite subtractive only where it may subtract#

Two changes take this from a net loss to a small net gain. The first is telling the model what it is protecting, in terms specific enough to act on.

REWRITE = """Rewrite the user's message as a search query for a document index.

Remove: greetings, apologies, hedges, meta-commentary about the assistant,
and requests for formatting.

Preserve verbatim, every one of them, even if the result reads awkwardly:
dates and periods, named entities, numbers with units, version strings,
negations, comparatives, quantifiers (only/all/any/each), modal verbs,
and the word naming the kind of answer wanted (cost, who, when, whether).

If removing the filler would leave fewer than four content words,
return the message unchanged. Return only the query."""

The second is not trusting it. A prompt is a request and a validator is a guarantee, so the rewrite gets checked before it is used.

PROTECTED = (dates | entities | numerals | versions
             | negations | comparatives | quantifiers | modals)

def safe_rewrite(original: str) -> str:
    candidate = llm(REWRITE, original)
    lost = PROTECTED.extract(original) - PROTECTED.extract(candidate)
    if lost:
        log.info("rewrite rejected, dropped %s", lost)
        return original          # fall back, do not repair
    return candidate

The requirement here is that the rewrite must not change what the query selects for. Everything else in those two blocks is a design decision, and worth separating from the requirement so that each can be argued with on its own. Enumerating the protected classes in the prompt is one; a different design would give the model examples instead of categories. Checking with a set difference over extracted tokens is another; a different design would compare retrieved results before and after. Falling back rather than retrying is a third. Refusing to shorten below four content words is a fourth, and it is the one that protects against the case where the filler and the question are the same words.

Falling back to the original rather than asking the model to try again is deliberate. A retry costs another 300 ms and a second chance to lose a different token, and the original query is a known quantity that was measured at 0.68. On this eval the validator rejects 19% of rewrites, and the combination of prompt and validator lands recall@10 at 0.72 against the raw baseline of 0.68 — four points, for two generation calls' worth of latency on every query, of which the second is only paid when the first fails a cheap string check.

There is one more population where the rewrite earns more than four points, and it is not made of questions at all. Users of an internal assistant frequently type a statement rather than a query — this keeps timing out on the export job since we moved to the new cluster — which is a complaint with a diagnosis embedded in it. The corpus contains a runbook titled "Export job timeouts after cluster migration", and the statement retrieves it at rank 6 while a rewrite that turns it into export job timeout after cluster migration retrieves it at rank 1. That transformation is not subtraction; it is a change of register, from first-person narration to the noun-phrase style of a document title. On the 29 statement-shaped inputs in this set the rewrite moved recall from 0.55 to 0.79, and none of the nine protected classes was at risk, because complaints of this kind rarely carry dates or version strings. Register conversion and filler removal are usually implemented by the same prompt and they are separable, and separating them lets the validator run only where it is needed.

Four points from a rewrite layer is roughly what a rewrite layer is worth. The filler-only slice sized in chapter 2 and the statement-shaped inputs above are the populations it repairs, and the honest description of the rest of the traffic is that the prompt and the validator are protecting it from the layer rather than helping it.

Where subtraction runs out#

Rewriting is subtraction. Every mechanism in this chapter is about which tokens leave, and the ceiling on subtraction is the information already in the query. That makes it the wrong tool for query C from chapter 1 — how do I rotate the signing key has no filler to remove, and no amount of tightening introduces the word rollover that the runbooks use. Shortening a query cannot add the corpus's vocabulary to it, which is why the vocabulary bucket needs the entirely different mechanisms in chapters 5 and 6.

It is also not the same operation as the contextual rewrite that query B needs, despite both being called rewriting and both being implemented with a model and a prompt. This chapter's rewrite has a complete question in front of it and is deciding what to remove. A query whose subject exists only in the previous turn is missing a constraint that is not in its input at all, so the model cannot preserve it, and the validator above cannot detect its absence — does that one cover weekends? contains no protected tokens to lose. That rewrite has to read something the user did not type, decide which of several candidates a pronoun points at, and put a noun into the query that was never there, and every one of those steps can go wrong in a way that deletion cannot.

That last observation is the one to carry out of the chapter, because it is a property of the validator rather than of the example. safe_rewrite computes a set difference and returns the candidate when the difference is empty. For a well-formed question whose rewrite kept every selector, the difference is empty and the function accepts. For does that one cover weekends?, the difference is also empty, because there was nothing in the original for the rewrite to lose, and the function accepts just as confidently. The check returns the same verdict for a query it has verified and a query it cannot see, and nothing in its output distinguishes the two cases.

A guard whose answer does not move when the thing it is guarding against does is not a guard. What safe_rewrite actually asserts is narrow and worth stating in those words: no protected token that was present has gone missing. It does not assert that the query is retrievable, that the query is complete, or that the rewrite improved anything. Read as the first, it is a correct and cheap check on a real failure mode. Read as a general claim that the rewrite is safe, it is a false sense of coverage over every defect that does not consist of a token disappearing — which is three of the four in chapter 1, and the whole of the next chapter's subject.