An oracle query tells a bad question from a missing document

A single unretrievable query can be diagnosed by reading it. A failing population cannot, because the four defects of chapter 1 all produce the same log line as three failures that have nothing to do with the query at all, and reading one example tells you which kind you happened to pick rather than which kind you have. What is needed before any transformation is built is triage: a way to sort a sample of failures into buckets that are repaired in different places, so that the size of each bucket is known before anybody funds a repair for it.

Two terms carry that work. A gold passage is the chunk in the index that actually contains the answer to a question, located by hand or by grep, independently of what retrieval returned — it is the ground truth for one failure, established without asking the retriever anything. An oracle query is a query written out of that passage's own wording rather than out of the user's, and asking the index with it tests whether the passage is reachable at all. The two together turn a question about phrasing into a question with a yes-or-no answer.

The system these are worked on is a support search over a corpus of 400,000 tickets and their resolutions, where the standing number on the dashboard is that 62% of questions get an acceptable answer. That figure comes from a 400-question eval set with human judgements, so 152 questions fail. The more useful question is how much of the failing 38% is even the kind of failure a query-side layer can address, because nobody in the room knows, and the answer determines whether a prototype's result is close to the ceiling or barely a third of it.

The measurement that comes before any transformation#

Take a random sample of the failures — 60 of the 152 is enough to size a bucket to within a few points, and it costs about four hours of one engineer's attention at three to five minutes per case. For each one you need the gold passage: the chunk in the index that actually contains the answer, located by hand or by grep, independent of what retrieval returned.

Then run the oracle query. Instead of the user's question, build a query out of the gold passage's own wording and retrieve with that. If the passage does not come back for a query written from its own text, the failure has nothing to do with how the user phrased anything.

def oracle_probe(gold_chunk_id: str, index) -> dict:
    chunk = index.get(gold_chunk_id)

    # 1. Literal self-query: the passage's own opening, verbatim.
    literal = " ".join(chunk.text.split()[:40])

    # 2. Generated question: what a reader would ask, in the passage's words.
    generated = llm(
        "Write the single question this passage answers. "
        "Use only vocabulary that appears in the passage.\n\n" + chunk.text
    )

    return {
        "literal_rank": rank_of(gold_chunk_id, index.search(literal, k=50)),
        "generated_rank": rank_of(gold_chunk_id, index.search(generated, k=50)),
    }

The two probes fail in different ways and the difference is the diagnosis. A literal self-query is close to the strongest possible retrieval signal, because the query and the document share their exact surface form; if the passage does not come back at rank 1 for that, something is wrong with the index rather than with anybody's question. The generated question is a milder probe that also spans the gap between how a question is worded and how an answer is worded, which is the gap every transformation in this topic is trying to close from the other side.

Why the literal probe should be that strong is worth walking rather than asserting, because the whole triage rests on it being a near-guarantee. An embedding model is a function from text to a point, so identical text maps to an identical point. A chunk's own opening words are not identical to the whole chunk, but they are a large and contiguous piece of it, which places their vector very close to the chunk's own vector — far closer than any other document's, since no other document was written from the same sentences. The retrieval step returns whatever sits nearest, and the chunk itself is what sits nearest. So rank 1 is the expected outcome, and any other outcome means the chunk's vector is not where its text says it should be. That is a statement about storage rather than about phrasing.

The second probe is built to move exactly one variable, and that is what makes the pair informative rather than merely redundant. The instruction pins the generated question's vocabulary to the passage's own words, so the only thing that changes between the literal probe and the generated one is grammatical form: a declarative fragment becomes an interrogative sentence. Any drop in rank between the two is therefore attributable to form and not to word choice, because word choice was held constant by construction. That is a controlled comparison of the kind the rest of this topic will keep wanting and rarely get.

That fourth row is the one people skip, and it accounts for a fifth of the sample here.

Nothing about the probe's design was forced. The requirement is to separate failures a query-side layer can reach from failures it cannot, and several designs satisfy it. Reading the retrieved chunks by hand would satisfy it, slowly. Comparing the user's query against the gold passage with a stronger model would satisfy it, expensively. Retrieving with the passage's own words satisfies it with one search call and no judgement, and the reason it is the choice made here is that its failure case is unambiguous: a chunk that cannot retrieve itself has a defect nobody can argue about. The sample size, the depth of k=50, the forty-word prefix and the two-probe pairing are all decisions of the same kind — chosen because they make the verdict cheap and legible, not because the requirement demanded them.

What the sample actually contained#

Of the 60 sampled failures, 11 did not survive the oracle probe at all. Four gold passages turned out not to be in the index: the ingest job filters attachments over 2 MB and those four resolutions lived in attached PDFs. Three were in the index as extraction noise, the same PDFs run through a text layer that produced ligature soup — specifi c confi guration with broken spacing, which embeds as nothing in particular. The last four were chunk-boundary damage: the resolution text sat in one chunk and the ticket's product identifier sat in the previous one, so the chunk containing the answer contains no evidence of which product it is about. No rewrite of the user's question reaches any of these eleven, and three of the four failure types are fixed in the ingest pipeline rather than anywhere near query time.

That count of failure types reads as one too many until you check it against the table above, and it is worth checking rather than correcting. The first row of that table names three ways a passage can fail the literal probe — absent, mis-parsed, or holding a degenerate embedding — and the chunk-boundary case is a fourth, which is why the bucket's repair is written as two places rather than one. Absent, mis-parsed and degenerate are all decided by the ingest job, and only the boundary damage is decided by the chunker. Three of four in ingest and one in chunking is exactly what the table's Ingest and chunking says, so the two artefacts agree and the sentence is right.

Of the 49 that survived, 13 had the gold passage at ranks 11 to 30 for the user's original query. Those are ranking failures. The retriever found the passage and put it just outside the window, and the correct interventions are a reranker over a wider candidate set or a larger k. A reranker is a second, slower model that reads the query and each retrieved chunk together and rescores the shortlist, so it can afford to be more accurate than the vector search because it only ever looks at fifty candidates rather than four hundred thousand. Both interventions are cheaper than a generation call in front of every query. Transformation can move these, but it moves them the way a lottery ticket moves your finances.

Another 9 had the gold passage inside the top-10 already. Retrieval did its job and the answer was still judged unacceptable, which puts the fault in generation: the passage was in the context and the model summarised the wrong part of it, or hedged, or answered a different question. Those nine are in the failure count and no query-side change touches them.

That leaves 27 with a genuine query defect, sorted against the four from chapter 1.

The count column adds to exactly the sample, with nothing left over, and the share column adds to exactly one hundred. Neither is guaranteed by anything. Seven counts add to the sample only if every failure was placed in one bucket and no failure was placed in two, and seven independently rounded percentages add to a hundred only if the rounding errors happen to cancel. The first of those is the load-bearing one, and it licenses a specific liberty taken repeatedly below: buckets may be added together and subtracted from the whole. If a failure could sit in two rows — a compound query whose gold passage also happened to be mis-parsed — then adding the vocabulary, compound, referent and filler rows would count it twice and every figure derived from that sum would be inflated.

Read the rows against each other rather than one at a time and a second thing falls out. The ranking bucket is larger than the vocabulary bucket, and larger than any other single query defect, yet it is the only one of the seven whose repair does not touch the query at all. Two rows that sit close together in size sit at opposite ends of the cost scale, which is the comparison the dashboard's single failure rate makes impossible to see.

Turning the partition into a number#

The four query-defect buckets are 27 of 60, so 45% of failures carry a defect that this layer addresses. Failures are 38% of traffic, which puts 17 points of the overall acceptable-answer rate inside the reach of query transformation, and 21 points permanently outside it.

Those two figures were reached from opposite directions and they close on the whole. Multiplying the reachable share of failures by the failing share of traffic gives the reachable points. Multiplying the unreachable share of failures by the same failing share gives the unreachable points. The two products add back to the full failing share with no remainder, which is what tells you the partition survived being turned into a rate — every failing question is counted once on one side or the other, and none of the arithmetic has quietly created or destroyed traffic. What it licenses is treating 17 as a property of the population rather than as an artefact of the order the multiplications were done in.

There is a residual, and it is not in the arithmetic. Every share in the table came from 60 cases drawn out of 152, so each one carries sampling error that the exactness of the column sums says nothing about. The identity closes inside the sample and does not close against the population, and reporting 17 without that qualification is the way a well-conducted four hours turns into a false precision.

Seventeen points is a ceiling in the strict sense: it is what you would get if every transformation were perfect, repaired every instance of its defect, cost nothing, and broke nothing else. Real conversion is well below that. A rewrite that binds referents correctly nine times in ten, a multi-query that surfaces the right vocabulary two times in three, and a decomposer that gets dependency order right most of the time compound down to something between a third and a half of the ceiling, so 6 to 8 points is the honest expectation before anyone builds anything.

The lower end of that range is the one that closes exactly: a third of the ceiling is the bottom of the stated band, to the point. The upper end is rounded down, because half of the ceiling falls between whole points and the band is quoted in whole points. So the range is honest at its pessimistic end and slightly conservative at its optimistic one, which is the right direction for a number that will be quoted back at a delivery review.

The partition also reprices the alternatives, which is the part that makes the four hours pay for themselves. Thirteen of 60 failures are passages sitting at rank 11 to 30 for the query the user already typed. A cross-encoder reranker over the top-50 — cross-encoder being the usual architecture for the rescoring model, which takes the query and the chunk as a single joined input instead of embedding them separately — is one additional model call at roughly 90 ms for 50 candidates, with no generation in the path, and it addresses a bucket almost as large as the vocabulary gap. Eleven of 60 are ingest defects, and lifting the attachment size limit is an afternoon. Compared against those, a transformation layer on every request is neither the cheapest available point nor the largest, and the only way to know that was to open sixty tickets.

The limits of a reachability test#

The oracle probe is a reachability test and not a quality measure. A passage that comes back at rank 1 for its own first forty words has told you it is indexed, embedded sanely and retrievable in principle. It has not told you that any realistic user question will find it, and running the literal probe across the whole corpus produces a number near 1.0 that means nothing at all — every chunk retrieves itself. The probe earns its keep only on the failing population, as a filter, and reporting it as a metric invites exactly the misreading that chapter 5 takes apart in a different guise.

It also does not predict that transformation will work on the 27. Those queries have a defect of a kind that a transformation targets, which is a necessary condition and not a sufficient one. A vocabulary-gap query where the corpus uses a term no model would ever guess — an internal codename, a customer-specific abbreviation — is correctly sorted into the vocabulary bucket and is still unreachable by any amount of paraphrasing, because the missing words are not recoverable from general knowledge of the language. Sizing the bucket tells you where to look, and it does not tell you what you will find there.

The sampling has its own limits. Sixty failures give a bucket estimate with a standard error around 6 percentage points, so a bucket reported at 20% is somewhere between roughly 14% and 26%, which is fine for deciding what to build and too coarse for deciding between two buckets that came out at 20% and 22%. That band is one standard error wide and should be read as such: an interval of that width misses the true value a fair fraction of the time, and a claim that two buckets differ needs a good deal more evidence than two non-overlapping point estimates. And the sample is drawn from questions users asked of a system they have already learned the habits of. People who have been told the assistant is bad at multi-part questions stop asking them, so the compound bucket in a mature deployment is smaller than the compound bucket in the traffic you would get from users who had no such training.

The bucket nobody plans for#

Three of the 60 failures are filler-only, which is 5% of failures and about 2% of traffic. The prototype under discussion runs a rewrite step on 100% of queries.

That asymmetry is not automatically wrong; a rewrite that leaves a well-formed query untouched costs a generation call and nothing else. The assumption doing the work is that it does leave the query untouched. A model asked to turn a question into a search query will do something to every input it is given, because that is what it was asked for, and the queries in this corpus that are already precise are precise because they contain dates, product versions, error codes and negations. Those are exactly the tokens a shortening pass has the least reason to keep.

The belief worth naming here is the one a reader will write down confidently after the paragraph above: that a rewrite step is safe on the queries it is not needed for. It is tempting because the rewrite's instruction is conditional — shorten if there is filler — and it is false because the rewrite's behaviour is not. The step runs the same model with the same prompt on a question stuffed with pleasantries and on a question consisting of an error code and a version number, and its output for the second is not marked in the log as having been left alone. A defence that behaves identically whether or not the condition it is defending against is present is not a defence; it is an unconditional edit with a conditional description. The share of traffic that needs it is the small one, and the share of traffic exposed to it is all of it.

What the four hours buy, then, is not an answer about transformation. It is a partition of a single failure rate into parts that are repaired in different places, with a size against each part and a ceiling on what the part in question can ever be worth. Seventeen points is the most a perfect query-side layer could return here, six to eight is what a real one should be expected to return, and the two buckets that are larger or cheaper than any of it were invisible in the dashboard that started the conversation.

The next question is what a transformation actually does to a query once it runs, starting with the mildest of the four repairs and the one that looks safest.