A stage that retrieval or a rule can do needs no model call

A split passes the information test when its second stage genuinely receives something the first did not. "Select the relevant sections, then answer using only those" passes: the answering call sees a subset chosen by a process that read the whole document, which is different information from what the answering call would have picked out for itself. It is still, usually, the wrong change, because the first stage is doing a job that has had non-model solutions for thirty years.

The habit this chapter is against is reaching for the most expensive instrument in the cabinet first. A model call is a general-purpose reasoner priced and timed like one; a stage that selects, looks up, computes or dispatches is asking for something narrower, and the narrower tools are two to three orders of magnitude cheaper and faster, auditable, and reproducible. Knowing that a third option exists changes the question in review from "is the extra call worth it" to "what is the cheapest thing that does this stage's job".

Selection is retrieval, and retrieval is not a new problem#

Picking the parts of a document that bear on a question is the founding problem of information retrieval. BM25, the ranking function most keyword search still runs on, was published in 1994, scores a passage by how often the query's rarer words appear in it, and takes microseconds over a few dozen sections.

The word "rarer" is carrying the whole idea there and is worth unpacking, since it is what makes a word-counting method work at all. A term appearing in most sections of a document separates nothing: if every section of an expenses policy contains "expense", then matching on "expense" ranks every section equally and the ranking is no better than the order they were written in. A term appearing in one or two sections is close to a pointer to those sections, so a match on it is strong evidence. Weighting each matched word by how rare it is across the document is therefore not a tuning trick — it is the only way a count of shared words can carry information about which section is relevant. That is also the method's limit, stated in advance: a question whose wording shares no rare term with the section that answers it gets no signal at all, which is precisely the case the next method exists to cover.

Dense retrieval over embeddings — an embedding being a list of a few hundred numbers computed from a piece of text and arranged so that texts with similar meaning land near each other — arrived commercially in the late 2010s and answers the same question with semantic rather than lexical matching. Both existed and were well understood before anyone thought to ask a language model to do the selecting.

On a 5,000-token policy document with 38 sections, each carrying a heading, the four ways of choosing sections compare like this:

The bottom three rows all produce a set of sections and all of them are auditable in a way a model call is not: you can print the BM25 scores, you can show the rule that fired, and when the wrong section comes back you can reproduce the selection deterministically from the same inputs. The top row produces a set of sections and a new thing that can hallucinate a section number, time out, get rate-limited, and drift when the model version changes.

That does not make the model call always wrong. There are selection decisions that keyword and vector search handle badly — a question whose answer lives in a section sharing no vocabulary with it, or a selection that depends on the customer's plan tier and the date of their contract at once. The point is that reaching for the model first, for a job with cheap deterministic solutions, is the error.

Auditability is the row of that table that gets undervalued in review and turns out to matter most in operation. When a wrong answer is traced back to a bad selection, the deterministic options hand you the reason immediately: here are the scores, here is the rule that fired, here is the same result reproduced from the same inputs on a laptop. A model-based selector hands you a completion that cannot be reproduced, because temperature, model version and provider-side batching all move it, so the investigation starts by trying to make the failure happen again. On a stage that selects, the cost of an incident is dominated by that step rather than by the fix.

Which substitute fits which stage#

The choice among the substitutes follows from what kind of decision the stage is actually making.

The last leaf is the one teams forget. Selection is only worth doing when the document does not fit, or when sending all of it degrades the answer. At 5,000 tokens a policy document fits comfortably in any current context window, which is the ceiling on how much text a model can be handed in a single request, so the only argument for selecting is that the model answers better from 1,200 relevant tokens than from 5,000 mostly-irrelevant ones — a real effect and worth measuring rather than assuming.

The first thing to measure is not accuracy but coverage: what fraction of questions have all their answer-bearing sections inside the retriever's top k. That number is the ceiling on what the whole approach can deliver, it is knowable in an afternoon without touching the prompt, and it comes from labelling the correct sections rather than from running any model.

That ceiling holds over what retrieval can supply, and the qualifier matters, because a measured accuracy sitting above the coverage number is possible and is a warning rather than a bonus. It means some questions were answered correctly without the answer-bearing section being present, which can only have come from what the model already believes documents of this kind usually say. Sometimes that belief is right. It is right on the general questions and wrong on exactly the specific ones — a threshold, a deadline, an amount — which are the questions a policy document exists to settle. So the pair of numbers to look at is coverage and accuracy together: accuracy comfortably under coverage is a system using its documents, and accuracy at or above coverage is a system that has learned it can sometimes skip them.

Sweeping k, because coverage and accuracy do not peak together#

Coverage rises monotonically and accuracy does not, which is the whole argument for selection stated as a measurement.

A curve that rises and then falls is always two effects pulling against each other, and naming both is what makes the peak predictable instead of lucky. Raising k improves one thing: the chance that the section holding the answer is in front of the model at all. That is the coverage column, and it can only go up, because a larger k is a superset of a smaller one. Raising k also worsens a second thing: the chance the model picks the right section out of what it was given. Every section added after the right one is, by construction, a section the retriever thought was nearly as relevant — and near-misses in a policy document are the sections that discuss the same kind of fact, so they are the worst available distractors rather than material the model can ignore. The product of a term that is rising and a term that is falling peaks somewhere in the middle, and the sweep is how you find out where.

The top row is where the framing is easiest to see, because only one of the two effects is really operating. At k=1 there is almost nothing to be distracted by, so nearly all the loss is the answer simply not being present. Every row below it trades some of that loss for some of the other kind, and the two have crossed over by the time the table reaches its last row.

At k=8 the retriever is almost never missing the right section and the answer is slightly worse than at k=4, because the four extra sections are plausible neighbours — a question about a reimbursement deadline pulls in the expense-approval section and the travel-advance section, both of which state deadlines that are not the one being asked about. The bottom row is the whole document in one prompt, and reading the table upwards from it is the clearest available statement of what a single prompt loses to irrelevant context.

Run the selection with BM25 and feed the result into the existing single prompt, and the arrangement still has one model call in it:

def answer(question: str, doc: Document) -> str:
    sections = bm25_top_k(doc.sections, question, k=4)   # ~3 ms, no tokens
    context  = "\n\n".join(s.render() for s in sections)  # ~1,200 tokens
    return model.complete(QA_PROMPT.format(context=context, question=question))

Sometimes the fix is inside the prompt you already have#

There is a fourth option that costs neither a call nor an index, and it gets skipped because it does not feel like an engineering change. A baseline that sends all 38 sections as an undifferentiated wall of text with the question at the top gives up points to its own layout, and three edits move the number without touching the design:

{numbered_section_index}          # 38 lines: "4.2 Reimbursement deadlines"

{numbered_policy_sections}        # the full document, headings preserved

The user's question is: {question}

Answer using only the sections above. Name the section number your answer
comes from. If no section answers the question, say that instead.

Two mechanisms are doing the work. Material in the middle of a long context is used less reliably than material at either end, so restating the question after the document rather than only before it puts the thing being asked in the position the model attends to most. Requiring a section number forces the model to locate the answer in the document instead of reconstructing it from what it already believes policies of this kind usually say, which is where a fair share of a 9% error rate goes on a baseline like this one.

The second mechanism deserves the extra step, because "forces" is doing a lot of work in that sentence. A section number is not decoration on the answer; it is a token the model can only produce correctly by having attended to a heading in the document it was handed. An answer with no citation can be assembled from a general sense of what expense policies tend to say, and nothing in the response distinguishes that from an answer read off the page. Requiring the number makes the two separable — to whoever reads the output later, because a cited section can be opened and checked, and inside the generation itself, because the citation has to come from the document rather than from memory. The numbered index at the top does the complementary job. It turns choosing a section from a recall task into a choice among named options that are all visible at once, which is a different and much easier thing to ask for.

Anyone proposing a second model call for accuracy owes an answer to why the layout of the first one was left alone, because layout edits are reversible in a deploy, carry no new failure surface, and can be evaluated the same afternoon they are written.

Retrieval moves the failure rather than removing it#

When BM25 misses the answer-bearing section, the answering prompt receives four plausible sections that do not contain the answer and produces a confident wrong answer with no signal that anything went wrong. A model-based selector, for all its expense, can be asked to say "none of these sections address this question", and a BM25 scorer returns its top four regardless of whether any of them are any good. You get some of that back with a score threshold — if the top section scores below a floor, refuse or escalate — and thresholds are their own tuning problem with their own eval. On this endpoint a floor at a BM25 score of 6.5 escalated 3.2% of questions, caught 61% of the selection misses, and escalated 1.9% of all traffic unnecessarily, which is the trade the threshold buys and the reason the floor was set by looking at the escalation queue's capacity rather than at the score distribution.

Chunk granularity — the size of the unit the retriever indexes and returns, a chunk being whatever slice of the document you decided to treat as one item — is the second failure retrieval brings with it, and it is invisible until someone reads the retrieved text. The 38 sections here are the document's own headings, which happen to be a sensible unit; a later change that sub-split anything over 400 tokens for a different consumer cut three rate tables across chunk boundaries, so a question about per-diem allowances retrieved the table header and the first four rows while the answer lived in row nine. That cost 1.1 points and took two weeks to attribute, because the answers were confident, well-formed and drawn from a section that was genuinely the right one.

Freshness behaves differently across the substitutes, which matters when the underlying documents change. BM25 over 38 sections is cheap enough to index at request time, so a policy amendment is live the moment it is written. An embedding index needs a re-embed job, and a nightly one means an amendment published at 09:00 is answered from the previous text until the next run, which is a correctness bug wearing the costume of a scheduling detail. A model call has no staleness of its own, because it reads whatever document you hand it, and that is one of the genuine advantages the expensive option retains.

Replacing a model call with retrieval does not remove the obligation to evaluate the stage; it changes what the eval measures. A model selector is scored on whether it chose well, which requires reading its output. A retriever is scored on coverage against labelled answer-bearing sections, which is a cheaper label to collect and a number that can be recomputed on every document change without spending a token. The labelling is the part that carries over: whichever option you pick, somebody has to say for a few hundred questions which sections contain the answer, and that set is the asset. It outlives the retriever, it prices the model call, and it is what turns "the split is expected to reach 94%" from a projection into a claim with a ceiling attached to it.

Business rules encode policy, and policy changes. A rule that routes enterprise-tier questions to the enterprise appendix will be wrong three months after someone renames the tiers, and nothing will fail loudly. A model call absorbs that kind of drift, which is genuinely why teams reach for it. The trade is flexibility against determinism, and the reasonable position is to use rules where the policy is stable and written down, and to accept a model call where the judgment is open-ended enough that no rule survives contact with the inputs.

The move to keep from all of this is the third column in a proposal that arrived with two. Before agreeing that a stage needs its own call, ask what kind of decision it makes: if it is decided by fields you already hold, it is a rule; by word overlap, BM25; by meaning without shared wording, an embedding lookup; and if nothing needs selecting at all, send more of the document and skip the stage. Then ask whether the single prompt's layout has ever been tried, because that option costs an afternoon and no runtime at all.

None of this tells you whether the arrangement you end up with is working. Whichever option you chose, you now have a multi-stage system whose end-to-end accuracy is a single number, and a single number does not say which stage to work on.