Prompt fixes and missing evidence

Every retrieval-augmented system eventually ships the abstention line: some form of if the provided context does not contain the answer, say you don't know, appended to the system prompt after a complaint about invented answers. It belongs in the prompt. What it cannot do is decide what the failures it is aimed at were caused by, and on most pipelines a large share of them are caused by a retrieval stage that returned nothing at all.

This chapter is about the boundary between what wording can reach and what it cannot: what an abstention line changed on a support assistant, including the cost that went unmeasured, the empty-context branch that does the thing the line cannot, why the prompt is where the work goes regardless of the evidence, and the arithmetic that says in advance how much a prompt sprint can buy. It closes on the failures where prompt work is the correct and cheap fix.

What the abstention line actually changed#

 SYSTEM = """You are a support assistant for Acme. Answer using the provided
 context. Cite the chunk ids you used.
+
+If the provided context does not contain the answer, say you don't know.
 """

Before the numbers, the mechanism, because the whole chapter rests on one structural fact about where a prompt lives. What the model receives is a single assembled string. Part of that string is the system instruction, written by you and stored in a file. Part of it is the retrieved passages, chosen at request time by the retrieval stage. Those two parts are produced by different code at different moments, and editing the file changes the first part only. If the sentence stating the retention period for the Team plan is in neither part — because retrieval did not select the chunk carrying it — then no edit to the system instruction puts it there, since editing the instruction is not one of the operations that can add a passage. This is not a claim about how well models follow instructions. It would hold for a model that followed every instruction perfectly.

Confidently wrong answers on the eval set fell from 25 to 2. The part that nobody had checked first was the trace field retrieve.out_ids, which on 25 of the 62 failing questions — 40% — was an empty list. The retriever had returned nothing at all, because a similarity floor — a minimum closeness score, on a cosine scale where 1.0 is an exact match, below which a candidate is discarded rather than ranked — was filtering every candidate out:

retrieval:
  top_k: 50
  min_score: 0.72     # cosine; nothing below this is returned

After a chunking change made chunks broader and their embeddings blurrier, scores on genuine matches drifted down into the high sixties, and for a whole class of questions every candidate fell under the floor. Those 25 questions did not have a generation problem. They had no evidence, and the new prompt line changed what the model said about having no evidence.

Following that drift through is worth the paragraph, because the config file contains no evidence that anything happened. A broader chunk covers more subjects, and it still gets one vector to stand for all of them, so its vector sits further from any one question than a narrower chunk's would. Similarity scores across the whole index therefore shift downward together. The retrieval is no worse at ranking; the numbers it is ranking with have been compressed toward the middle. min_score is a constant. It was chosen when the old distribution of scores made 0.72 a sensible dividing line between a plausible match and noise, and it went on meaning that until the distribution moved beneath it. Nothing in the diff that changed chunking mentions min_score, and nothing in the file that holds min_score mentions chunking. A threshold is a claim about a distribution, and it expires silently whenever the distribution does.

Splitting the eval set by whether retrieval returned anything makes the effect legible.

Reading the three rows as a set is what makes the trade visible, since each row on its own supports a different conclusion. The rows partition the 300 eval questions: 25 with no evidence at all, 37 with evidence and a wrong answer, 238 with evidence and a right answer. The change acts on all three, and the release note reported one of them.

The first row is the intended effect and a genuine improvement in safety: a support assistant that says it does not know is better than one that states a 90-day retention window that does not exist, because the wrong answer creates work downstream and erodes trust in every other answer. What the row does not contain is a single additional answered question. The user who asked how long audit logs are kept on the Team plan still does not know, and the retrieval defect now produces a response that reads like correct, cautious behaviour rather than like a bug.

The sentence somebody writes after reading that row is "we fixed the hallucinations", and it is worth holding it up next to what changed. The count of answered questions did not move. The count of correctly answered questions did not move. What moved is the form the failure takes, from a confident false statement to a refusal — a real improvement, because a false retention window creates downstream work and a refusal does not. The reason the stronger reading is tempting is that hallucination is the name of the symptom, so removing the symptom sounds like removing the thing. It is also the reason the defect then survives for weeks: a refusal is the response a well-behaved assistant gives when it genuinely has nothing, so a system producing refusals for a broken reason is indistinguishable from a system producing them for a good one, from the outside.

The third row is the cost that goes unmeasured when a change is judged by the metric it was designed to move. Eleven previously correct answers became refusals, because an instruction to abstain when the context is insufficient is applied by the model to a continuous judgement about sufficiency, and it will apply it to contexts that were adequate but terse. That is a 4.6% loss on the questions that were working — eleven out of the 238 that had been coming back right — traded for a safety gain on questions that were not. It may well be a trade worth making. It is not a trade anyone made deliberately, because the release note said "reduced hallucinations" and the eval that was watched was the confidently-wrong count.

The reason that cost is structural rather than a wording problem is worth being plain about. "Does the context contain the answer" is not a question with a yes and a no on either side of a bright line. Contexts run from a passage that states the fact outright, through a passage that states most of it, to a passage on a related subject, and the model has to draw a line somewhere on that continuum. Any line it draws produces refusals on the far side of it and answers on the near side, so tightening the instruction to catch more of the zero-evidence questions necessarily catches more of the terse-but-adequate ones. There is no phrasing that separates them, because the thing being separated is continuous and the output is binary. What varies with the wording is where the line sits, not whether there is one.

The branch that does what the wording cannot#

What should ship alongside the line is a branch on the empty case rather than a rewording of it. When out_ids is empty, the pipeline knows before generation that it has nothing, and that fact can drive behaviour: retry retrieval with the floor removed, fall back to lexical search, return the top three document titles as suggestions, or route the conversation to a human. Each of those does something for the user. The prompt line handles the case where retrieval returned chunks and none of them are relevant, which is a genuinely fuzzy situation the model does have to judge.

def answer(question):
    hits = index.search(embed(question), k=50, min_score=0.72)
    if not hits:
        hits = index.search(embed(question), k=50, min_score=0.0)[:10]   # floor removed
    if not hits:
        hits = bm25.search(question, k=10)                               # lexical fallback
    if not hits:
        return Escalate(question, reason="no_evidence",
                        suggestions=index.nearest_titles(question, n=3))
    return generate(question, context=[h.text for h in hits[:5]])

Applied to the 25 zero-chunk questions, removing the floor recovered evidence for 14 of them, the lexical fallback recovered 4 more — mostly questions containing exact strings like SCIM and /v1/exports that the dense retriever scored below 0.72 — and 7 fell through to escalation with three document suggestions attached. Those three tiers account for all 25 questions, and each recovers a different cause. Fourteen were questions where good candidates existed and the floor had discarded them, so lifting the floor was enough. Four more carried an exact string that a dense retriever scores by meaning and a lexical index scores by literal match, so they needed the second search rather than a looser threshold on the first. The remaining seven had nothing to find under either method, and for those the honest output is an escalation with suggestions rather than a refusal dressed as caution. Fourteen and four is eighteen; that is 18 questions that now get an answer instead of a refusal, from a change that touches no prompt and no model. It also exposes the floor itself as a defect worth revisiting, since a similarity threshold tuned against one chunking configuration is a constant that silently stopped meaning what it meant.

Why the prompt is where everyone looks first#

The pull toward the prompt is not carelessness, and understanding it is what makes the discipline stick. A prompt change deploys in minutes, requires no reindex, needs no labels, and can be reviewed by anyone on the team by reading it. A chunking change requires a full re-ingest of 12,400 pages, invalidates gold labels stored as chunk IDs, and cannot be evaluated without a retrieval eval that in this case did not yet exist. The prompt is the option that can be attempted today, which means the decision is often made by deployment friction rather than by evidence.

The vocabulary compounds it. "The assistant hallucinates" names a property of the model, and a defect described as a model property is routed to whoever owns the model prompt. Describing the same failures as "40% of them had no retrieved evidence" routes the work somewhere else and is derivable from the same traces. Categories in a failure taxonomy are causal claims wearing descriptive clothes.

The sprint that could not have paid#

The same pattern at larger scale is a team that spends three weeks on a prompt-engineering sprint against the answer generator and moves the end-to-end score 0.4 points on a 100-point scale. Afterwards somebody builds a retrieval component eval for the first time and recall@5 comes back at 0.58.

At recall 0.58, something in the region of four questions in ten reach the generator without the passage that answers them. For those questions the generator's job is to produce a correct answer from material that does not contain it, and no arrangement of instructions changes that — it can only choose between guessing, hedging and abstaining. Three weeks of careful work on wording was therefore operating on at most six questions in ten, and within that six, only on the subset where wording was the binding constraint rather than distraction, ordering or model capability. Using the distribution from chapter 5, on attributing a failure to a stage, where three of thirty-seven failures landed in the cell a prompt change can address, the reachable population for a prompt sprint on this pipeline is small enough that 0.4 points is roughly what you would predict rather than a disappointment.

Stacking those two figures is what turns "small" into something a plan can be argued against. Recall of 0.58 means fifty-eight questions in a hundred arrive with their evidence, so the sprint's population starts at a little under six in ten before anything else is considered. Inside that six, the failures wording can move are the ones where the passage was present and the answer was still wrong for a reason that is not distraction, ordering or capability, which was three of the thirty-seven attributed failures in the nightly build — a little under one in twelve. A fraction under six in ten multiplied by a fraction under one in twelve leaves a reachable population in the low single digits of percent. That is the ceiling on what three weeks of wording could have delivered, and it was computable before the three weeks from two numbers that either already existed or cost an afternoon.

Deciding where the afternoon goes is therefore a question with an answer rather than a matter of taste, and the answer does not turn on anyone's view of how much prompts matter in general. It turns on which row of the failure distribution the failure in front of you sits in.

The process failure is not that prompt work happened. It is that the sprint was scoped before anyone knew the ceiling on what it could deliver, and the number of measurements needed to know that ceiling is one. A retrieval eval is cheap — the recall_at_k of chapter 3, on retrieval recall and answer groundedness, runs without a generation call — and it was built after three weeks of engineering time rather than before, which is an ordering choice rather than a technical constraint. The sprint was also steered by the same end-to-end score throughout, so individual prompt variants could not be told apart from noise on a 300-question set where a 1-point difference is inside the standard error.

Where prompt work is the right move#

Nothing here says wording does not matter, and there is a version of this argument that overcorrects into a claim that generation is never the problem. The cells in the attribution table that belong to generation are real, and they respond to changes that are made in the prompt and in the context assembly around it.

Two rows are worth reading against each other. Distraction failures — where the isolation test showed the model answers correctly from the gold chunk alone — sit downstream of retrieval and are often fixed by supplying fewer chunks, which is a retrieval configuration change even though the symptom appears at generation. And format failures are the case where prompt work is unambiguously correct and cheap, which is part of why prompt work is so attractive: some of it works immediately and visibly, which generalises badly to the cases where it cannot work at all.

The general rule that falls out of this is to read the retrieval span of the trace before opening the template. What that buys is knowing which row of the table you are in, and therefore whether the afternoon you are about to spend has any chance of changing the outcome. An abstention line on a zero-evidence failure is not a fix that partially worked; it is a change to the wording of a symptom, and the more convincing the new wording, the longer the defect survives.

Knowing that a prompt sprint is bounded does not say what to do instead, and "measure retrieval first" still needs defending against the alternative of instrumenting every boundary at once. It also leaves a forecasting question open: if retrieval is the broken stage and you fix it, how much of the end-to-end score comes back, and what evidence would show that estimate to be wrong? Chapter 7, on sequencing which component eval to build first, is where that arithmetic makes the case to a team deciding where a quarter goes.