An instruction to abstain fixes what top-k makes worse

A question-answering system that has already retrieved the right passage has run out of retrieval moves. One such case runs through this chapter: a question asking how many seats are included in a software vendor's Team plan, whose documented answer is ten, and which an internal support assistant answered with five. The passage stating ten was retrieved first and appears in full, unmangled, in the string sent to the model. That case has a property that makes it a good place to start, since the model gets it right when the correct chunk is alone in the prompt and wrong when four other chunks are present. The entire failure is therefore contained in a string your code assembles, built by the render stage — the step that takes the passages retrieval returned and lays them out as text around the user's question.

Retrieval returns a fixed count of passages, conventionally written top-k, where k is the number of highest-scoring chunks requested; this system asks for five. The template that assembles them into a prompt, unchanged since the first chapter of this topic laid the whole chain out as six inspectable boundaries, is four lines long and makes five decisions by omission.

TEMPLATE = """Answer the question using only the context blocks below.

{context}

Question: {question}
Answer:"""

It says nothing about the order the blocks appear in, nothing about where one document ends and the next begins beyond a blank line, nothing about what to do when two blocks disagree, nothing about what to do when none of them answers the question, and nothing about which block to sacrifice when the budget runs out. Each of those five silences has a default, and the defaults were chosen by whoever wrote the retrieval loop rather than by anyone thinking about answers.

Why five blocks arrive whether or not one of them answers#

The number of blocks in that string is fixed before the question is asked, and the reason is a property of what a retriever does. It scores every chunk in an index against the query vector and returns the highest few, which is to say it produces a total ordering over the whole store and then cuts that ordering at a fixed depth. A total ordering always has a top five, so long as the store holds five chunks. Nothing in that procedure asks whether the fifth chunk, or the first, has anything to do with the question, and there is no value of a score that the retriever treats as a refusal. The render stage therefore receives five blocks for a question the documents answer plainly, and five blocks for a question the documents do not address at all.

That is a defect rather than a neutral fact about ranking. What the system has to discriminate is whether the retrieved text contains the answer, and what the retrieval contract varies as that changes is nothing at all: five blocks either way, in the same order, under the same instruction. A reading identical in the case it must catch and the case it must pass carries no information about the distinction, and every wrong-and-confident answer in this chapter comes out of that.

A confident wrong answer on a question the documents cannot answer needs two conditions at once, and they are worth reading as two factors in a product. The first is that plausible-looking blocks arrive regardless of relevance. The second is that the model has no output available to it meaning this is not here. Take either factor to zero and the product goes to zero, and only one of the two is cheap to zero. Withholding results requires the retriever to judge relevance, which is exactly what a cosine score does not tell it. Declining requires one sentence in an instruction block, evaluated by a model looking at the actual text — the only place in the chain where the question does this text state the answer can be asked at all.

The sentence a reader would confidently write after reading that template is that the model has been told to use only the blocks, so a question the blocks do not answer will not get answered. It is tempting because the instruction looks like a safety rule, and it is wrong for a reason unrelated to the model's willingness to obey. Answer the question using only the context blocks below constrains where the words may come from and says nothing about whether words have to come out at all. The obligation to produce output was never on the table for the template to remove, and a model with no sanctioned way to decline satisfies both constraints at once by stretching the nearest block until it covers the question.

As an analogy — and it is an analogy, not a description of how the model works — this is a multiple-choice paper with no none of the above option, where a candidate who recognises none of the answers still marks one. It breaks where it matters most: an exam has a marking key, so a guess is scored wrong within the hour, whereas an answer built out of a superseded pricing page is fluent, names a real document, and is scored wrong only when somebody outside the team notices. The comparison understates the problem.

Five decisions the block is already making#

Order is the cheapest lever and the one with the largest measured effect on this class of failure. The chunks arrive sorted by descending score, which puts the most relevant chunk furthest from the question, and the position table from the previous chapter — twenty chunks held constant while the correct one is moved around inside the block — showed the answer flipping to ten seats when the correct chunk sat last. Reversing the sort so the highest-scoring chunk is adjacent to the question is a one-line change with no cost in tokens.

The step under that observation is worth making explicit, because the obvious version of it does not survive contact with the table it comes from. A long-context model does not weigh every position equally, and material in the middle of a long input is used less reliably than material at either end. That rule predicts the first position should be as good as the last. It was not: the correct chunk at the head of a twenty-chunk block still produced five seats, and only the position adjacent to the question produced ten. The break is the informative part. What distinguishes the winning position is adjacency to the question rather than proximity to an edge, so reversing the sort spends the strongest evidence where the model actually uses it, without adding a token.

Delimiters do two jobs. They stop one document bleeding into the next, which matters more than it sounds when a chunk ends mid-sentence and the following chunk starts mid-table, and they carry metadata the model can use to break a tie. A block header holding the document path and an effective date turns "two documents disagree" from an unanswerable situation into an ordinary precedence question.

def block(hit, chunk, meta):
    return (f'<doc id="{hit["chunk_id"]}" source="{meta["path"]}" '
            f'effective="{meta["effective"]}" status="{meta["status"]}">\n'
            f'{chunk}\n</doc>')

The status field comes from the ingest manifest — a document under policies/ with a superseded_by header gets status="superseded" — and it costs about 25 tokens per block. On the seats query it is decisive on its own, because the 2023 pricing page and the reseller FAQ both arrive labelled superseded next to a chunk labelled current.

That decisiveness is worth one step of unpacking, since a metadata field does no work by itself. The seats failure was a majority vote: one document said ten, two said five, and nothing in the prompt distinguished them. The header supplies a property on which the three blocks differ, and the precedence rule in the instruction block turns that property into an ordering. Neither half works alone, which is why the block header and the system rules are one change rather than two.

Confinement and abstention are separate instructions that are usually written as one. Confinement tells the model to answer only from the blocks; abstention tells it what to emit when the blocks do not contain the answer, so that the pipeline has a way of saying it does not know. Without the second, a confined model still produces something, because producing nothing is not an option it has been given, so it stretches the nearest block to cover the question. The abstention has to name an exact output string, because a fluent apology is not a machine-readable signal and you will want to count these.

Naming an exact string does more work than it appears to. A count is only possible over a set with a decision procedure, and did the model refuse has none when the refusal can be phrased a hundred ways. Fixing the string collapses that to a string comparison, which is what lets the runner below carry an abstained column at all — and without that column there is no way to tell a change that converted wrong answers into refusals from one that converted them into right answers.

Citation is what makes any of this checkable. Requiring every sentence to end with the id of the block it came from costs about 15 output tokens per answer and gives the trace record a field you can verify programmatically: for each cited id, does the chunk text actually contain the claim. On this pipeline, running that check over 200 answers found 11 where the cited chunk did not support the sentence, which is a rate you would never have discovered by reading.

The budget decision was already exposed in the third chapter of this topic, which put the rendered prompt into the trace record alongside a chunks_dropped field naming what the budget clamp discarded, and exposing a decision is not the same as making it. A continue that skips any chunk too large to fit will happily drop rank 1 while keeping rank 5, because rank 1 might be the long chunk. The rule worth writing is to keep chunks in rank order until the budget is exhausted and never to consider a lower rank once a higher one has been dropped, which makes the dropped set always a suffix of the ranking.

SYSTEM = """You answer questions about internal policy using only the <doc> blocks provided.

Rules:
- Use only text inside <doc> blocks. Do not use prior knowledge.
- If two blocks disagree, prefer the block with the later `effective` date,
  and ignore any block with status="superseded" when a current one covers it.
- End every sentence with the id of the block it came from, in square brackets.
- If no block addresses the question, reply with exactly: NOT_IN_CONTEXT
"""

USER = """{context}

Question: {question}"""

Selection and ordering have to happen in that order, which is easy to get backwards once both exist. The budget clamp decides which chunks survive and must do so by rank, so that the dropped set is always the lowest-scoring suffix; the ordering decides where the survivors sit and reverses them afterwards. Sorting first and then clamping from the end of the list drops rank 1, which is a bug that produces answers just plausible enough that nobody investigates. Keeping the two steps as separate functions with the clamp first makes the mistake hard to write.

That ordering is not arbitrary. A clamp is written over list positions while a ranking is a property of the elements, and the two coincide only while the list is still in score order. A clamp that walks from the front and stops when the budget is spent deletes whatever is at the back. Before the reversal the back of the list is the lowest-scoring chunk; after the reversal the back of the list is rank 1. The code has not changed and its meaning has inverted, which is the shape of bug that survives review, because both halves read correctly on their own.

What is required here is worth separating from what has been chosen. The requirement is that the assistant must not state a policy figure the documents do not support, and it holds whichever way the chain is built. Reversing the sort, wrapping blocks in <doc> tags with an effective date, confining the answer to those blocks, naming an exact abstention string, demanding a citation per sentence, and dropping a suffix of the ranking are six decisions that meet it, and each fails differently. Nothing about the requirement selects among them.

Putting the rules in a system message rather than at the top of the user message is worth doing for a reason beyond tidiness. The system message is the instruction block a chat model receives ahead of the user's turn on every request, held separately from the conversation itself. The system block is identical on every query, so it sits at the front of a stable prefix and is eligible for prompt caching, which on a 400-token instruction block saves a small amount per query and, more usefully, keeps the instructions in a place where they cannot be displaced by a long context. The variable part — the context blocks and the question — goes in the user message, and the trace record persists the concatenation of both, because what you need later is the whole string the model saw and not the half your template owns.

Rerunning the ones you were not worried about#

A prompt change is global. It applies to every query the pipeline will ever serve, and the query it was written for is one of them, so the only way to know what it did is to run the queries you were not thinking about. That requires a baseline, which is a file with twenty questions, an expected answer and the chunk id that ought to support it, and a recorded run of the current pipeline against a pinned index.

{"qid": "e07", "q": "how many seats are included in the Team plan?", "expect": "10", "gold": "pricing/plans-current.md#0"}
{"qid": "e11", "q": "what is the SLA credit for a four-hour outage on Business?", "expect": "25%", "gold": "policies/sla-credits.md#800"}
{"qid": "e14", "q": "who approves a refund above €5,000?", "expect": "Head of Support", "gold": "policies/refunds-2024.pdf#1200"}

The gold field is what makes a case diagnosable rather than merely scoreable. A case recording only a question and an expected answer tells you that something failed; a case that also names the chunk which ought to have supplied the answer lets the attribution function from the previous chapter run automatically, since that function needs a gold chunk id to compare the retrieved set and the rendered prompt against. The seats case, e07, names pricing/plans-current.md#0, which is the chunk id that sat at rank 1 in the record that started this investigation — the match is what makes the eval case and the incident one failure rather than two similar ones.

Pinning matters more than it looks. Every change in this chapter is a change to the render stage, which means the index is untouched, which means retrieval is bit-for-bit identical between runs and any movement in the results is attributable to the prompt alone. That is a property worth protecting deliberately, and the runner enforces it by refusing to compare across index versions.

The argument for pinning is a subtraction, and it is the whole basis on which the table below can be read. Each run's outcome is a function of four things: the index, the retrieved set, the rendered prompt and the model. Hold the index fixed and retrieval is deterministic, so the retrieved set is identical between two runs and cancels exactly, and the model and temperature are fixed by configuration and cancel too. One term is left, and it is the render function. Let the index drift and nothing cancels: the difference is a sum of two changes with no way to split it, and the number at the end measures neither.

def run_eval(cases, index_version, render_fn, out_path):
    assert current_index_version() == index_version, "index changed; not comparable"
    rows = []
    for c in cases:
        trace = answer(c["q"], render_fn=render_fn)          # writes a full record
        rows.append({"qid": c["qid"],
                     "gold_rank": rank_of(trace, c["gold"]),
                     "abstained": trace["output"].strip() == "NOT_IN_CONTEXT",
                     "correct": c["expect"] in trace["output"],
                     "prompt_tokens": trace["prompt_tokens"]})
    write_jsonl(out_path, rows)
    return rows

def diff(before, after):
    b = {r["qid"]: r for r in before}
    return [(qid, b[qid]["correct"], a["correct"])
            for qid, a in ((r["qid"], r) for r in after)
            if b[qid]["correct"] != a["correct"]]

Applying the changes one at a time against the same pinned index produced this, starting from a baseline of 14 correct out of 20.

The three count columns close. On every row, correct plus wrong-and-confident plus abstained comes to twenty exactly, with no residual, because each question ends in one of those three states and there is no fourth state for it to be in. That closure licenses reading the table as transfers rather than as scores: a question leaving one column has to arrive in another, and the note on the right has to name it. Read that way, the first two changes are the same shape as each other — each moves one question out of wrong-and-confident into correct, leaves the abstained column at zero, and names exactly one query, e07 for the reordering and e03 for the block headers.

The fourth row is where the abstention first appears, and its three columns move by minus one, minus two and plus three. Two questions arrive in the abstained column from wrong-and-confident and one arrives from correct. The note names three queries — e11, e18 and e05 — and three is the whole of the movement, with nothing unaccounted for. A count that closes like that is the difference between a note you can act on and a note that might have omitted a fourth query nobody looked at.

The fifth row is the one to stare at, because it is identical to the third in all three columns: sixteen, four, zero. The difference between running with that abstention rule and running with none at all is zero in every column, across all twenty questions, so the rule sits in the instruction block on every query and changes no outcome. Written that way — abstain only if no block addresses it — the condition is satisfied by any block that so much as mentions the subject, and in a document collection organised by subject there is always such a block. The instruction reads the same on the question the documents answer and on the question they do not. A rule that does not vary across the cases it exists to separate is not a weak defence; it is not a defence.

The last row is the one worth shipping, and the reason is visible as a cancellation. Against the third row the correct column is unchanged at sixteen, so the whole difference sits in the other two columns, where two questions move from wrong-and-confident into abstained. The residual is one-directional: confident wrong answers became refusals and no correct answer was lost. Set the last row against the fourth instead and both leave the wrong-and-confident column at two, so both wordings reach the same two failures. The entire difference between them is e05, which means narrowing the condition from the model's confidence to the presence of the text recovered a correct answer at no cost in the column the change existed to move.

Five levers is thirty-two combinations, and at thirty seconds a run you could try all of them before lunch. Doing so would tell you which combination scored highest and nothing about why, which matters because the highest-scoring combination on twenty questions is frequently a combination that happens to suit two of them. Applying one change at a time costs six runs instead of thirty-two and produces a row per change that names the queries it moved, and the rows compose: the ordering change and the block-header change are independent, and their effects added exactly as the single-change rows predicted. Where they do not add, that is itself a finding worth recording, because an interaction means the two changes are addressing the same underlying failure by different routes.

The failures abstention converts and the one it cannot reach#

The abstention did not fix e11, the SLA credit query from the previous chapter, where the question asks what a four-hour outage on the Business plan is worth in service credits. That failure lives in retrieval, the correct chunk is at rank 34, and no instruction in the world puts text into a prompt that does not contain it. What the abstention changed is the shape of the failure: instead of a confident 10% sourced from a marketing page, the user gets NOT_IN_CONTEXT and escalates to a human. That is a genuine improvement in the cost of the failure and no improvement at all in its cause, and the two are easy to conflate when the eval only counts correct answers. An eval that scores abstentions as failures will reject this change; one that scores them as successes will accept a pipeline that abstains on everything. Counting them in a third column is the only version that stays honest.

Checking a citation against the chunk it names#

Citations did not fix groundedness either, groundedness being the property that every claim in the answer is actually supported by the text the model was given rather than by anything else. A model that is told to cite will cite, including on sentences its cited block does not support, which is why the check that matters is the programmatic one — take each cited chunk id, look up the text, and verify the claim appears. The eleven unsupported citations out of two hundred answers were all cases where the model merged two blocks into one sentence and attributed the result to one of them, and none of them would have been visible from reading the answers.

That is the third appearance of one defect in this chapter. A citation marker is present on a supported sentence and present on an unsupported one, so the marker is constant while the property it is meant to certify varies, and reading the answers cannot separate the two however carefully it is done. What breaks the tie is a second artefact — the chunk text — which does vary with the thing the marker does not.

def unsupported_citations(trace, chunks_by_id):
    bad = []
    for sentence in re.split(r"(?<=[.!?])\s+", trace["output"]):
        cited = re.findall(r"\[([^\]]+)\]", sentence)
        claim = re.sub(r"\[[^\]]+\]", "", sentence).strip()
        numbers = re.findall(r"\d[\d.,%]*", claim)
        if cited and numbers and not any(
                n in chunks_by_id.get(c, "") for c in cited for n in numbers):
            bad.append({"sentence": claim, "cited": cited})
    return bad

Checking only the numbers in a sentence is a deliberate narrowing, because a general entailment check — asking whether the cited text logically supports the sentence, in the sense that anyone reading the chunk would have to accept the claim — needs a second model and produces its own errors, while a figure that appears in the answer and in no cited chunk is unambiguous. On this corpus that narrow check found every one of the eleven, since the answers people complain about are almost always answers containing a wrong number.

The narrowing is a scope decision and should be recorded as one. The check has no opinion about a sentence containing no digits, so an unsupported claim phrased entirely in words passes it silently. That is an acceptable trade here because the population of complaints is numeric, and it stops being acceptable the moment the assistant is asked to summarise a policy rather than quote a figure from one.

None of these changes touched a vector. Every result in the table came from rerunning twenty queries against an index that never rebuilt, at about 30 seconds and 4 cents per full run, which is why a prompt change can be tried six times in an afternoon and reverted without ceremony. That cheapness is a property of the render stage specifically, and it does not transfer.

The two remaining confident wrongs, and e11's fate, both point at the retrieval side, where the equivalent experiment means re-chunking and re-embedding the corpus. Before spending 214 seconds and every vector on that, there is a prior question worth settling, because both of the moves the team is now considering — a similarity floor below which the pipeline refuses to answer, and a chunking change justified by the eval going up — rest on the assumption that the numbers involved mean what they appear to mean. The next chapter tests both assumptions, and neither survives in the form it was proposed.