A fixture that reproduces the defect beats an anecdote about it

The thread starts with a screenshot. Someone from platform support pastes an answer the internal assistant gave to "how do I rotate the signing key?", and it reads well: five sentences, confident tone, two citations, no invented API. Underneath, a reply says the answer is wrong because it skips the step where you drain the token cache first, and if you follow it as written you sign new tokens against a key the verifiers have not picked up yet. By the time forty messages have accumulated, three fixes have been proposed — a bigger embedding model, a better system prompt, and moving to a different vector database — and nobody in the thread can say which chunks were in the context window when that answer was generated.

Read that thread as an evidence problem and its shape is easy to state. An answer from a system of this kind is produced from two inputs: the question, and a set of passages the system selected and pasted into the prompt above it. The screenshot records the question and it records the output. It records nothing whatsoever about the second input. Every claim in the thread about why the answer was wrong is therefore a claim about a quantity nobody has seen, and a claim about an unseen quantity cannot be checked against anything. That is the reason three fixes can sit side by side for forty messages without one of them being eliminated: all three are consistent with the evidence, because the evidence does not constrain them.

The assistant in that thread is a retrieval-augmented generation system, RAG for short, and the shape of it is worth stating before anyone argues about which part of it broke. A language model knows what was in its training data, which does not include this company's runbooks, this quarter's expenses policy, or anything written after the model was frozen, and retraining a model every time somebody edits a wiki page is not something any team can afford. So the system does its reading at question time. Every document in the corpus has been cut in advance into passages of a few hundred words, called chunks, and each chunk is stored with a numeric summary of its meaning that lets the store be searched by similarity rather than by keyword. When a question arrives, the system pulls back the chunks that look most like it — the number it asks for is conventionally called k — pastes their text into the prompt above the user's question, and asks the model to answer from that text. The passages that get pasted in are the context for that answer. Products built this way are ordinary now: the internal assistant over a wiki, the documentation search that replies in sentences, the support bot that cites a help-centre article.

An anti-pattern is a design choice that looks reasonable, gets copied widely, and reliably degrades the result, which is what makes it worth naming and cataloguing rather than rediscovering. The RAG list is short and specific: chunks cut too small to stand on their own, a k of twenty with all twenty pasted into the prompt, no step that reorders candidates before they reach the model, an ingest that embeds every file on the shared drive, similarity search running with no keyword search beside it, and a system shipped with none of it measured. All three fixes proposed in that thread assume the defect lies somewhere other than on that list, and no one in the thread can tell them apart while the context stays invisible.

That is the state most RAG debugging starts from, and it is why the first thing to build is not a fix. It is a fixture: a corpus small enough to hold in your head, a query set that does not change between runs, and a set of switches that let you turn each suspected defect on and off. The point of the fixture is to make the failure happen on demand, in under a minute, with the internals visible. Until the defect reproduces, every proposal in the thread is equally plausible, which means the team will pick whichever one the loudest person favours.

The belief being corrected here is the reasonable one, and it is reasonable because for most software it is simply true. A bug report carrying the input and the wrong output is a reproduction, whenever the input it carries is the whole input. Here it is not. The input to a generated answer is the assembled context, and the context is a function of three things: the query, every configuration setting in force at that moment, and the contents of the index. None of the three appears in a screenshot, and two of them change without anybody editing code — a re-ingest changes the index, a library upgrade changes a default. So a report of this shape pins the output and leaves the input free, and evidence that leaves the input free is the one shape of evidence that cannot falsify a proposal.

What goes in the corpus, and why it is small#

A useful fixture corpus is not a sample of production. It is a construction, sized so that you can enumerate the documents by hand, and shaped so that every failure mode you want to study has something to bite on. Two hundred to four hundred documents is enough. The version we use through this topic has 312 documents and about 1.1 million tokens, assembled from a real internal wiki with four deliberate properties.

It contains multi-step procedures that are longer than any reasonable chunk — the key rotation runbook is 2,100 tokens across eleven numbered steps, and step 4 is the token-cache drain that the bad answer omitted. It contains identifiers that carry no meaning to an embedding model, including a TLS troubleshooting runbook whose title is the literal string E_2231_TLSHANDSHAKE. It contains three generations of the same expenses policy, two of them superseded, with nothing in the body text saying so. And it contains roughly ninety documents that no reasonable person would want in an answer: meeting invites, a spreadsheet of desk assignments, an off-site retrospective, and the department's collection of memes about the migration to the new identity provider.

Those four properties and the size are different kinds of thing, and keeping them apart is what stops the fixture from being redesigned every time somebody disagrees with it. The size is a decision. It is taken so that one person can hold the document list in their head and say, without checking, whether a retrieved chunk had any business being retrieved; nothing about the failures themselves requires a small corpus, and a larger one would exhibit every one of them. The four kinds of content are requirements. Each anti-pattern needs something to bite on, and a corpus with no long procedure in it cannot exhibit a procedure cut in half, in the same way that a corpus with no meaningless identifiers in it cannot exhibit a dense retriever failing on one. A decision can be reopened by argument. A requirement can only be met or not met.

The query set is forty questions with a known answer and, for each, the identifiers of the chunks that actually contain that answer. Those hand-checked identifiers are the gold set for a query, borrowing the term from information retrieval, where the gold or ground-truth labels are what every measurement is scored against. Writing those labels is the expensive part, and it is worth spending an afternoon on, because every measurement in this topic depends on knowing what should have been retrieved. Forty is chosen to be small enough that one person can label it and large enough that a class of query has eight to ten members, which matters later when a result comes back ambiguous. The classes are the ones the corpus was built to stress: multi-step procedural, exact identifier, superseded-document, single-fact lookup, and cross-document synthesis.

The size of the query set, the number of classes and the members per class close on each other, which is why forty is the size rather than a round number somebody liked. Five classes at eight members apiece is forty exactly, with nothing left over and nothing borrowed. The stated range of eight to ten is what the same total looks like once a class you care about more is given extra members, and every extra member has to come out of another class, so the per-class counts are not free of one another. What the closure buys is a floor: no class is represented by one or two examples, so a per-class result rests on at least eight observations rather than on an anecdote of a different kind. What it does not buy is resolution. At eight observations a single query changing its verdict moves that class by more than a tenth, so a per-class difference smaller than that is inside the noise of the instrument and should not be reported as a finding.

The labels are attached to documents and character spans rather than to chunks, because chunk identifiers change the moment you move the chunk size, and a query set whose ground truth is invalidated by the experiment you want to run is worthless. Label the span in ops/key-rotation.md that contains the drain step, then resolve spans to chunk identifiers at run time for whatever chunking the current configuration produced. Pull the queries themselves from real support tickets and search logs where you can, lightly rewritten, since invented questions drift toward the phrasing your documentation already uses and quietly make retrieval look better than it is.

The switches are the anti-patterns#

The fixture's configuration exposes one flag per defect, and each defect is a real setting that a real system somewhere is running in production right now, usually because it was the default. Three of the switches carry vocabulary the table below uses without ceremony. Overlap is how many tokens each chunk repeats from the end of the chunk before it, so that a sentence cut at a boundary survives in at least one of them. A reranker is a second, slower model that runs after retrieval and scores the candidates again, in order to reorder them before any of them reach the prompt. And retrieval itself comes in two families: dense retrieval compares the question's numeric summary against each chunk's and ranks by how close they are, while lexical retrieval, BM25 being the standard implementation, ranks by the words the two actually share.

Read that table across rather than down. Each row is a pair, and the pair is the measurement. What a fixture reports is never the absolute quality of the setting in the second column; it is the distance between the second column and the third, with every other row held still. A level taken from the second column on its own is a number about 312 documents and forty questions, and that is not a fact about anybody's production system. The gap between the two columns is a fact about the mechanism named in the fourth, and a mechanism travels to other corpora where a level does not.

Each switch is independent, which is the property that makes the fixture worth building rather than just editing production config. In a live system these settings are entangled with each other and with everything else that has changed since the last deploy, so a comparison takes a week and still cannot attribute the result. Here a full sweep of one axis takes a couple of minutes.

from dataclasses import dataclass

@dataclass(frozen=True)
class Config:
    chunk_tokens: int = 128
    chunk_overlap: int = 0
    k: int = 20
    rerank: bool = False
    retrieval: str = "dense"          # "dense" | "bm25" | "hybrid"
    ingest_filter: str = "none"       # "none" | "scope" | "scope+status"

BASELINE = Config()                               # every defect on
CANDIDATE = Config(chunk_tokens=512, chunk_overlap=64,
                   k=20, rerank=True, retrieval="hybrid",
                   ingest_filter="scope+status")

The two named configurations are the ends of the range rather than the two options. BASELINE has every defect switched on at once and CANDIDATE has every one switched off, and neither is what you run when you want to learn something: a comparison between them moves five settings together and tells you only that five settings matter. What they are for is bracketing. Anything that does not change between those two ends is not a defect on this list, and a per-axis sweep between them is where the attribution comes from. The one figure that governs how freely you can sweep is the cost of rebuilding the index, and that scales with the corpus token count rather than the document count, since re-embedding is charged per token and this corpus is about 1.1 million of them. Doubling the number of documents while halving their length would cost the same.

Inspect the context, not the answer#

The habit that separates a fixture that teaches you something from one that just runs is looking at what entered the context window, every time, before reading the answer. A generated answer is a lossy summary of the context, and it is lossy in exactly the direction that hides the problem: fluent, plausible, and silent about the eighteen chunks it ignored and the one it never saw.

The discipline is worth its cost because the properties you would naturally read an answer for are the properties that do not vary with the thing you are trying to measure. Fluency is produced by the generator, and the generator is fluent whether it was handed three relevant chunks or twenty. Citation is produced by the same step, and the answer in that screenshot carried two citations which both resolved to real documents. Tone, length and formatting are the same across the whole range. So the visible surface of an answer stays approximately constant while evidential quality moves from complete to almost absent, which makes that surface useless as a signal about evidential quality — and worse than useless, because a fluent answer with citations reads as reassurance. Reading the input side is the only way to see a quantity that actually moves.

So the fixture's run function returns a manifest before it returns text.

def run(query: str, cfg: Config) -> Trace:
    candidates = retrieve(query, cfg)          # k candidates, ranked
    context = assemble(candidates, cfg)        # what actually goes in the prompt
    return Trace(query=query, config=cfg, context=context,
                 answer=generate(query, context))

def manifest(t: Trace, gold: set[str]) -> None:
    total = sum(c.tokens for c in t.context)
    for i, c in enumerate(t.context, 1):
        mark = "GOLD" if c.id in gold else "    "
        print(f"{i:>3} {mark} {c.score:.3f} {c.tokens:>4}t "
              f"{c.doc_id}#{c.chunk_ix}  {c.text[:52]!r}")
    print(f"    context: {len(t.context)} chunks, {total} tokens, "
          f"{sum(1 for c in t.context if c.id in gold)}/{len(gold)} gold")

Two details of that output are doing more work than they look. The mark is computed by set membership against the gold identifiers rather than from anything the retriever reported, which keeps the ground truth independent of the system under test. And the summary line ends with two counts separated by a slash, which is the pair the whole topic turns on: how much of the gold arrived, against how much of it exists.

Here is what that prints for the key rotation query under the baseline configuration, abbreviated to the rows that matter.

  1      0.834  128t  ops/key-rotation#7   'Step 8. Update the JWKS endpoint and wait for…'
  2      0.831  128t  ops/key-rotation#3   'Step 4. Drain the token cache before issuing…'
  3      0.822  126t  sec/key-policy-v2#1  'Signing keys must be rotated at least every 9…'
  4      0.819  128t  ops/tls-renewal#2    'Step 3. Rotate the intermediate certificate a…'
  5      0.817  128t  eng/onboarding#12    'You will be issued a signing key for the stag…'
 ...
 19      0.786  128t  minutes/2024-03-14#4 'Anna raised the key rotation runbook being ou…'
 20      0.784  127t  memes/rotation.md#0  'when you rotate the signing key on a friday'
      context: 20 chunks, 2534 tokens, 3/11 gold

Three of the eleven gold chunks arrived. The remaining eight steps of the procedure are in the index, correctly embedded, and were beaten to the top twenty by a meme, a set of meeting minutes, and a TLS runbook that happens to talk about rotating something. The answer generated from this context omitted the drain step because the drain step ranked second and the model, reading twenty chunks of which seventeen were about key rotation in some loose sense, reconstructed a plausible procedure rather than the actual one. Nothing about the answer text reveals any of this, which is the reason the Slack thread went forty messages without converging.

A meme and a set of meeting minutes beating eight gold chunks into the list reads like an ingest problem, and the excerpt says it is not. Both of those rows sit at the bottom, at ranks nineteen and twenty, and that placement separates two switches a reader will otherwise merge. Roughly ninety of the 312 documents are junk, close to a third of the corpus by count, and ingest_filter is the switch that removes them — but turning it on would not have improved this query. The context is twenty chunks because k is twenty, and it stays twenty chunks whatever is missing from the index: the two junk rows would simply have been replaced by whatever sat at ranks twenty-one and twenty-two, which on this corpus is more material of the same kind as ranks three, four and five. The context is not thin because junk got in. It is thin because near-misses did — a signing-key policy, a TLS renewal runbook, an onboarding page, every one of them a document you would want in the index and none of them containing the procedure that was asked for. Those are precisely the rows an ingest filter has no opinion about.

Look also at the score column, which spans 0.834 down to 0.784 across all twenty rows. That score is cosine similarity, the measure a dense retriever ranks by: it reads 1.0 when the question's numeric summary points in exactly the same direction as the chunk's and falls towards zero as the two diverge. Fifty thousandths of it separates the top of the list from the meme at the bottom, and that compression is not an artefact of this corpus. Embedding models place everything that shares a topic into a narrow band, so once the candidate list is longer than the handful of genuinely on-point chunks, the ordering inside it carries very little information. A threshold set anywhere in that band would either admit all twenty or reject all twenty, which is worth remembering when someone proposes fixing precision by filtering on score.

That compression is the clearest instance in this chapter of a shape worth learning to recognise, and it is the shape most bad defences have. Relevance across those twenty rows runs the entire range it can run: from a numbered step of the exact procedure that was asked for, to a joke about doing key rotations on a Friday. Similarity across the same twenty rows moves by five hundredths. So the quantity being tested is very nearly constant over precisely the interval where the decision has to be taken, while the property the decision is about varies from one end of its scale to the other. A score threshold is therefore not a weak defence against irrelevant chunks; it is no defence at all. What makes it tempting is that every part of it is sound in isolation — the score is a real measurement, it is computed correctly, and it does order the list in roughly the right direction. It simply has no resolution left at the scale the cut has to be made.

The last line of the manifest is the one to watch. Twenty chunks, 2,534 tokens, three of eleven gold: the context is simultaneously too large and too thin. That combination — high token spend and low signal — is the shape almost every anti-pattern in this topic produces, and it is invisible from the output side.

Three of the numbers on that line close on each other, and checking them is faster than trusting them. Twenty chunks at the configured chunk size of 128 tokens gives an upper bound of two thousand five hundred and sixty tokens, and the printed total sits a little under it. The shortfall is not slack in the accounting — it is the chunks that ran out of document before they ran out of budget, two of which are visible in the excerpt at a hundred and twenty-six and a hundred and twenty-seven tokens, with the rest among the ranks the excerpt elides. The counts close too: three gold chunks and seventeen others make twenty with no residual, which is why the fraction of this context that was worth sending needs no second instrument to compute. The manifest line already contains it. What that line cannot give you is the other direction. Three of eleven is a statement about this query's gold set, and the eight labelled chunks that are missing were never in a candidate list this output shows, so nothing printed here can say how close they came.

Benchmarks, production traces and frameworks#

A fixture is not a benchmark. The forty queries are chosen to stress specific failures, so the numbers it produces are not an estimate of production quality and should never be quoted as one. If the fixture says context precision rose from 0.15 to 0.62, that is evidence about a mechanism, and the size of the effect on real traffic is a separate question with a separate measurement.

It is not a replacement for production traces. A fixture reproduces the defects you thought to build switches for, and the ones that hurt most in practice are usually the ones nobody wrote down — a similarity threshold in a library default, a query rewriter that drops quoted strings, an ingest job that has been silently failing on PDFs since March. Chapter 8 goes looking for those in a real trace, and the fixture is where you reproduce whatever you find so you can work on it.

It is also not an evaluation framework, and the choice of one does not matter here. The metrics this topic uses — recall@k, context precision, groundedness — were defined in topic 9.10, and the fixture computes them from the manifest with about thirty lines of code. Adopting a framework is fine; adopting one before you can produce a manifest like the one above is how teams end up with a dashboard full of numbers that all look healthy while users keep complaining.

There is a comparison worth making across all three of those, and it needs its limit stated with it. A fixture stands to a retrieval defect roughly as a failing unit test stands to a logic bug: it pins the failure to a command anybody can run, it turns an argument into an experiment, and it goes on failing until the cause is actually removed. The analogy breaks at the verdict. A unit test asserts on an exact value and comes back green or red, whereas a fixture returns a metric over forty queries that moves by tenths, and no threshold in it decides anything on its own. Somebody still has to judge whether a change is an improvement, and a fixture treated as a pass/fail gate will be tuned until it passes.

A fixture, then, is a device for converting claims about a retrieval system into claims that can be shown false. Everything in its design serves that: the corpus is small so that a human can adjudicate any row of the manifest, the query set is frozen so that two runs differ only where you made them differ, the labels are attached to spans so that the experiment you most want to run does not destroy its own ground truth, and the switches are independent so that a difference can be attributed to one of them. The manifest is the part that does the work, and it does it by putting the input side of the request on the screen next to the output side.

The next thing to do with it is uncomfortable. Run the baseline configuration across all forty queries, compute the metrics the team already has on its dashboard, and recall@20 comes out at 0.94 with groundedness at 0.91, on the same system that produced the context above. Both numbers are correct, and neither of them can see that seventeen of twenty chunks should not have been there.