The lexical control group

A retrieval failure arrives as one observation: an agent asked for ERR_TOKEN_STALE and got four pages that were not the definition. From inside a vector-only system that single observation is consistent with at least three completely different root causes, and nothing available on the dense side separates them.

The document may never have been indexed — an ingestion job skipped a docs/errors/ directory because of a glob that excluded files without front matter, and no vector exists. The document may be indexed but chunked badly, so that the error code sits in a table row the chunker split from the paragraph explaining it, and the chunk containing the code is 14 tokens of table syntax with no context. Or the document may be indexed and chunked correctly and the embedding simply does not place it near the query, which is the case examined earlier in this topic. The three call for entirely different work — fixing a glob, changing chunk boundaries, or changing retrieval strategy — and from the vector side they look identical, because the only signal available is "the right chunk was not in the top k", and that sentence is true in all three.

Without a second retriever over the same chunks there is no experiment available that distinguishes them, and the reason is worth stating as a property rather than as a complaint. A dense retriever always returns exactly k results. It returns k when the document is present and well placed, k when the document is present and badly placed, and k when the document was never ingested at all, because the structure it searches has no concept of a document it does not contain. So its output is the same shape in all three cases, and an observation that does not vary with the thing it is meant to identify carries no information about it. A term dictionary is different in exactly the way that matters: a missing string is a missing key, and a missing key is an observation.

One lexical query separates them#

A BM25 index over exactly the same chunks answers the question in a few seconds, because lexical retrieval fails for legible reasons. If the string is not in the term dictionary, the document is not there. If the string is there but the chunk containing it is tiny or context-free, you see that immediately in the retrieved text. If the string is there, the chunk is good, and BM25 ranks it first, then ingestion and chunking are both fine and the problem is entirely on the dense side.

Each branch is one command. The dictionary check is a terms enumeration; the source check is rg -n 'ERR_TOKEN_STALE' docs/; the rank check is the harness built earlier in this topic. Wrapping them makes the triage repeatable rather than a thing one person knows how to do:

def diagnose(query: str, literal: str, gold_id: str | None = None) -> str:
    if not terms_enum("body.exact", prefix=literal.lower()):
        if not grep_source(literal):
            return "NOT_INGESTED: no vector, no posting, no source hit"
        return "ANALYSER: string is in the source but not in the dictionary"

    hits = bm25_search(query, size=10, with_source=True)
    if gold_id and rank_of([h.id for h in hits], gold_id) is None:
        return "CHUNKING: term is indexed but the gold chunk does not rank"
    if hits and len(hits[0].text.split()) < 40:
        return "CHUNKING: top lexical hit is a fragment, not an explanation"
    return "EMBEDDING: ingestion and chunking are sound; dense retrieval missed"

The other thing a standing baseline gives you is a canary on ingestion. A nightly job that queries the lexical index for fifty known strings — one per documentation area — and alerts when any of them stops returning a hit will catch a broken glob, a failed sync, or a source repository that moved, on the night it happens rather than when someone complains a month later. The equivalent check against a vector index is unreliable, because a vector index always returns its k nearest neighbours and a missing document produces plausible-looking output rather than an empty result.

Nothing here needs to be in the serving path. The index can live on one machine, be rebuilt nightly, and be queried by three engineers and a CI job, and it will still have paid for itself the first time it converts "the model can't find things" into "the ingestion glob is wrong".

The number worth reporting#

Building a baseline invites a request for a metric, and the tempting one is BM25's own recall@10. That number is a floor and is worth knowing, but on its own it invites the argument that it is lower than the vector system's, which it usually is, and the conversation ends there.

The number to report is the lexical-recoverable miss rate: of the queries where the production vector retriever fails to return the gold chunk in its top 10, what fraction does BM25 return in its top 10.

vector_misses = [c for c in cases
                 if rank_of(vector_search(c.query, 10), c.gold_id) is None]
recovered = [c for c in vector_misses
             if rank_of(bm25_search(c.query, 10), c.gold_id) is not None]

print(f"vector misses: {len(vector_misses)}/{len(cases)}")
print(f"lexical-recoverable: {len(recovered)}/{len(vector_misses)} "
      f"= {len(recovered)/len(vector_misses):.0%}")

vector misses: 64/340
lexical-recoverable: 26/64 = 41%

That number is denominated in the failures people are actually complaining about, which is what makes it hard to argue with. Forty-one percent of current retrieval failures are recoverable by a component that costs 2.6 GB of RAM and 25 minutes to build. It also carries its own bad news honestly: the other 59 percent are not recoverable this way, and reporting both halves is what keeps the baseline from being oversold. Aggregate recall comparisons, as the previous chapter showed, move whenever the query mix moves; the recoverable-miss rate is computed on a fixed set of observed failures and is stable in the way a diagnostic should be.

The denominator in that output can be checked against the previous chapter's table, and it closes. Three hundred and forty labelled queries, at the recall@10 the tuned graph reached there, leave a little under a fifth of them unfound — and a little under a fifth of 340 is 64, which is the miss count printed here. Two measurements taken for different purposes, describing the same run.

Which makes the recovered figure convertible into the currency that comparison was argued in. Twenty-six of the 340 queries would come back from a union of the two result lists and do not come back now, and 26 out of 340 is between seven and eight points of aggregate recall added on top of what the vector system reaches alone. Set that against the two other levers already measured. Raising ef_search across its whole useful range bought seven points and tripled latency. An exhaustive scan — the most any amount of vector work can reach on these embeddings — sat two points above the tuned graph. The lexical union buys more than either, and it goes past the exhaustive ceiling, which no reranker and no ef_search value can do. BM25 is not the better retriever here. The ceiling was a property of where the embedding model put things, and a term match is not looking at where the embedding model put things.

The ordered day#

Given a vector-only RAG — retrieval-augmented generation — over internal engineering docs, complaints from coding agents about error codes and service names, and one day:

1. Collect the failures first, for 30 minutes. Pull 20 to 40 real failing queries out of the logs or from the complaining team. Everything after this is measured against that list, and building the list is the step people skip in favour of theorising about embedding quality.
2. Confirm the documents exist, for 20 minutes. rg the source repository for each literal string. Any query whose answer is not in the source corpus is an ingestion or coverage problem and leaves the retrieval conversation entirely.
3. Stand up BM25 over the identical chunks, for about an hour. Same chunk ids, same boundaries, so that ranks are comparable. This does not require a cluster:

import sqlite3

db = sqlite3.connect("baseline.db")
db.execute("""CREATE VIRTUAL TABLE chunks USING fts5(
                 chunk_id UNINDEXED, body, tokenize='unicode61 tokenchars=_-.')""")
db.executemany("INSERT INTO chunks VALUES (?, ?)", load_chunks())

def bm25_search(q: str, size: int = 10) -> list[str]:
    rows = db.execute(
        "SELECT chunk_id FROM chunks WHERE chunks MATCH ? "
        "ORDER BY bm25(chunks) LIMIT ?", (q, size)).fetchall()
    return [r[0] for r in rows]

SQLite's FTS5, the full-text search extension compiled into the library itself, implements BM25 with the same k1 and b semantics, and tokenchars=_-. is the setting that keeps ERR_TOKEN_STALE, pricing-svc-v3 and cache_ttl_seconds intact — the same analyser decision that the second chapter of this topic was about, in a different dialect. 4. Verify the terms are really there, for 15 minutes. Dump the terms for one document you know contains an error code. If the identifiers are not in the dictionary, fix the tokeniser settings before measuring anything, or you will measure your own misconfiguration. 5. Run the head-to-head and record ranks, for an hour. Both retrievers, the failing queries, ranks in a table. This produces the recoverable-miss rate. 6. Classify the residue, for an hour. Run diagnose over the queries BM25 also missed. Expect a mix of not-ingested and badly-chunked, and expect at least one genuine surprise. 7. Ship the narrowest fix the evidence supports, for two hours. Usually that is querying both indexes and combining their result lists, or routing identifier-shaped queries — anything with an underscore, an internal hyphenated service name, or a leading uppercase run — to the lexical index first. Combining ranked lists properly is its own subject; the point today is that a two-index retrieval step is a small change to one function. 8. Freeze the cases as tests, for an hour. The standing cases from earlier in this topic, so that the next chunker change has to explain itself.

Adding the eight durations up is worth doing before committing to the day, and they come to a little over seven hours — which is a working day with nothing padded and nothing hidden. The shape of the sum is the more interesting part. Steps one through six are five hours of measurement, step seven is two hours of change, and step eight is an hour of making the change durable. Most retrieval work is proposed the other way round, with the change first and the measurement deferred until somebody asks whether it helped. The reason for the order here is that six of these steps are cheap and one of them is irreversible, and the cheap ones determine which version of the irreversible one is correct.

What this plan deliberately does not start with is re-embedding on a larger model. That move costs a full pass over 4.4 million chunks, an index rebuild of six hours, and a week of evaluation to establish whether anything improved, and it does not address the reported symptom: an identifier fails because the embedding tokeniser shatters it into subwords and cosine similarity ranks by topic, and a bigger model built the same way has the same two properties. Teams reach for it because it is the lever with a version number on it, which makes it feel like progress that can be scheduled.

Line it up against the three causes and the objection sharpens into something checkable. If the document was never ingested, a larger model embeds the same absent text and changes nothing. If the chunk boundary split the code from its meaning, a larger model embeds the same fragment and changes nothing. Only the third branch is one where a different model could conceivably help, and the earlier chapters of this topic argue that it does not help there either, because the tokeniser and the training objective are the same at every size. So the proposal is being made without knowing which of three states the system is in, and it is inert in at least two of them. A day of triage costs less than the evaluation week that would follow the rebuild, and it tells you which branch you are on before anything is spent.

Three mechanisms a term match cannot bridge#

The rows divide into three mechanisms. The first three are vocabulary failures: the user's words and the document's words do not overlap, either because they are paraphrases, because the organisation uses two names for one thing, or because the string was typed slightly wrong. Nothing in a term-matching system can bridge a gap where the shared term does not exist, and the available remedies are dense retrieval, a curated synonym list, or fuzzy matching with an edit distance of one or two — the last of which is cheap and worth enabling on identifier fields precisely because misremembered codes are common. The fourth and sixth rows are not retrieval failures at all; they are questions whose answers do not exist in any single chunk, and improving retrieval of any kind cannot manufacture a chunk that was never written. The fifth is a discrimination failure, where the term is present everywhere and rarity provides no signal, and the fix is metadata filtering rather than better ranking. That row is the identifier argument of this topic run backwards, and the same one-lookup test predicts it. A rare identifier wins because its document frequency is far below the point at which its weight drops to what a common word can reach — around a thousand documents on this corpus. 404 appears in 380,000, which is more than two orders of magnitude past that boundary, so its weight is comparable to an ordinary English word and the query has nothing to discriminate with. The same df check that promises a win on ERR_TOKEN_STALE refuses to promise one here, before anybody runs the query.

Those rows are not caveats appended for balance; they are half the deliverable. A team that installs a lexical baseline and reports only the 41 percent it recovers will be asked, correctly, why the other 59 percent persist, and the answer needs to be ready and specific. Dense retrieval owns the paraphrase and synonym cases, chunking owns the multi-chunk answers, and misspelled identifiers need fuzzy matching or query expansion, which is neither of the two retrievers discussed here.

The availability argument — that a lexical index is a useful fallback when the vector database is down — is true and is not the reason to build one. It earns its keep by making failures separable, by fixing a floor that the production system must clear, and by giving every future retrieval change a control to be measured against. None of those three require it to be in front of a user, which is why the cheapest version of this — an FTS5 file on one machine, rebuilt nightly — is often the right one to start with.

Which leaves the obvious question of what to do with two ranked lists once you have them, since routing by query shape is a heuristic and running both retrievers on every query produces two orderings with incomparable scores.