Lexical search

Read these first

• The retrieval problem

What you will know

• Implement BM25 search over a corpus, or configure it correctly

  BM25 is a strong, cheap baseline that most vector-first systems never bother to measure against.

• Beat a naive vector search with BM25 on identifiers and error codes

  Identifiers, error codes and product names are where lexical wins outright and embeddings fail.

• Trace a query through an inverted index and measure the postings examined before ranking results

  Term dictionaries, postings lists, and skip structures determine latency, while analysis choices determine whether exact matches survive indexing.

• Construct three query cases where lexical retrieval ranks the correct document above a naive vector search, and record why

  Rare strings, exact phrases, and code-like tokens expose semantic similarity's tendency to blur distinctions that users need preserved.

• Compare lexical-first and vector-first retrieval on the same corpus using recall, latency, and index-size measurements

  The comparison reveals whether semantic retrieval adds enough relevant coverage to justify its operational and storage cost.

Chapters

1. 1

   Inverted indexes and postings lists

   A lexical engine answers a query with one dictionary lookup per term and a walk over the postings lists those lookups point at, so the cost is set by the document frequencies of the words typed rather than by how many documents exist. Count the postings examined by a linear merge and by a skipping intersection on the same two-term query, read the same counts back out of an engine profiler, and see why a query pairing one very common word with one rare word is the cheapest shape of the three measured here.

   Read chapter
2. 2

   Analysers and keyword fields

   Tokenisation, lowercasing, stemming and stopword removal all run before anything is stored, so a part number such as A-100-B is split at the hyphens into fragments that match nearly every document and no scoring change can promote a document the engine never considered. Configure the chain per field using keyword sub-fields, narrower token filters and character filters, then prove with a term dump that the exact string you expect is genuinely in the index rather than assuming it from the mapping.

   Read chapter
3. 3

   TF-IDF term weighting

   Term frequency counts how often a term occurs in one document, document frequency counts how many documents contain it at all, and inverse document frequency turns the second into a weight, so a token in one document out of millions contributes enormously while a token in most of them contributes almost nothing. Work the logarithm and the product by hand on real corpus statistics, then read off the two properties the score lacks — an unbounded term count and no awareness of document length — because those are precisely what BM25 changes and IDF is what it carries over untouched.

   Read chapter
4. 4

   BM25 saturation and length normalisation

   BM25 changes two things about the term-frequency half of TF-IDF and leaves inverse document frequency alone. The k1 curve makes repeated occurrences saturate, so twenty mentions beat five by seventeen percent rather than by four times; the b parameter scales the score by document length, from b=0 ignoring it to b=1 normalising fully. Work both curves numerically, sweep b against a labelled query set instead of accepting 0.75, and see why chunking to a narrow length band makes the parameter stop mattering at all.

   Read chapter
5. 5

   Rare identifiers and error codes

   A token appearing in one chunk out of 4.4 million carries an inverse document frequency of 14.89, which no combination of ordinary vocabulary can reach, so BM25 returns that one chunk first while a dense retriever offers four plausible network pages and never the definition. Work the ceiling arithmetic term by term, then build the twelve-row harness that records the win as a measured rank and an MRR rather than as an anecdote — and read the three conditions under which the advantage disappears.

   Read chapter
6. 6

   Phrase queries and code-like tokens

   Two query classes beyond rare identifiers reliably favour lexical retrieval: order-bearing phrases, where shuffling a sentence moves its embedding by four hundredths of a cosine point and an inverted statement outranks the true one, and code-like tokens, where sibling configuration keys cluster within 0.03 cosine of the correct answer. Measure your own model's order-insensitivity in twenty lines, learn the match_phrase and exact-term constructs that win each class, and keep the cases as a test rather than a demonstration.

   Read chapter
7. 7

   Dense retrieval and HNSW

   Embed every chunk into a fixed-dimension vector, embed the query the same way, return the nearest by cosine — then discover that an exhaustive scan over 4.4 million 1024-dimension vectors reads 18 GB per query and has to be replaced by an approximate index. HNSW searches a navigable graph by hopping from neighbour to neighbour, so the next address is unknown until the current node has been read, which is why the graph must be resident in memory while a postings list can stream off disk. Includes the per-element memory arithmetic and the parameters that move recall, latency and build time.

   Read chapter
8. 8

   Recall, latency and index size

   Running a lexical and a dense retriever over one chunk set produces three numbers that answer different questions and trade against each other. Recall and latency are tunable at query time through ef_search; index size is fixed by the model dimension the moment you choose it. Read the p50, p95 and p99 columns separately, break recall down by query class before quoting an aggregate, and know the two ways the benchmark lies — a judgment set pooled from one system, and latency measured on a warm process.

   Read chapter
9. 9

   The lexical control group

   A vector miss is consistent with three unrelated causes — a document never ingested, a chunk boundary that split the answer, or an embedding that placed it badly — and from the dense side all three produce the identical observation. A BM25 index over the same chunks separates them in seconds, because absence is legible in a term dictionary and invisible in a similarity ranking. Includes the triage function, the ordered one-day plan, the lexical-recoverable miss rate worth reporting, and the failing queries lexical retrieval cannot rescue.

   Read chapter