Six months of drift with no deploys is a maintenance failure, not corpus growth

Diagnosing drift is a different exercise from diagnosing a failure, and it is harder for a reason worth stating up front: there is no event to anchor it to. A retrieval service that has not been deployed in months, whose embedding model is pinned, whose chunker and prompts are untouched, still answers every request, still returns k results — k being the number of chunks each query retrieves and hands downstream — and still passes every operational check. What has changed is which chunks those are. Documents that used to arrive inside the top ten now arrive below it, the reranker that rescores the retrieved candidates before any of them reach the prompt only ever sees what retrieval handed it, and a document that was never retrieved cannot be rescued by rescoring.

The subject of this chapter is a corpus that had grown, over that same period, from 5.2 million chunks to 6.3 million, and an index over it whose measured recall against exact search had fallen. Recall against exact search is the fraction of the true nearest neighbours — the ones found by scoring the query against every stored vector, with no graph involved — that the approximate index also returned. That measurement matters here because it is the one number in the investigation that cannot be argued with: it compares the index against the correct answer over the index's own contents. Anyone running a vector index for longer than a year will meet this situation, and the cost of answering it in the wrong order is a quarter spent on the wrong remedy while the real one keeps compounding.

Five checks, in an order that is not arbitrary#

The order runs from mechanical to semantic. The first four checks ask whether the index still faithfully represents the corpus it was built from and still has good internal structure — whether what is stored matches what the primary store says should be stored, and whether the graph that connects it still routes a search to the right region. The fifth asks a different kind of question: whether the corpus itself changed in a way that changes what good retrieval means, so that an index doing exactly what it was built to do now returns a worse top ten than it did.

Running them the other way round produces the answer "the corpus grew and competition for the top ten increased", and that answer has a property which makes it dangerous to reach for first. It is always partly true. Every corpus that grows does acquire more documents competing for the same slots, so no observation contradicts it and no amount of evidence tests it. A diagnosis that starts there also stops there, because there is nothing left for it to do except agree with itself.

The first four earn their place at the front for a different property: they are cheap and they are falsifiable. Each returns a number in minutes, each has a value that would clear it, and each can therefore come back clean in a way that means something. Eliminating four falsifiable causes is what converts the composition hypothesis from a shrug into a claim with a measurable residue — once the mechanical faults are fixed and the relevance loss is re-measured, whatever gap remains is the part composition has to account for, and its size is the evidence that was missing.

There is a shorter route past all five, and it deserves naming before the table rather than after it, because the reason it survives is that it works. Raising k from 10 to 30 recovers a document that has slipped out of the top ten. It recovers that document whether the slip came from tombstoned vectors crowding the candidate heap, from an orphaned copy of the same document outranking it, from a degraded graph never entering its neighbourhood at all, or from long chunks from a new source filling the ranks above it. That is a remedy which is constant when the thing it must discriminate varies, and a constant response carries no information: it returns "fixed" for all four causes, so its returning "fixed" says nothing about which one you have.

The temptation is real and should be granted rather than dismissed. It works on the reported examples, it closes the ticket the same afternoon, and its whole cost is context and latency — 20 extra chunks of context per request and a measurable amount of reranker time. What makes it dangerous is that the four causes above do not stop when the symptom does. Whatever moved those documents keeps moving them, so the same ticket returns in four months with a proposal to raise k to 60, and the second proposal is harder to argue with than the first because the first one worked.

Read that table downwards as a chain rather than across as a list, because each instrument in it is blind to exactly the thing the next one measures. A tombstone gauge — a tombstone being a vector marked deleted, still stored and still traversed — cannot see an orphan, because an orphan was never deleted. That is chapter 2's result, where a shortened revision leaves its trailing chunks behind: the writer upserted the ids the new revision produced, overwrote what it could, and returned success, and no operation anywhere recorded a deletion for a gauge to count.

An orphan audit against the primary store cannot see a degraded graph, for the mirror-image reason. The audit works by asking, of a sample of stored vectors, whether the primary store still holds a row referencing each one, and a badly wired vector is referenced perfectly well. It has a live row, a current revision and a correct embedding, which is precisely the condition of every vector the audit clears — an orphan is a live vector by every definition the engine has, and so is a vector the graph can no longer reach. An instrument that sorts vectors into referenced and unreferenced has no column in which the quality of the wiring could appear.

And a recall-against-exact measurement cannot see a corpus whose composition changed, because both sides of that comparison retrieve from the same stored vectors. Exact search and the approximate index are ranking the same 6.3 million chunks with the same model and the same distance function, so a change in what those chunks are enters both sides equally and cancels out of their disagreement. This is why the sequence is a sequence and not a checklist that can be worked in any order, and it licenses something specific about a clean result: a clean result on check n is informative only because check n+1 exists. Check 1 coming back low means "not tombstones" only because something after it will go looking for orphans.

Check 1 came back at 9%, which is real but unremarkable for a corpus with modest deletion, and the monthly delete volume had not changed. A clean result is a result, and the way to cash it in is to work out what it rules out. Chapter 3, where tombstoned vectors keep spending search budget, gives the relation: because dead nodes occupy slots in the candidate heap and only survivors can become results, the effective search width is ef_search × (1 − tombstone_ratio) — ef_search being the number of candidates a search keeps in flight. Substitute nine hundredths for the ratio and the effective width is ef_search times ninety-one hundredths, which is barely reduced.

A reduction that small cannot account for what was reported. Losing under a tenth of the candidate slots perturbs the tail of a result list; it does not push a document from inside the top ten out to rank 17, which is a fall of seven ranks. Tombstones at that level are a contributor rather than the cause, and the difference between those two words is the whole value of the check. Had check 1 come back at 30%, the arithmetic would have said the opposite, the investigation would have stopped there, and the whole thing would have become a compaction exercise — compaction being the pass that physically drops tombstoned nodes and rewires their neighbours. A check that rules something out is doing the same job as a check that finds something, since both narrow the set of causes still standing, and this is the clearest example of it in the topic.

Check 2 found 2.7% of chunks unreferenced by the primary store, and the distribution was the informative part. The orphans were not spread evenly; they clustered in the several thousand documents that had been edited more than three times, which are the release notes and policy pages — the documents people search for most. Concentration is what made the same count worse. The same number of orphans spread thinly across rarely queried documents costs storage and nothing else, while a cluster of them sitting on the most-searched documents costs a top-ten slot on the queries that run most often.

The harm follows from what an orphan is. It is a near-duplicate of live content by construction, because it holds a previous revision of the same document rather than unrelated text, so it embeds close to the query that the live version answers. Those documents were therefore competing against their own previous revisions for top-10 slots, and that produces two distinct harms. Every slot an old revision takes displaces some other document that should have been in the window. And the slot it takes is next to the live version of itself, which produces the specific user experience of "the right document is there, but the wrong version of it" — the failure that gets believed rather than reported.

Check 3 is the measurement that turns opinion into a number, and it came back at 0.94 against a build-time figure of 0.985 that had been recorded in a spreadsheet during the original rollout. That is a gap of four and a half points. Its value lies in what the comparison holds fixed: exact search and the index rank the same stored vectors, produced by the same embedding model, scored by the same distance function, with one difference between them, which is that one visits every vector while the other navigates a graph. Anything the comparison detects is therefore a property of the navigation.

That makes the four-and-a-half point gap a structural loss rather than a corpus loss, and the distinction is what makes it actionable. New documents change which chunk a user ought to receive, and they cannot move this number, because they are present on both sides of the comparison. Neither can a shift in what people are asking, since the frozen query set is the same on both sides too. The only thing left that the number can be measuring is the index's own ability to find its own contents, which is why it is the number that justified everything that followed.

The rebuild that was never scheduled#

Check 4 asked when the HNSW graph — hierarchical navigable small world, the layered proximity graph most vector stores build by default — had last been built from scratch, and the answer was that it had been built once, fourteen months earlier, when the corpus was 2.6 million chunks. Everything since had arrived through incremental insertion. Both halves of the healthy criterion had been breached rather than one: a rebuild per quarter would have fallen due four times over, and a rebuild per 50% growth would have fallen due at least twice on the way to 2.4× the construction size.

Incremental HNSW insertion is not wrong. It is the design, and it degrades in a specific way that is worth walking rather than asserting. Inserting a node runs the same search a query runs, over the graph as it stands at that instant, and wires the new node to the neighbours that search finds. The pruning heuristics that keep edge lists diverse — the rules that stop one node's out-edges from all pointing into the same small cluster — also operate on the graph as it exists at that moment, because they can only diversify among candidates the current graph offers them.

Every insertion is therefore local, and locality is what makes it cheap. That step is a design decision rather than a physical requirement: an insert could in principle reconsider the global layout, and it does not, because reconsidering it is what a rebuild is. So the upper layers, which are the coarse levels a search descends through, and the entry points a search starts from, were chosen for a corpus less than half the current size, and nothing that has happened since was allowed to revise them. Searches consequently enter in the wrong neighbourhood more often. A search that enters in the wrong neighbourhood spends its hop budget correcting course instead of exploring, and with ef_search fixed it can exhaust that budget before it arrives. Which shows up as exactly the recall loss check 3 measured.

Rebuilding is the remedy, and on this corpus it took 3 hours 40 minutes on 16 cores, done on a standby node and swapped in with the alias mechanism from chapter 7, where two fully built versions and one atomic routing switch make the change reversible. The point of doing it that way is that the operation had a rollback rather than a repair path. Recall against exact went from 0.94 to 0.981.

The orphan sweep from chapter 2 ran first, so the rebuild did not carry 170,000 stale chunks into the new graph. That figure is not a second measurement. It is check 2's result written as a count: two point seven percent of six point three million chunks is a hundred and seventy thousand, so the ratio and the absolute number are one measurement seen two ways. Doing that conversion on your own corpus before a rebuild is worth the minute it takes, because the rebuild is what makes orphans permanent. An orphan in an incrementally updated graph is a stale vector a later reconciliation can still remove; an orphan copied into a freshly built graph has been re-inserted deliberately, wired into edge lists as a legitimate neighbour, and immortalised.

Together those two actions recovered roughly 60% of the measured relevance loss, which left a real remainder and pointed at check 5. The remainder is what makes the next section a finding rather than a guess. Had the rebuild and the sweep recovered all of the loss, the composition hypothesis would have been eliminated by the same measurement that had been used to eliminate the others.

What the new documents did to the score distribution#

The product manuals were ingested through the same pipeline as everything else, and that pipeline chunked on a fixed token budget with a fixed overlap. Manual pages are dense, uniformly formatted and heavily cross-referential, so a splitter working to a token budget rarely meets a natural break before it meets the budget, and their chunks came out at a mean of 740 tokens against a corpus mean of 320. That length difference is the whole mechanism, and it acts through the embedding rather than through anything in the index.

A chunk holding 740 tokens of related technical content covers many subtopics at once, and its embedding is a single vector summarising all of them together. A vector that summarises a broad region sits near the middle of that region, which puts it a moderate distance from a great many technical queries without being especially close to any one of them. The consequence is visible in where those chunks landed: they did not win top-1 often; they filled ranks 4 through 9 across a broad range of queries, pushing shorter and more specific chunks out of the window.

That is a corpus composition effect and it is genuinely caused by growth, which is why the original hypothesis was not stupid. The remedy is not a bigger k, though, and the reason is in the mechanism above: a wider window admits more of the same population that was filling the window already, since the long chunks are competitive at every depth. The remedies that engage the mechanism are chunking the new source to match the distribution of the rest of the corpus, or accepting the length difference and compensating in retrieval with per-source diversity constraints so a single source cannot occupy more than a set number of the top 10.

The gauges belong on the latency dashboard#

Every number in this investigation was recoverable, and every one of them cost an afternoon to recover because none of them were being recorded. Four gauges, one per mechanical check, emitted on a schedule, would have made the whole sequence a fifteen-minute read of a graph.

def emit_index_health(collection: str) -> None:
    stats = store.collection_info(collection)
    live, deleted = stats.vectors_count, stats.deleted_vectors_count
    gauge("index.tombstone_ratio", deleted / max(live + deleted, 1), {"c": collection})
    gauge("index.deleted_total", deleted, {"c": collection})

    gauge("index.orphan_ratio", orphan_sample_ratio(sample=5_000), {"c": collection})
    gauge("index.recall_at_10_vs_exact", recall_at_k(FROZEN_QUERIES, k=10), {"c": collection})
    gauge("index.days_since_rebuild", days_since(stats.built_at), {"c": collection})
    gauge("index.vectors_since_rebuild", live - stats.count_at_build, {"c": collection})

Those belong on the same dashboard as p95 retrieval latency — the figure 95% of queries come in under — rather than on a separate page. The failure mode this whole topic describes is quality degrading while operational health looks fine, so a quality gauge on a separate dashboard is a quality gauge nobody opens. Placement is doing real work here rather than being a matter of taste: the person who opens the latency dashboard opens it for an unrelated reason, on a day when nothing has prompted them to ask about relevance, and the only way a slow decline gets noticed is if it is in front of someone who was not looking for it.

The recall number is the one to alert on, with a threshold set as a relative drop from the value recorded at build time rather than an absolute figure, since the right absolute value depends on your ef_search and your corpus. An absolute threshold fails in both directions. A generously configured index can sit comfortably above a fixed line while having quietly lost several points of its own build-time recall, and a deliberately tuned low-latency configuration can sit below that same line on the day it was built, healthy and alerting forever.

The frozen query set needs a maintenance policy of its own, and the trap is the one chapter 7 identified, where an evaluation set frozen six months ago exercises the documents that existed six months ago. Applied here, that means a set frozen at build time only exercises content that predates everything you are worried about — the growth, the edits and the new source are all invisible to it. Rotating in fresh queries quarterly, sampled from recent traffic, keeps the measurement pointed at the current corpus, at the cost of a discontinuity in the time series each time you rotate — which is manageable if you run both sets in parallel for one cycle and record the offset.

None of the five checks look at the queries#

Users ask different questions over time, and a corpus that acquires product manuals also acquires users who ask manual-shaped questions, so a portion of any six-month drift can be the query distribution moving away from what the retriever was tuned and evaluated for. Detecting that means clustering query logs by month and comparing the distributions, and the remedy lives in chunking, in hybrid weighting, or in the reranker rather than in index maintenance.

It presents identically to everything in this chapter, which is a reason to run the mechanical checks first: they are cheap and falsifiable, and eliminating them is what makes the query-side hypothesis testable instead of a shrug. It is also where the chain of blind spots runs out. Each check in the table is answered by the one below it, and below check 5 there is nothing in the index to look at, because the query distribution is not a property of the index at all. That is the honest boundary of this diagnosis, and a residue that survives all five checks is the signal to cross it.

Raising k also has a legitimate version, worth separating from the rationalisation. A system with a strong cross-encoder reranker genuinely should retrieve more candidates than one that serves the vector ranking directly, because the reranker's job is to be precise over a wider net, and a net that is too narrow wastes it. The distinction is in what the number was chosen from. Choosing k = 30 because you measured that the reranker improves nDCG at that width — nDCG being the ranking quality score the relevance review reports — is engineering; choosing it because documents keep falling out of the top 10 is a way of not finding out why.

What carries over from here into daily work is a small set of habits: treat every update as a reconciliation, keep a revision key on every chunk, put the tombstone ratio and the recall check on the latency dashboard, schedule the rebuild before someone files a ticket, and run a model change as a versioned migration with a rollback window rather than as a config edit. The individual mechanisms in this topic are each an afternoon of work. Their absence is what turns a healthy index into one that has been quietly getting worse for two quarters while every operational signal stayed green.

The order those habits are checked in is the part that survives the specifics of any one corpus. Mechanical faithfulness first, because it is cheap and it can be proved absent; structural quality next, because it can be measured against a correct answer the index itself supplies; and the state of the corpus last, because that hypothesis is the one that will agree with any evidence put to it and therefore has to be reached with the other four already eliminated. An answer arrived at in that order tells you what to fix. The same answer arrived at first is a sentence about growth that will be just as true, and just as useless, in four months.