Backfill into a new generation and flip the alias, never in place

A new splitter was better on every offline measure its team had. A splitter is the step that cuts a document into the pieces a retrieval system stores and scores separately, and each piece is a chunk. This one respected headings, kept tables intact, and produced a median chunk of 340 tokens against an old fixed splitter's 800, and on a 500-query golden set — a fixed list of questions whose correct documents somebody has already established — it moved nDCG@10 from 0.61 to 0.68, that metric scoring how closely the ten results returned match the known ranking, with 1.0 being perfect. A corpus of 8M documents needed it applied, serving about 400 queries per minute, with roughly 6% of it changing every day through an incremental connector — a process that watches each source system for changes and feeds the changed documents back in.

The proposal on the table was to rewrite in place: iterate over documents, re-chunk each one, and write it in a single transaction so that no query ever sees a half-rewritten document. Per-document atomicity is real and the transaction does what it claims. The plan is still wrong, and the reason has nothing to do with torn reads.

Two chunking regimes in one corpus is not a transient inconvenience#

The belief worth naming first is that per-document atomicity is the safety property this migration needs. It is tempting for good reasons: it is a real guarantee rather than a hopeful one, it costs almost nothing, and it is genuinely delivered, since across the whole run no query reads a document that is half old chunks and half new. What makes it the wrong guarantee is the unit it is scoped to. A transaction protects one document. A top-k query reads across every document and returns the ten that scored highest, so what has to be internally consistent is the entire searchable set, and a per-document transaction says nothing about that.

The mismatch shows in what the guarantee reports while the damage accumulates. The count of torn documents is zero in the first second of the run, zero at hour twelve, zero at hour twenty, zero at the end. It never moves. The quantity that decides whether a result can be trusted — what share of the collection sits in the losing regime relative to whatever it competes against — sweeps from all of it to none, passing through every value in between. A defence whose own indicator reads perfect throughout the window in which the harm is done is not measuring the harm.

The harm itself comes from how an embedding is built. A chunk is stored as one vector summarising all of the text in it, and there is only the one vector however much text went in. Adding more text therefore moves that summary away from any one thing the chunk discusses, because it now has to account for the rest as well. A 340-token chunk that is entirely about carryover limits scores higher against the query "PTO carryover limit" than an 800-token chunk containing the same sentence plus six paragraphs about parental leave, because the extra content dilutes the embedding. On this corpus the measured gap was substantial: for the golden set queries, top-1 cosine similarity averaged 0.62 against new-regime chunks and 0.54 against old-regime chunks holding the same answer.

The figure to read there is the gap, not either level. An absolute similarity of 0.62 means little on its own, depending as it does on the model, the pooling and the query. The eight-hundredths of separation between the two populations means a great deal, because ranking is a comparison and a comparison sees only differences. Top-k retrieval is a comparison across the whole index, so that gap becomes a ranking bias: two chunks holding the same answer, differing only in which regime produced them, are ordered by the regime. Documents the backfill has not reached yet are systematically outranked because of when it happened to visit them, which is a property of the migration schedule rather than of relevance.

Where thirty-seven hours comes from#

At a sustained 60 documents per second — a realistic figure once parsing and embedding are in the loop, not the 200/s the index alone could absorb — 8M documents take 37 hours. That is a division, not an estimate: eight million documents over sixty a second is a little over a hundred and thirty thousand seconds, which is thirty-seven hours. The two rates are worth separating. Absorbing a vector into an index is over three times cheaper than producing one, so the same corpus written at index speed alone would finish inside twelve hours. Parsing and embedding set the pace, making the window a consequence of the pipeline's slowest stage rather than of its size.

For a day and a half the corpus consists of two populations whose similarity scores are not drawn from the same distribution. Every measurement taken during the window inherits that. An A/B test cannot run, because both arms read the same mixed index and therefore share the same contamination. Live quality metrics move for reasons that have nothing to do with the splitter.

The eval suite fails in a more specific way, and that is the second belief worth naming. Running the golden set hourly during the migration feels like a safety net and is not one, because the number it returns drifts upward hour by hour as the mix changes and means nothing at any given point. It rises whether the new splitter is good or bad, for the same reason the ranking bias exists: each hour a larger share of the collection sits in whichever regime scores higher, so the eval is scoring that share rather than the change. A monitor that can only move one way reports the passage of time rather than a regression.

The last thing the window takes away is the way out. When something does look wrong at hour 20, there is no rollback: the old chunking for those 4M documents has been overwritten, so recovering it means running the whole migration again backwards, with the old splitter, for another 20 hours. That second number is not independently estimated. Reversing means re-chunking exactly the documents already converted, at the same rate, so the time to undo an in-place migration is the time already spent on it. The cost of going back grows with how far you have gone, and is largest at the moment you are most committed. What could break the symmetry is the splitter: the old one makes fewer chunks per document, so the reverse run is cheaper if embedding throughput binds, and not if document parsing does.

A generation is a full copy that nothing reads yet#

The alternative is to treat the rewrite as building a new artifact rather than editing the existing one. The backfill writes into a separate namespace — chunks_v3, a second Pinecone namespace, a second Elasticsearch index — while chunks_v2 continues to serve every query. Nothing the backfill does is visible to retrieval until an alias moves, an alias being the name the query path actually reads from, resolved at request time to whichever index it currently points at.

The indirection is the whole trick. A query path naming an index directly changes only when every service holding that name changes; a query path naming an alias holds a name whose meaning lives in one place, so data and the name it is known by can be changed at different times. Writing chunks_v3 becomes file copying rather than a deployment, and the deployment, when it comes, moves no bytes.

Topic 8.17 established why two embedding generations cannot share a namespace; the same isolation applies here for a different reason, since the model is unchanged and it is the chunk-length distribution that makes cross-population scores incomparable. Whichever cause applies, the remedy is that a population you cannot compare against another population does not go in the same searchable space as it.

Separating the populations also gives back the measurement the in-place plan destroyed: each generation is internally consistent, so an evaluation against chunks_v3 measures the splitter and nothing else.

Storage, tokens, and the quota that actually binds#

The costs of a second generation are storage and quota, and both are worth computing before committing. The old regime produced about 4.1 chunks per document, so 33M vectors; the new regime produces about 9.2, so 74M.

Those two pairs of figures are not four independent measurements. Chunks per document is a document's length divided by a chunk's length, and the documents did not change, so the ratio of the two chunk counts had to come out as roughly the inverse of the ratio of the two chunk sizes. It does: a little over two, read either way round. The residual between the two ratios has two sources. One is that 340 and 800 are medians while the chunk counts are means. The other is that the new splitter respects headings, forcing cuts the old fixed splitter would not have made. What the identity licenses is this section's whole premise: the vector count of a proposed splitter can be predicted from its target chunk size on a spreadsheet, without running it over 8M documents.

At 1536 dimensions in float32 that is 6.1 KB per vector before index overhead, which is 198 GB for chunks_v2 and 442 GB for chunks_v3, with a peak of 640 GB while both exist. The per-vector figure is a multiplication worth doing rather than accepting: float32 means four bytes a number, and a vector of 1536 numbers at four bytes each is a little over six thousand bytes. Multiplying that by the two vector counts lands within a few percent of the two storage lines — rounding in the chunks-per-document figures, plus the gap between a gigabyte counted in thousands and one counted in powers of two. These are sizing figures rather than invoice lines, and dividing one back out of another to recover a byte width is not safe at this precision.

The peak is the exact sum of the two lines, with no third term. That exactness describes the procedure rather than the arithmetic: at no point does it hold a third copy or need free space to merge one thing into another. What it excludes is named in the sentence that produces it — the per-vector width was quoted before index overhead, and nothing here counts replicas — so the peak is a floor on peak disk rather than the number to provision to.

Embedding 74M chunks at a mean 220 tokens is 16.3B tokens, which at $0.02 per million is $326 — trivial — but at a provisioned 2M tokens per minute it is 8,150 minutes, or 5.7 days of continuous throughput. The two conclusions come from one product divided by two different things. Sixteen point three billion tokens against a price per million gives a sum of money; the same sixteen point three billion against a rate per minute gives a duration. Money and time are the same measurement here, seen through two divisors, which means the question of which resource binds is only ever a question of which divisor is scarce. The binding constraint on a backfill of this size is almost always the embedding quota rather than money or storage, and finding that out in week two is worse than finding it out on a spreadsheet.

The mean of 220 tokens and the median of 340 are not in conflict, and the gap between them is informative. A splitter aiming at a size produces mostly chunks near that size, plus one short remainder at the end of every document it cuts, plus a single short chunk for every document too small to cut at all. That distribution has a long low tail and a mean below its median. The median suits an argument about quality, describing what a typical retrieved chunk looks like; the mean suits an argument about cost, since a bill is a total and a total is a count times a mean. Substituting the median into the token calculation would overstate the bill by about half as much again.

Holding the quota against the throughput is the check that matters most. Sixty documents a second is over five hundred chunks a second, and at a mean of 220 tokens that is over seven million tokens a minute, more than three times what a 2M-per-minute quota allows. The two are not a contradiction but two ceilings on one job: thirty-seven hours is what document throughput permits, and 5.7 days is what the quota permits. A backfill runs at the slower, which makes the thirty-seven-hour window used above the optimistic reading of an in-place rewrite, and every consequence drawn from it — the mixed window, the rollback, the staleness on a freeze — correspondingly understated.

Read down the rollback column rather than across the rows, because that column is where the four differ in kind: the first two recover by redoing work, in hours, and the last two by changing a name, in seconds or minutes. The rest of the table is the price of which of those two you end up in.

The second row is a real proposal that surfaces in these discussions and it deserves its line. Filtering queries to a single regime does restore score comparability, at the price of searching half the corpus, which is a recall loss of roughly the same magnitude as the quality gain the new splitter was supposed to deliver. That loss is the ranking bias seen from the other side: the backfill visits documents in an order unrelated to what any query is about, so the answer a query needs is as likely to sit in the hidden half as in the searched one. Restricting the search does not select a worse half. It removes an arbitrary half, and how large that removal is changes hour by hour as the split moves.

The fourth row is what you do when the index software itself is changing version, and it costs a duplicate cluster rather than a duplicate namespace.

What the connector does for 37 hours#

The corpus is not static during the backfill; 6% of 8M documents is 480,000 changes a day. Freezing ingestion for the duration would leave roughly 740,000 documents stale at the moment of cutover, which is a worse regression than the one the new splitter fixes. That staleness figure is the churn rate times the length of the window and nothing else, which is why the window's length keeps returning: everything that goes wrong here is priced in hours of it.

Dual-writing is the straightforward answer. For the duration of the backfill the connector writes each document twice — once through the old splitter into chunks_v2 and once through the new splitter into chunks_v3 — which doubles the connector's chunking and embedding work and leaves both generations continuously current. At 480,000 documents a day the extra load is about 4.4M chunks and 970M tokens per day, which fits inside the same quota conversation as the backfill itself. That figure counts one pass rather than two, for a reason easy to lose: the chunks_v2 pass was already happening, so what dual-writing adds is the new splitter's pass alone. It comes to about a third of what a 2M-per-minute quota delivers in a day, leaving the backfill the remainder and stretching its window accordingly.

The ordering hazard is that the backfill and the connector both write chunks_v3, and they can collide on a document the backfill is currently processing. The version guard from chapter 3 — where a write carrying an older source_version than the one already stored is rejected rather than applied — resolves it without any special handling: the backfill reads the document's recorded source_version and writes with it, the connector writes with the version it fetched, and the guard rejects whichever is older. A backfill that ignores the guard and writes unconditionally will overwrite fresh connector output with snapshot content, which is the same stale-write failure at corpus scale.

The alternative to dual-writing is to record the connector's cursor at the moment the backfill starts, let the connector write only chunks_v2, and then run a catch-up pass over chunks_v3 from that cursor once the bulk load finishes. It avoids doubling the steady-state load, and it introduces a catch-up phase whose duration depends on how far behind you are — 37 hours of backlog at 6% daily churn is about 740,000 documents, roughly 3.4 hours at 60 documents per second, during which the source keeps changing. The flip waits until measured lag drops under 60 seconds.

That clause about the source not standing still is the whole of the catch-up calculation. A backlog over a processing rate gives the time to drain a queue nobody is adding to, and here something is adding to it, at the churn rate, throughout. The queue shrinks at the difference between the two rates, so the honest figure is the backlog over that difference. Six-ish documents a second of churn against sixty a second of throughput leaves a difference of about nine tenths of the throughput, making the true drain roughly a tenth longer than 3.4 hours. That correction is small only because churn is a small fraction of capacity. As churn climbs toward throughput the drain runs away, and once churn exceeds throughput it never converges at all, at which point dual-writing stops being an option and becomes the only one.

Written that way, both catch-up figures in this chapter are one formula: a drain is the length of the gap times churn over throughput, a ratio that sits under a tenth here. Thirty-seven hours of gap gives 3.4 hours of drain; a six-hour gap gives about half an hour. Catch-up is a fixed fraction of however long you were not writing rather than a fixed cost, which is also why a lag threshold works as a flip gate at all.

Dual-writing is simpler to reason about; catch-up is cheaper. Pipelines that already have a durable change log tend to choose catch-up, and pipelines whose connector is the only record of what changed tend to choose dual-write.

Moving a name in a single operation#

The flip itself has to be a single atomic operation, which is why the alias exists rather than a config value read by each service instance:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "chunks_v2", "alias": "chunks-live" } },
    { "add":    { "index": "chunks_v3", "alias": "chunks-live" } }
  ]
}

Both actions apply in one cluster-state update, so no query ever sees zero indexes or two. Each link under that sentence is checkable. An alias is resolved when a request arrives, so the answer depends on the cluster state that node holds at that instant. Cluster state arrives and is applied as a whole version, so a node holds either the version before this update or the one after it, and both the removal and the addition sit in the same version. Every resolution, on every node, therefore sees both old mappings or both new ones; the intermediate states — a chunks-live pointing at nothing, or at both — exist in no version and are never observed.

The equivalent in a vector database without aliases is a single row in a control table that every query path reads through a short-TTL cache, and the TTL becomes the window during which some replicas serve the old generation — acceptable at 5 seconds, not acceptable at 5 minutes. This is per-document atomicity's mistake one layer along, and the substitute looks atomic because the control table really is updated by one write. What a single write does not buy is a single read: each service instance reads on its own schedule and caches the answer, so for up to one TTL some resolve to the old generation and some to the new. The write count stays at one however long the TTL is, while the split serving window is the TTL exactly.

That window costs less than the 37-hour version, and the difference is worth being precise about. Each individual query still reads exactly one generation, so scores within a result set stay comparable and no ranking is contaminated. What breaks is consistency between one query and the next: two users asking the same question in the same moment can get different answer sets, and a user who reloads can watch the answer change. Hence a TTL in seconds. A reload inside a few seconds is rare enough to be noise; a reload inside a few minutes is ordinary behaviour, and a window that wide makes the inconsistency something users meet rather than something that merely exists.

Deciding to go back, and what going back costs#

Before the flip, the decision is offline: run the golden set against chunks_v3 and compare. The threshold that had been agreed here was nDCG@10 no worse than the current generation minus 1 point, plus recall@50, the share of the known-relevant documents that appear anywhere in the top fifty results, no worse at all, evaluated on the finished generation rather than a partial one — which is precisely what in-place rewriting makes impossible.

That tolerance of a point is wider than it looks against an offline gain of seven, and the width is doing a job. A 500-query set carries a sampling error of its own, and with per-query scores spread over a few tenths, the error on a mean of five hundred lands at around a point. The gate therefore reads as "not measurably worse" rather than "not worse", the strongest thing a set that size can certify.

After the flip, the signals are live and they need to be fast, because a bad cutover degrades every query. Citation click-through on answers, measured over rolling 30 minutes, is the most responsive quality proxy at 400 queries per minute, giving about 12,000 queries per window and enough resolution to detect a 10% relative drop. That resolution claim is a square-root argument: noise in an observed rate falls as the square root of the number of observations, so a window of twelve thousand resolves a tenth-scale move in a rate of a few tens of percent many times over, and runs short of room only when the rate itself is down at a few percent.

The fraction of queries returning nothing above the score threshold catches a regression that recall metrics can miss, since smaller chunks shift the absolute score distribution and a threshold tuned for 800-token chunks may cut too much. That shift reads backwards against the 0.62-versus-0.54 gap until two populations of query are separated. Where an answer fits inside one new chunk, the new regime scores higher, exactly as that gap says. Where an answer spans more text than one new chunk holds, no single chunk contains all of it, and the best available score falls below what the old 800-token chunk achieved by containing the whole thing. A threshold sees both at once, and it is the second that pushes queries under it.

p95 retrieval latency matters because 74M vectors search more slowly than 33M — expect 15–25% more, and treat anything beyond that as a sign the new index needs different parameters rather than a reason to roll back. More than doubling the vector count for well under a quarter more latency is not luck. An approximate index answers a query by visiting a bounded number of candidates rather than every stored vector, and that number grows with the logarithm of how many are stored rather than in proportion to it. That is what makes the ceiling diagnostic: latency rising in step with the vector count means the index has stopped behaving like an approximate one, and what to change is how it was built and searched.

Rolling back is an alias swap taking milliseconds, and its real cost is whatever has happened to chunks_v2 since the flip. If dual-writing continued after cutover, chunks_v2 is current and the rollback is free, which is the reason to keep paying for dual-write through the retention window rather than stopping at the flip. If dual-writing stopped, chunks_v2 is stale by the time you were live: six hours at 6% daily churn is 120,000 documents, about 33 minutes of catch-up before the old generation is trustworthy again, during which retrieval is serving content up to six hours old. Seven days is a common retention choice because it covers a weekend plus the time it takes for a subtle quality regression to be noticed by users rather than by dashboards, after which dropping chunks_v2 reclaims the 198 GB — the same figure the storage calculation produced, and the same one the diagram ends on.

Which parts of this are forced and which were chosen is worth separating, since they are not defended the same way. Three are forced: the two populations must not share a searchable space, because top-k is a comparison; the collection must not go stale, because of the churn rate; and there must be a way back cheaper than the way forward, because of how the reverse-migration arithmetic came out. The rest are decisions. A second namespace rather than a second cluster is one, with the table's fourth row as its alternative. An all-at-once flip rather than a weighted ramp buys a clean boundary and gives up gradual exposure. Seven days of retention, a 60-second lag gate and a TTL ceiling of seconds are decisions too, each movable without touching anything above.

None of this machinery is warranted below a certain size. A 100,000-document corpus rebuilds from scratch in about twenty minutes, so the whole procedure collapses into "build a new index, point at it, keep the old one until Friday" — which is the same idea with none of the coordination. Generations earn their complexity when a rebuild is measured in days.

What the last seven chapters describe is a pipeline whose failure modes are bounded: a cursor that cannot skip, a write that cannot duplicate or go backwards, deletions that reconcile, failures that are retained and drained, and rewrites that are reversible. All five turn on what this chapter turns on: a guarantee is worth only what its scope covers, and the scope a retrieval system needs is wider than the one a storage engine hands you.

Running it is a matter of watching a small number of quantities — connector lag against each source, coverage per source, quarantine age, staleness at the p99 — and treating a change in any of them as a defect in the same way a rising error rate would be. Those numbers are what the rest of the retrieval stack is standing on, and they are the ones to have on a dashboard when somebody asks why an answer cited a document that no longer exists.