The corpus keeps changing for every day the backfill is running

Catching up is the half of a reindex that has no batch job in it. A backfill — the job that walks every document you already have, embeds it with the new model and writes it into the new index — reads from a snapshot, a consistent read handle that fixes what the data looked like at one instant so that a job scanning it for days sees a single unchanging version rather than a moving target. That property is what makes the scan safe to restart, safe to parallelise and safe to checkpoint. It is also what makes the scan obsolete from the moment it starts. A corpus of eighty million chunks belonging to a live product does not stop while its vectors are rebuilt: its documents are inserted, edited and erased throughout, at a rate worth naming as churn, the fraction of the corpus written in a day. One and a half per cent a day is ordinary for a document store, five per cent is ordinary for a ticket or log corpus, and the difference between those two numbers decides more of this chapter than anything else in it.

So a reindex measured in days needs an answer for the writes that land during it, and there are two mechanisms that give one. Either the write path is told about both index generations and updates both, or the database's change data capture stream — the ordered feed of row-level inserts, updates and deletes that a database emits from its write-ahead log — is read from the snapshot's position and applied afterwards. Getting this wrong is expensive because the result is silent. A new index that missed six days of writes does not error, does not fail a count, and does not look behind in aggregate. It holds the corpus as it stood on the day the snapshot was taken, for exactly the documents that have changed since, and it holds that content confidently until somebody notices at cutover — the moment query traffic is moved from the old index to the new — or, more often, a week after it.

Two writers, one of them reading the past#

Follow one row through a backfill and the defect stops being a story and becomes a shape. The job reads from a snapshot taken before it began, so every row it processes describes the corpus as it stood at that instant. A worker that picks such a row up on the Thursday is therefore writing Sunday's truth into the new index, and it has no way to know that is what it is doing: the row carries no mark saying it is stale, because at the moment the snapshot was taken it was not. Meanwhile the live system is writing Thursday's truth into the old index, which is the only index serving traffic. Two writers are building the new generation, and one of them is reading the past.

Edits and deletes come apart at exactly that point. An edit that the backfill overwrites leaves evidence behind, because the newer content is still sitting in the index it was written to and the older content that replaced it is still attributed to the same document. A delete leaves nothing. A delete issued on the Wednesday against a document the new index has not yet received is a no-op: the id is not there to remove, the operation succeeds, and it leaves no trace for the backfill to consult when it arrives at that document a day later. The document is therefore deleted everywhere it exists and then created somewhere it does not, in that order, by two components each doing its own job correctly.

What makes this hard to catch is that every component's books balance. The live index is correct, since the erasure removed everything it held. The backfill processed every record in its snapshot exactly once, so its checkpoint arithmetic is clean. The audit trail says the erasure succeeded, and it did. There is no pending operation to retry and no error to alert on. The defect only exists in the relationship between a job reading the past and a system writing the present, and a monitor watching either side on its own cannot see a relationship.

Two ways to stay caught up#

A backfill needs a story for every write that happens after its snapshot, and there are two that work.

Dual-writing means that from the moment the snapshot is taken, every ingest, edit and delete is applied to both generations. The live index gets the old-model vector, the new index gets a new-model vector produced by an encoder call on the write path, and the backfill fills in history behind them.

The two Note lines are the whole design. A backfill worker reading a snapshot is a writer from the past, and its writes must lose every race against live traffic, which means the write to the new index is conditional on the revision key established in chapter 2, where each document carries a monotonically increasing revision and its chunks are attributed to the revision that produced them, rather than unconditional. Without the guard, a document edited on Tuesday is silently reverted to its Sunday content on Thursday when the backfill catches up to it, and the resulting index passes every count check while holding stale content for exactly the documents that changed most recently.

The two mechanisms in those notes are different in kind, and the difference is the reason both are needed. An edit that arrives late can be rejected by a comparison, because the live write and the backfill write each carry a revision number and the higher one wins. The guard is a conditional: it is evaluated against something the index still holds at the moment the late write shows up. A delete cannot be handled that way, because a delete removes the very thing that would have carried the revision. After it runs there is nothing left in the index to compare against, and an id that is absent because it was erased is indistinguishable from an id that is absent because the backfill has not reached it yet.

Deletes therefore need a durable record rather than a conditional — a deleted_since_snapshot set, written on the delete path and read by the backfill before each insert, which closes the gap by remembering the fact that an erasure happened after the thing erased has gone. The revision guard is a conditional; the tombstone set is a memory. A design carrying only the guard resurrects documents that were deleted during the backfill, and a design carrying only the tombstone set silently reverts documents that were edited during it. The two cover disjoint failures and neither substitutes for the other.

def backfill_batch(rows: list[SnapshotRow]) -> None:
    doc_ids = [r.doc_id for r in rows]
    deleted = tombstones.members(doc_ids)              # Redis set / table, durable
    current = new_index.revisions(doc_ids)             # what v3 already holds

    pending = [
        r for r in rows
        if r.doc_id not in deleted                     # erased since the snapshot
        and r.revision > current.get(r.doc_id, -1)     # already superseded by a live write
    ]
    if not pending:
        return
    vectors = encoder_v3.embed([r.text for r in pending], batch=128)
    new_index.upsert([
        Point(id=f"{r.doc_id}:{r.ordinal}", vector=v,
              payload={"doc_id": r.doc_id, "rev": r.revision})
        for r, v in zip(pending, vectors)
    ])
    checkpoint.commit(rows[-1].snapshot_offset)

Skipping before embedding rather than after is worth the extra round trip: on a corpus with meaningful churn, 8% to 12% of the snapshot has been superseded by the time the backfill reaches it, and those are provider tokens you do not spend.

The two orders differ only in what they waste, and the sizes are not comparable. Filter after embedding, and the encoder has already been called on every superseded row before the index rejects the write, so the money is spent on work that was thrown away. Filter before, as the two lookups at the top of that function do, and the cost is one extra round trip to the new index and one to the tombstone store. What makes that the cheaper side is that both lookups are batched by id, so a batch pays for two calls rather than one per row, and both calls are to infrastructure you already run. The embedding call is per row and billed by a provider.

Dual-writing raises a question the diagram does not answer, which is what a writer does when the new index rejects its write. Making the second write synchronous and mandatory means a v3 outage becomes a user-facing write outage during a migration that was supposed to be invisible, which is an unreasonable trade: it attaches the blast radius of an index nobody is querying yet to the product itself. Making it fire-and-forget means the new index quietly develops holes exactly when it is under stress.

Which holes those are is what settles the argument. The documents lost to a fire-and-forget write during an incident are the documents written during the incident, which is to say the most recently written documents in the corpus. The freshness gate that runs before cutover compares the newest timestamp each index holds against the source, so it catches an index that is behind. A hole is not lag: an index can hold a document written a second ago and be missing three written a minute ago, and its maximum timestamp reads perfectly fresh either way. The one defence that will definitely be running is blind to precisely the damage that option produces.

The resolution most teams land on is an outbox: the writer commits the row and a v3 work item in the same database transaction, and a relay process drains the outbox into the new index with retries. The write path then depends only on the primary database, the new index can be down for an hour without losing anything, and outbox depth becomes the lag metric you alert on. It resolves the dilemma rather than splitting the difference, because the atomicity of that one commit is what removes the choice — the work item cannot be lost without also losing the write that produced it, so durability is no longer something the second write has to buy with availability.

The alternative is change-log replay, which decouples the write path from the migration entirely. The backfill reads the snapshot, a separate consumer reads the database's change stream from the snapshot's log position forward, and once the backfill finishes, the consumer replays every insert, update and delete that happened during it, in per-document order. The write path never learns that a migration is happening, which is the main attraction — no dual-write code in the hot path, no risk of a new-index outage taking down user writes, and no encoder call added to user-facing latency.

The tail row is where the schedule gets decided. Replay has to process everything that happened during a six-day backfill, and it does so at whatever rate the consumer can embed and write, so the same T = N / (R − C) arithmetic from chapter 5, where the reindex schedule is sized against sustained embedding throughput, applies again to the tail: on 80 million chunks at 1.5% daily churn, replay starts with about 7.2 million changed chunks to work through and accumulates more while it does, which is roughly another 14 hours at the backfill's own rate. At 5% daily churn it is four days, and at that point the tail is not a tail.

Both of those numbers are worth deriving rather than accepting, because the same three quantities produce them. Start with the backlog. Churn of one and a half per cent a day against eighty million chunks is eighty million times fifteen thousandths, which is one and a fifth million chunks changed per day, and six days of backfill accumulates six of those — about seven and a fifth million changed chunks waiting when the scan finishes. That is N, the work in front of the consumer at the moment it starts. R is the rate at which the consumer embeds and writes, and it is the backfill's own rate, since it is doing the same two operations against the same provider and the same index. C is the churn that keeps arriving while the replay runs, another one and a fifth million chunks a day, and it is subtracted from R because every chunk of it is work the consumer must also do before it may call itself caught up.

Substituting gives the fourteen hours. The backfill moved eighty million chunks in a little over six days, so R is a bit over thirteen million chunks a day, and subtracting the one and a fifth million of daily churn leaves a net rate of about twelve million a day. Seven and a fifth million divided by twelve million a day is a bit over half a day, which is the fourteen hours the row promises. The subtraction is what turns thirteen hours into fourteen: the consumer is not catching up at its full speed, it is catching up at its speed minus the speed at which new work keeps arriving.

At five per cent daily churn every one of those three quantities moves, and all three move the wrong way. C is now four million chunks a day, and because C is subtracted from R inside the schedule calculation too, the backfill itself takes longer — chapter 5, sizing the same job, put it at nearly nine days rather than six. A longer backfill accumulates a larger backlog, so N is four million a day across those nine days rather than one and a fifth million across six. The same larger C is then subtracted again during replay, so the denominator shrinks at the same time as the numerator grows. Divide the one by the other and the answer is four days, at which point the tail is not a tail but a second migration — and the arithmetic that has to choose between dual-writing and replay turns out to be the same arithmetic that chose the schedule.

Replay has two operational requirements that are easy to get wrong. The snapshot and its log position must be captured atomically — a pg_export_snapshot alongside the LSN, the log sequence number identifying that position in the database's write-ahead log, or the equivalent consistent-read handle your database offers. A snapshot taken at 18:00 and a log offset noted at 18:04 leaves four minutes of writes that neither the backfill nor the consumer will process: the backfill cannot see them, because they happened after the instant its snapshot fixed; and the consumer will not replay them, because it begins from an offset that is already past them. Nothing downstream can detect the gap, which is what makes it worse than a loud failure — those writes were never enqueued anywhere, so there is no queue to inspect, no retry to exhaust, and no error count that stays stubbornly above zero.

The consumer also needs per-document ordering rather than global ordering, which means partitioning the change stream by doc_id and running one worker per partition; that preserves the property that matters — an edit never overtakes the delete that followed it — while letting you run sixty-four workers in parallel. The ordering is the requirement and the partitioning is a design decision that satisfies it, which is worth separating because only the second one is negotiable. Global ordering satisfies the requirement as well, and it caps replay at the throughput of a single consumer, which drives R down and makes the tail arithmetic above far worse in the one term that was already the problem.

Two gates that decide whether you may cut over#

Both strategies converge to the same claim — the new index now reflects the corpus — and that claim needs evidence before traffic moves. The evidence everyone reaches for first is a count: how many vectors does v3 hold, how many does v2, and do the two agree. It is cheap, it covers the whole index rather than a sample, and it is the wrong instrument, because it is a defence that is constant when the thing it must discriminate varies. One document wrongly present and one document wrongly absent shift a total of eighty million by less than ordinary ingest timing shifts it between two measurements taken a second apart, so the comparison returns the same verdict on a clean migration and on one carrying a contract that was legally erased.

The check that catches the specific failures in this chapter is freshness, because every one of them produces a new index that is behind rather than wrong in aggregate.

def freshness_gate(max_lag_s: int = 120) -> GateResult:
    newest_source = db.scalar("SELECT max(updated_at) FROM documents")
    newest_v3 = new_index.max_payload_value("updated_at")
    newest_v2 = live_index.max_payload_value("updated_at")

    lag_v3 = (newest_source - newest_v3).total_seconds()
    lag_v2 = (newest_source - newest_v2).total_seconds()
    return GateResult(
        passed=lag_v3 <= max_lag_s and abs(lag_v3 - lag_v2) <= max_lag_s,
        detail={"source": newest_source, "v3_lag_s": lag_v3, "v2_lag_s": lag_v2},
    )

Comparing both indexes against the source rather than against each other matters because a stalled ingest pipeline makes the two agree perfectly while both fall behind, and a gate that only diffs v2 against v3 would pass. The chain is worth walking. An ingest pipeline — the path that turns a new or edited document into chunks and vectors — stops. Both indexes stop receiving writes at the same instant, because both are fed by it. Their newest timestamps therefore stay equal to each other while both slide further behind the documents table. A gate that diffs one index against the other reads that agreement as health. The primary store's newest timestamp is the only clock in the comparison that is not downstream of the stall, which is why it, and not the other index, has to be the reference. Reading the newest timestamp per shard rather than globally is a refinement worth adding when the store is partitioned, since a single lagging shard is invisible in a global maximum.

The second gate is chunk-count parity on a sample. Draw a few thousand documents at random, weighted towards recently updated ones, and compare the number of chunks each generation holds for them against the primary store.

WITH sample AS (
  SELECT id FROM documents
  WHERE updated_at > now() - interval '14 days'
  ORDER BY random() LIMIT 2000
)
SELECT s.id, c.n_db, v2.n AS n_v2, v3.n AS n_v3
FROM sample s
JOIN LATERAL (SELECT count(*) n_db FROM document_chunks WHERE document_id = s.id) c ON true
JOIN LATERAL index_chunk_count('v2', s.id) v2 ON true
JOIN LATERAL index_chunk_count('v3', s.id) v3 ON true
WHERE c.n_db <> v3.n OR v2.n <> v3.n;

Any row returned by that query is a defect with a name, and reading which of the three columns disagrees is what turns the gate into a diagnosis rather than an alarm. n_v3 = 0 with n_db > 0 means a document the backfill skipped or a tombstone-set false positive — the document exists in the primary store and the new index holds nothing at all for it, so either it was never written or something told the backfill not to write it. n_v3 > n_db is an orphan of the kind chapter 2 described, arriving in a brand-new index because the backfill wrote revision 7's chunks and a live write later added revision 9's shorter set; the surplus ids are the tail of the longer revision, unreferenced and fully retrievable. n_v3 < n_db with n_v2 = n_db is a truncated write, usually a batch that failed after partial success, and the n_v2 = n_db half of that condition is what rules out the primary store having simply moved on since. The sample is deliberately biased towards recent documents because that is where the migration's failure modes concentrate; a uniform sample over 80 million chunks would need to be far larger to find the same defects.

What that sample can and cannot find follows from its size, and the arithmetic takes one line. Drawing 2,000 documents at random from a corpus of 4.1 million documents gives each document roughly one chance in two thousand of being looked at, so a defect confined to a single document is missed on all but about one draw in two thousand. It is essentially certain to be missed. That is what licenses the division of labour in this chapter: the tombstone set is a preventive mechanism, and a sample of this size is not a detective one for defects of this size.

The mistake is a tempting one and it is worth saying why. A parity check on a random sample is exactly the gate a careful team would build, and it does catch the failures it is built for. A skipped shard, a truncated write, a whole batch lost to a crash — each of those affects thousands of documents at once, and a draw of two thousand hits one of them with near certainty. Building it is right. Reading a pass as evidence that nothing got through is the error, because the class of defect this chapter is about is one document wide.

A chunker change, and content the change log never sees#

Neither strategy handles a change to chunking. If the migration also alters chunk size, overlap or the splitter, then a document's chunk ids in v3 have no correspondence with its ids in v2, the revision guard still works because it is keyed per document, and the parity check comparing n_v2 against n_v3 becomes meaningless and has to be dropped in favour of comparing v3 against a freshly computed expected count. Bundling a chunker change into a model migration is common, since both require a full rebuild, and it removes one of your two cheap gates.

Replay also inherits whatever the change log does not capture. A CDC stream built on the database's write-ahead log sees row changes and nothing else, so a document whose content lives in object storage and whose row only holds a pointer produces no change event when the file is overwritten in place. The row is untouched, the log is therefore silent, and the consumer that is doing exactly what it was asked to do replays nothing. Corpora with that shape need a change signal on the storage layer, or an application-level event, and discovering the gap after a six-day backfill is an expensive way to learn it.

The two gaps share a structure with the delete that opened this chapter. In each case a mechanism is correct within its own boundary and blind just outside it: a parity check that compares two indexes cannot see a chunker change that redefined what a chunk is, and a change log that watches rows cannot see content that does not live in one. Where a mechanism's boundary sits is a question worth asking before the six days start rather than after, because both of these are cheap to discover in advance and expensive to discover at cutover.

What both strategies do produce, when they work, is two complete indexes over the same corpus, each internally consistent, each queryable, one carrying live traffic and one carrying none. Getting there means holding one idea steadily: a backfill is a writer from the past, so it must lose every race it enters, and losing a race against a delete requires remembering the delete rather than checking for it.

Deciding how traffic moves between those two indexes is a separate design with its own failure modes, and the choice made there determines whether a relevance regression discovered next week can be attributed to anything at all.