Sustained embedding throughput, not the token price, sets the reindex date

A reindex date is the day a second embedding generation is finished and can be routed to. Chapter 4, where two embedding generations cannot share one namespace even at equal dimensionality, is why there has to be a second one at all: changing the model means every chunk in a corpus is embedded again and written into a new index rather than updated in place. Naming the day that finishes is an arithmetic problem with five terms in it, and four of the five are usually left uncomputed.

Three of those terms need naming before anything can be worked out. Sustained throughput is the number of tokens a minute a job actually completes end to end over many hours — after rate limiting, after retries, after the stretches where a client is waiting rather than sending — as distinct from the rate it reaches in its best minute. A tokens-per-minute quota, usually written TPM, is a provider's admission rule: within a rolling minute it accepts requests until the tokens inside them add up to the quota, and rejects whatever arrives after that. Churn is the fraction of a corpus rewritten per day, and it matters here because every document edited while a backfill is running produces chunks the backfill has already passed or has not yet reached. A date named from the bill alone is routinely wrong by a factor of two and can be wrong by more than an order of magnitude, and while it is wrong two index generations sit resident on the same hardware and an ingest path competes with live queries for the same cores.

Five quantities, worked through#

Each row below is a number that can decide the date on its own, and each is computed from figures a team already has. The right-hand column works all five for one corpus of eighty million chunks averaging 380 tokens apiece.

The first row is two multiplications and a division, and it is worth walking because its cheapness is the whole problem with it. Eighty million chunks at a mean of 380 tokens each is eighty million multiplied by 380, which is thirty billion tokens — 30.4 billion, to the precision the table carries. Prices are quoted per million tokens, so the token count is divided by a million before the price is applied: 30.4 billion over a million is thirty thousand four hundred million-token units, and at $0.02 each that is $608. Swap the unit price for a premium model at $0.13 and the same thirty thousand four hundred units come to $3,950, which changes nothing structural about the plan. The bill is the quantity that is easiest to compute and least likely to constrain anything, which is why it is the one that gets computed. Four other numbers decide whether a date is achievable, and each of them can be the binding one depending on the shape of a corpus and the contract with the provider.

The second row is where a weekend plan dies. A tokens-per-minute quota of five million sounds generous next to a thirty-billion-token job until you divide, and dividing a token count by a rate in tokens per minute is what turns a size into a duration: 30.4 billion tokens over five million tokens a minute is 6,080 minutes of pure embedding time. Sixty of those minutes make an hour, so 6,080 minutes is 101 hours. Set that against a 72-hour window and the job is already 40% longer than the time available, before a single retry, and before utilisation has been applied at all.

Utilisation is the fraction of the quota a job sustains over hours rather than peaks at, and it is below one for a structural reason rather than through carelessness. A rate limiter that admits five million tokens a minute does not queue the request that would exceed that figure — it rejects it. The client sees the rejection and backs off, which means it stops sending. The window during which it was backing off ends with tokens still unspent, and the next window therefore starts underfilled. Sustained utilisation of 65% to 80% against a stated quota is a normal outcome for a well-written batch client, and the whole loss comes from that gap between rejection and resumption rather than from any inefficiency in the code. At 70% the effective rate is five million times 0.70, or 3.5 million tokens a minute, and 30.4 billion divided by 3.5 million is a little over eight thousand six hundred minutes, which is the 145 hours the table quotes and which is six days.

The third row of the throughput calculation is the one that produces order-of-magnitude errors rather than factor-of-two errors. Providers enforce a requests-per-minute limit alongside the token limit, and which one binds is decided entirely by how many texts you put in each request. Both limits are live at once, and neither one relaxes because the other is slack. Converting the requests limit into the same units as the token limit is what makes them comparable: requests a minute, times texts per request, times tokens per text, is tokens a minute.

At 2,000 requests per minute with 128 texts per request, you can push 256,000 chunks per minute and the token quota binds first. Those 256,000 chunks at 380 tokens apiece are ninety-seven million tokens a minute of implied capacity, which is far above the five-million-token quota, so the quota is what stops you and the request limit never comes near being reached. At one text per request — which is what a naive port of the online encoding path does — you get 2,000 chunks per minute, and 80 million chunks takes 27 days. That second figure is one division: eighty million chunks over 2,000 chunks a minute is forty thousand minutes, and forty thousand minutes is 27 days. That is the same code, the same provider and the same quota, differing only in batching, and it is the single most common reason a backfill estimate is off by an order of magnitude. The estimator in the next section computes both capacities in tokens per minute and takes the smaller of the two with min, because a job that satisfies one limit and violates the other is still a job the provider stops: the smaller capacity is the one you meet first.

Storage amplification is easier to compute and easier to forget. A 1024-dimension float32 vector holds four bytes for each of its dimensions, so it is 1024 times four, which is 4,096 bytes, and 80 million of them is a little over three hundred and twenty-seven billion bytes, or 328 GB before any index structure. During the migration both generations are resident, because the old index keeps answering queries until the new one has been validated, which doubles that to 656 GB of raw vectors. On top of each generation sits the graph: HNSW keeps a bounded neighbour list per vector, and at M = 32 those edge lists come to roughly 21 GB per generation, plus payloads. Two generations of vectors and their edges is therefore just under seven hundred gigabytes with nothing moving.

Compaction or segment merges on either side need transient headroom on top, so provisioning at about 2.4 times the steady-state footprint of one generation is a reasonable planning figure — and 2.4 times 328 GB is the ~790 GB peak the table quotes. That multiplier is the one link in the chain that is a convention rather than a measurement. The 4,096 bytes are arithmetic on a storage format, the 656 GB follows from a decision to keep both generations queryable, and the 2.4 is a planner's allowance for merge space chosen to be comfortable. Scalar quantization to int8, which stores each dimension as a single byte rather than a four-byte float, cuts the vector portion by four and is worth considering for the new generation specifically, since you are rebuilding anyway and the quality cost can be measured on the new index before cutover.

An estimator you run before you name a date#

The arithmetic is short enough to write down once and keep, which matters because the inputs change between migrations and the mistakes come from doing it in your head.

def plan(n_chunks: int, mean_tokens: int, price_per_mtok: float,
         tpm_limit: int, rpm_limit: int, batch: int,
         utilisation: float, churn_per_day: float,
         dim: int = 1024, bytes_per_dim: int = 4) -> dict:
    tokens = n_chunks * mean_tokens
    spend = tokens / 1e6 * price_per_mtok

    tok_capacity = tpm_limit                       # tokens/min
    req_capacity = rpm_limit * batch * mean_tokens  # tokens/min implied by RPM
    effective_tpm = min(tok_capacity, req_capacity) * utilisation
    hours_static = tokens / effective_tpm / 60

    rate_per_day = n_chunks / (hours_static / 24)   # chunks/day achievable
    churn_per_day_chunks = n_chunks * churn_per_day
    if churn_per_day_chunks >= rate_per_day:
        raise Unschedulable("churn exceeds backfill rate; the tail never closes")
    days_real = n_chunks / (rate_per_day - churn_per_day_chunks)

    raw_gb = n_chunks * dim * bytes_per_dim / 1e9
    return {
        "spend_usd": round(spend, 2),
        "binding_limit": "tokens" if tok_capacity < req_capacity else "requests",
        "hours_static": round(hours_static, 1),
        "days_with_churn": round(days_real, 2),
        "peak_storage_gb": round(raw_gb * 2.4, 1),
    }

plan(80_000_000, 380, 0.02, tpm_limit=5_000_000, rpm_limit=2_000,
     batch=128, utilisation=0.70, churn_per_day=0.015)
# {'spend_usd': 608.0, 'binding_limit': 'tokens', 'hours_static': 144.8,
#  'days_with_churn': 6.68, 'peak_storage_gb': 786.4}

The last two lines of that function are one identity, and it is the real algebra of this topic. Call N the number of chunks that have to be embedded, R the rate at which the backfill completes them in chunks per day, and C the rate at which the corpus generates fresh work in chunks per day while the backfill is running. Over a day the backlog falls by R and rises by C, so it falls by R minus C, and that difference is the net rate at which the remaining work shrinks. Clearing N chunks at a net rate of R minus C chunks a day therefore takes N divided by that difference, which is T = N / (R − C). The subtraction is allowed because both terms are already in the same units — chunks per day — which is why the estimator converts an hours figure into a daily rate before touching churn at all.

Work the static case first, where C is zero and the identity collapses to N over R. The 145 hours from the throughput row is 145 divided by 24, which is 6.1 days, and eighty million chunks completed in 6.1 days is a rate of about 13.2 million chunks a day. That number is R, and every churn case below reuses it unchanged, because churn does not slow the embedding job down — it adds to what the job has to finish.

Now add churn. At 1.5% per day, C is 1.5% of eighty million, which is one point two million chunks a day, so R minus C is a little under twelve million and eighty million over twelve million is 6.7 days rather than 6.1. That is a denominator that has barely moved and a schedule that has barely moved with it, which is exactly why low-churn corpora make the churn term look ignorable. At 5% per day the term is four million a day, the denominator falls to nine point two million, and eighty million over that is 8.7 days instead of 6.1 — a 43% overrun that no one budgeted. Push the churn to 12% per day, as happens on log-like or ticket-like corpora, and the term is nine point six million a day, the denominator goes to 3.6 million, and the answer is 22 days.

Those four results trace out the shape of the identity, and the shape is not a straight line. As C climbs toward R the denominator approaches zero, and a fixed numerator divided by a denominator approaching zero grows without bound, so the time to completion diverges. The Unschedulable branch is not defensive programming for its own sake — it is that singularity, made into control flow. What it licenses is a decision rather than an alarm: past that point the schedule cannot be repaired by patience or by a slightly larger quota, because there is no completion date at any finite amount of waiting, and the migration strategy has to change rather than the schedule.

Whether the store can absorb the writes deserves a check even when it is not the constraint. Ingesting 80 million vectors into an HNSW index at a rate that does not disturb serving traffic usually means throttling to a few thousand inserts per second, and at 4,000 per second the write side needs eighty million divided by four thousand, which is twenty thousand seconds, which is 5.5 hours of the six days — so it disappears into the schedule. It stops disappearing if you decide to compress the embedding phase — provisioning a higher tier, or running several API keys in parallel — because a build that finishes embedding in 12 hours has to write 80 million vectors in 12 hours. Twelve hours is forty-three thousand two hundred seconds, and eighty million spread over that is 1,850 per second sustained, at which point the store's indexing threads are competing with query threads for the same cores. Speeding up one stage moves the constraint rather than removing it.

Every input to that function except the price is an assumption, and the utilisation figure is the one worth replacing with a measurement, because it is the only one nobody can guess from a contract or a schema. A pilot that embeds one million chunks with the production client, the production batch size and the production retry policy takes about half an hour, costs under ten dollars, and returns the number that matters: chunks per minute actually achieved, end to end, including the time spent backing off. Its bill closes on the same two multiplications as the full job — one million chunks at 380 tokens is three hundred and eighty million tokens, which is three hundred and eighty million-token units, and at $0.02 each that is well inside ten dollars. Teams who run the pilot usually find utilisation somewhere between 0.55 and 0.85, and they also find out whether their client's concurrency is high enough to saturate the quota in the first place, which is a separate and common shortfall — sixteen in-flight requests against a five-million-token quota leaves most of it unused no matter how patient you are.

A six-day job also has to survive six days, and that is an engineering requirement rather than an estimating one. Checkpointing needs to be at the granularity of a batch, with the last committed offset in durable storage, because an offset that survives the process is what lets a restart resume instead of replay — and a backfill that loses its place at hour 90 and starts over has turned a six-day migration into a twelve-day one. Poison records need a dead-letter path for the same kind of reason: a chunk that exceeds the model's context length, or contains a byte sequence the tokenizer rejects, fails its batch, so the batch is retried, so it fails again, so a worker stalls indefinitely on a record it will never accept — and on 80 million chunks there will be thousands of them. Rate-limit responses need backoff with jitter and a concurrency controller that lowers in-flight requests when 429s, the HTTP status a provider returns for too many requests, appear, because a 429 is the provider saying the quota is already spent and a client that keeps hammering at a fixed rate spends the next window's quota on rejections too. These three are the mechanics that turn an estimate into an achieved number, and what they have in common is that each one silently converts sending time into waiting time — which is why measured utilisation is so often half of what was assumed.

When the provider is not the constraint at all#

Running the encoder yourself changes the shape of the problem, because a self-hosted model has no rate limit and a throughput you can buy more of. A mid-sized encoder on a single modern inference GPU handles on the order of 500 to 1,500 chunks per second at 380 tokens each, depending on the model size and batching, so eight GPUs put you in the range of 4,000 to 12,000 chunks per second — meaning the same 80-million-chunk job finishes in two to six hours of GPU time rather than six days. Both ends of that range are the same division as before: eighty million over four thousand a second is twenty thousand seconds, which is between five and six hours, and eighty million over twelve thousand a second is under seven thousand seconds, which is under two hours.

The ~4 h in that table is the midpoint of the two-to-six-hour range the prose gives, which is also what the cost line is priced against: eight GPUs at $25 an hour for four hours is about $800. The costs land close enough that the decision is about schedule and operational appetite rather than money. Teams already running their own embedding infrastructure should reach for it; teams who would be standing it up for one migration are trading six days of waiting for an unknown number of days of debugging, and the six days are usually the better deal because they are predictable.

The quality gate, and the machinery inside the churn term#

None of these five numbers say anything about whether the new index is better. They size the work; the quality question is a separate evaluation that runs after the new index exists and before any traffic moves to it, and it is entirely possible to complete a flawless six-day backfill and discover that the new model is worse on your domain. Running the evaluation first, on a stratified sample of 200,000 chunks embedded with both models, costs about $1.50 and half a day, which makes it the cheapest gate in this entire chapter.

That $1.50 closes on the same arithmetic as everything else here, which is what makes it worth insisting on. Two hundred thousand chunks at 380 tokens each is seventy-six million tokens, a small fraction of a billion; seventy-six million tokens is seventy-six million-token units; and at $0.02 a unit the bill is about $1.50. Set that against the $608 and it is a rounding error, and set the half day against the six days and it is a rounding error there too. The gate that would have told you the migration was not worth running costs less than a quarter of one percent of the migration.

The estimate also assumes a corpus that sits still except for the churn term, and the churn term is doing an enormous amount of hidden work. It compresses "documents were inserted, edited and deleted for six days while we read from a snapshot" into a single rate, as if catching up were a matter of embedding a few million extra chunks at the end. Those writes have to be captured, ordered, and applied to an index that did not exist when they happened, and a delete issued on day two against a document the backfill has not yet reached needs somewhere to go. That machinery is the difference between a plan and a migration.

The belief that gets teams into this is that a finance approval is a schedule approval, and it is worth stating in its strongest form, because the strong form is what makes it tempting. The spend estimate is a defence that is constant when the thing it must discriminate varies: $608 is the same $608 whether the job takes three days or twenty-seven, because a token count is the product of a chunk count and a mean length and neither of those depends on the rate at which the tokens are pushed. Anything that cannot tell a three-day job from a twenty-seven-day one cannot be used to pick a date, however precisely it is computed.

It is tempting because the bill is the one quantity in the plan with an authoritative unit price behind it, so it can be computed with confidence rather than estimated, and because it genuinely is the number finance needs. The four quantities that do discriminate are the ones in the other rows — sustained throughput after utilisation, storage at both generations plus merge space, the contiguous window actually available, and the catch-up tail from churn — and any one of them can be the binding one. Compute all five, and the date you name is the largest of the durations they imply rather than the smallest.