Only the chunking axes force a reindex, and they set the real bill

A proposal to tune a retrieval pipeline over a corpus of 210,000 internal support and engineering documents lands in your inbox as four lines. Sweep 5 chunk sizes × 3 overlaps × 4 values of k × 3 rerank depths × 2 query transforms; score every configuration end to end with an LLM judge on the 500-query evaluation set; 360 trials; about a week. It asks for approval on the infrastructure budget and nothing else.

The number 360 is doing all the work in that proposal, and it is the wrong unit. Trials are not fungible. Changing k from 5 to 10 costs a few hundred milliseconds of extra retrieval per query. Changing chunk size from 512 to 384 means every document in the corpus has to be split again and every resulting chunk sent through the embedding model again, which for a 210,000-document corpus is several hours and a few hundred dollars before a single query has been answered. Pricing the sweep means separating the axes that touch the index from the axes that do not, and then noticing that the largest line item is neither of them.

A trial count is easy to accept as a budget because it is a real count of real runs, honestly arrived at. What makes it useless is that it does not move when the thing it stands in for moves. The cost of one trial varies across this grid by more than an order of magnitude, and the proposal reads the same trial count and the same week at every point in that range; replace the four values of k with four larger ones and the count is unchanged while the bill is not. A quantity that holds still while the dangerous property varies cannot be a check on that property, however carefully it was computed.

Two axes rebuild the corpus, three do not#

Chunk size and overlap are properties of the index, and the reason they are the expensive pair is worth taking one step at a time. An embedding is a vector computed from one specific span of text, and that span is what the vector describes, so a stored vector is attached to a chunk rather than to the document the chunk was cut out of. Change the chunk size and the spans change, which means every vector in the store now describes a span that is no longer a retrievable unit. Nothing is left to reuse, and the whole corpus goes back through the embedding model. Overlap moves the same boundaries for the same reason, and that is the sense in which those two axes and no others determine what text exists as a retrievable unit.

Everything else in the proposal is a query-time decision: k selects how many of the existing vectors you pull, rerank depth decides how many of those a cross-encoder scores, and the query rewrite happens before the vector store is touched at all. None of the three changes what a stored vector is, so none of them can invalidate one. The grid is therefore not flat. It is fifteen index builds, and inside each one, twenty-four query-time configurations that reuse it. Fifteen times twenty-four is 360, which is worth checking rather than assuming: the regrouping is a partition of the proposal's own count, not a rival way of counting it.

The shape of that diagram is the shape of the bill. Fifteen passes through the top half, 360 passes through the bottom half. Any accounting that multiplies the top half by 360, or that treats the bottom half as free because it does not build anything, is off by a large factor in one direction or the other.

What one index build costs, and where each column comes from#

Take a corpus of T tokens of source text, a chunk size of c tokens and an overlap of o tokens. Consecutive chunks start c − o tokens apart, because each chunk repeats the last o tokens of the one before it and then runs on until it is c long. That gap is the stride, and it is the amount of new source text each chunk brings in, so covering T tokens of source takes T / (c − o) chunks.

Each of those chunks is submitted to the embedding endpoint in full, and the endpoint charges per token of input on every submission it receives. A source token that falls inside two chunks is therefore paid for twice. What you are billed for is the total length of the chunks rather than the length of the corpus, and the total length of the chunks is their count multiplied by their size, which is c × T / (c − o), or T multiplied by c / (c − o). Multiplication is the right operation because every chunk is the same length and the charge is per token: if the chunks varied in length the bill would have to be a sum over them, and it is only because they do not that a single multiplier settles all fifteen builds at once.

Everything else in a row falls out of that one quantity. Cost is billed tokens multiplied by a price per token. Wall clock is billed tokens divided by a sustained rate. Chunk count is the same expression with the c taken back off, and index size is chunk count multiplied by the size of one stored vector.

Here is the top half with numbers for this corpus, which averages 1,800 tokens per document for 378M tokens total. Embedded token count is inflated by overlap, because a chunk of size c with overlap o means the corpus is covered by c / (c - o) tokens of embedding input per token of source. The rate assumed is $0.02 per million tokens for the embedding endpoint and a sustained 1.5M tokens per minute at the concurrency the team is allowed.

The corpus total every one of those rows is computed from is itself a product rather than a measurement: the document count multiplied by the average document length reproduces the token total with nothing left over. That is convenient and it is also a warning, because an error in the average propagates straight through every cost, hour and gigabyte in the table, and no cross-check inside the table would catch it.

The second row is where the multiplier is easiest to see working. At 256 tokens with an overlap of 128 the stride is half a chunk, so covering the corpus takes twice as many chunks as at 256 with no overlap while each chunk stays the same length: the count goes from 1.48M to 2.95M and the billed tokens from 378M to 756M. Both doublings come from the same halved stride.

The step most commonly got wrong is the size of the overlap surcharge, and getting it wrong always understates the bill. The fraction of a chunk that repeats its predecessor is o / c, which for an overlap of 128 on a 256-token chunk is one half, and it is natural to read that as adding a half to the bill. The multiplier is not one plus that fraction. It is one divided by one minus it, because repeated tokens come out of the stride and it is the stride that divides into the corpus. Half of each chunk repeated does not raise the embedding bill by half; it doubles it. The two readings agree closely while the overlap is small — an overlap of 128 on a 768-token chunk repeats a sixth of each chunk and costs a fifth more — and they part company without limit as the overlap approaches the chunk size.

Two of the columns in that table are the same column in different units, and noticing it closes one question for good. Cost is billed tokens times a price and wall clock is billed tokens over a rate, so both are the billed-token count scaled by a constant and their ratio is fixed. Divide any row's cost by its hours and the same figure comes back, a little under two dollars per hour of embedding, including on the totals row. What that licenses is a negative claim worth having: there is no per-build fixed cost anywhere in this table. A setup charge, a minimum billing unit or a per-request overhead would show up as drift in that ratio between the short builds and the long ones, and there is none. Any fixed cost a build carries therefore sits outside the table and has to be added by hand, which is exactly what the graph construction time further down turns out to be.

The index-size column closes against the chunk column the same way. Divide one by the other row by row and every row returns about four kilobytes per chunk, which is 1,024 dimensions at four bytes each and nothing else. So that column is pure vector payload: no stored text, no document identifiers, no neighbour graph, no per-chunk metadata. That is why the memory figure in the next paragraph has to be built on top of the column rather than read out of it.

Ninety gigabytes of vector storage is the line most proposals forget, and it is a real constraint rather than a rounding error: if you keep all fifteen indexes live so that the sweep can interleave them, you need the whole 90 GB resident at once, and at 1,024 dimensions in float32 with an HNSW graph on top — hierarchical navigable small world, the layered graph of neighbour links that lets a vector store find near matches without comparing the query against every stored vector — the memory footprint is closer to 130 GB. If instead you build one index at a time, run its 24 configurations and tear it down, you need 12 GB of headroom and you have serialised 84 hours of embedding into the critical path. That is a genuine design decision with a cost either way, and it is invisible if you are counting trials.

The 12 GB in that comparison is the largest single row of the index column rather than an average, because the sequential plan has to hold whichever build is biggest, and it is raw payload on the same basis as the 90 GB. The graph overhead that lifts 90 to 130 applies to it too, so the sequential plan's true headroom is above 12 GB by the same proportion. Requirement and design decision are worth keeping apart here. That changing chunk size costs a rebuild is a requirement: it follows from what an embedding is and no engineering removes it. Whether the fifteen rebuilds are held resident together or run one at a time is a decision, and it trades memory against elapsed time on axes that have nothing to do with retrieval quality.

The totals row cannot be audited by adding the column above it, because the table prints five of the fifteen builds. What can be checked is a bound. No overlap in the grid reaches half its chunk size except the 256 with 128 pairing, which reaches it exactly, so every build's multiplier lies between one and two and the fifteen together must bill somewhere between fifteen and thirty times the corpus. The printed total is a little over twenty times it, which sits inside that bound. It is worth knowing which way the printed sample leans, though: three of the five rows shown carry the heaviest overlap in the grid, where only a third of the fifteen builds do, so the displayed rows are a more expensive selection than the population they are summarising and the total should be read as an estimate rather than a sum.

The line item that actually dominates#

The 84 hours and the $152 are not what makes this proposal expensive. Judging is.

Every one of the 360 configurations is scored end to end on 500 queries, so the sweep performs 180,000 generations and 180,000 judge calls. Those two counts are not extra multipliers stacked on top of the sweep; each scored query is one generation and one judge call, so the sweep and the judging are the same 180,000 events counted twice over. At k=10 with 512-token chunks the retrieved context is about 5,100 tokens, the answer about 300, and the judge sees the query, the context and the answer for roughly 5,600 input tokens and returns 200. With the generator at $3 per million input and $15 per million output tokens, and a cheaper judge model at $1 and $5:

QUERIES = 500
CONFIGS = 360

def per_scored_query(ctx_tokens: int, answer_tokens: int) -> float:
    gen = ctx_tokens / 1e6 * 3.00 + answer_tokens / 1e6 * 15.00
    judge_in = ctx_tokens + answer_tokens + 200
    judge = judge_in / 1e6 * 1.00 + 200 / 1e6 * 5.00
    return gen + judge

unit = per_scored_query(ctx_tokens=5_100, answer_tokens=300)   # $0.0263
print(unit * QUERIES * CONFIGS)                                # $4,734
print(unit * QUERIES * CONFIGS / 4)                            # $1,183 at k=3-ish contexts

A scored query carries four charges and no others: the context read by the generator, the answer written by the generator, the same context and answer read again by the judge alongside its own prompt, and the judge's verdict. Sorting those four by whether they move with the context length is what makes the block usable beyond the one case it prices. The generator's reading of the context and the judge's reading of the context both scale with it. The answer the generator writes, the judge's own prompt and the judge's verdict do not — a shorter context does not make the model's answer shorter. Those two groups sum to the whole unit with nothing left over, which is what licenses treating them separately at all.

The consequence lands on the last line of the block. Dividing the whole unit by four prices a quarter-size context only if every charge is proportional to context, and a little under a quarter of the unit is not. Cutting the context to a quarter cuts the unit by rather less than three quarters, and the true figure for small-k contexts is something like two-thirds larger again than the line prints. The same split gives the floor, which is the more useful half: however small k gets, the judging bill cannot fall below the fixed group, so no amount of shrinking the retrieved context takes the sweep below roughly a quarter of its headline cost. The printed figures are also computed from a rounded unit and run a shade low against what the block would actually output, so read them as the round numbers they are.

Somewhere between $3,500 and $5,500 depending on how the context sizes fall across the grid, against $152 of embedding. So the answer to "is 360 trials 360 times the cost of one trial" is that it is fifteen times the cost on the index side and very nearly 360 times on the scoring side, and since the scoring side is thirty times larger to begin with, the sweep does scale close to linearly in trials — just not for the reason the proposer thinks, and not on the axis they were worried about.

That band has a centre, and it is not in the middle. The context priced in the block above is smaller than the grid's average, since the average of the four k values is a shade under 10 and the average chunk size is above 512, and the charge rises with context. So the honest estimate sits nearer the top of the band than the bottom of it.

That also tells you which lever actually reduces the bill. Halving the evaluation set from 500 queries to 250 halves the dominant term. Dropping end-to-end judging for the inner loop and scoring the 24 query-time configurations on retrieval metrics alone — nDCG@10 and recall@k against the labelled relevant passages, which cost nothing but compute — removes the generator and judge entirely from 336 of the 360 trials, then re-scores only the top handful end to end. That count is exact rather than approximate: the 24 configurations that keep their end-to-end scoring are subtracted from 360, and the rest lose it. That is a factor of ten, and it is the single change that turns this proposal from a quarter's budget into a fortnight's.

Elapsed time is a different resource from spend#

Wall clock and spend are different resources and the proposal has only estimated one of them, badly. The 84 hours of embedding is elapsed time on a rate-limited endpoint and does not compress by throwing more workers at it, because the sustained rate is the provider's ceiling rather than your fleet's. The 180,000 judge calls at, say, 40 concurrent requests and 6 seconds each are 7.5 hours of pure judging, which sounds fine until the provider's tier limits you to 20 concurrent and it becomes 15. Index construction after embedding — building the HNSW graph over 2.95M vectors — is another 40 to 90 minutes per build that nobody costed.

Those two judging figures are not in conflict and neither is a correction of the other. The load is the same in both: 180,000 calls of a few seconds each is a fixed amount of judge work, on the order of three hundred hours of it, and that number does not care how you schedule it. What differs is the capacity you are allowed to bring, and elapsed time is the load divided by the capacity, so halving the concurrency doubles the wall clock exactly. The embedding step is the case where that division is not available: the ceiling is on the work, not on your workers, so the load and the elapsed time are locked together. Recognising which of the two shapes a step has is what tells you whether more machines will help before you rent them. The graph construction figure needs the same care in the other direction, since it is quoted for the largest build in the grid and the smallest holds an eighth as many vectors, so the total across fifteen builds is well under fifteen times it and still a working day or two that the 84 hours does not contain.

Then there is everything that is not machine time. Somebody has to validate that the judge agrees with human labels on a sample before 180,000 of its verdicts are trusted, which is a day of labelling and an afternoon of analysis. Somebody has to notice that build 7 silently truncated documents over 8,000 tokens and rerun it. At least one axis will turn out to have been specified wrong — an overlap of 128 on a 256-token chunk was in the grid and means 50% redundancy, which is probably not what anyone intended — and re-running the affected slice is five more builds. A week of compute is a month of calendar in almost every team that has run one of these, and the honest version of the estimate says so.

The cost that arrives after the sweep ends#

The sweep produces a configuration that then runs on every production query for the next year, and its serving cost is a separate number from its evaluation cost. The parameters that win are frequently the expensive ones, because deeper reranking and larger k genuinely help the metric and neither of them shows up in an nDCG score. The comparison below reports p50 latency, the median query, which is the figure half of production traffic beats and the one most serving dashboards lead with.

The cost column decomposes cleanly, and it closes. Two of the three rows send the generator the same number of context tokens, so everything that scales with context is identical between them and the whole of their cost difference has to come from what else differs, which is rerank depth at double the value in one row. That difference prices twenty-five extra candidates at forty cents per thousand queries. Carry that rate and the constant the two rows share across to the winner, add the winner's much larger context at the generator's input rate, and what remains unexplained is a little under a dollar per thousand queries — which is the one component the winner has and the other two do not, the rewrite call. Nothing is left over. What that licenses is the claim that this column has four drivers and no fifth: context tokens, the answer, rerank depth and the rewrite, with no per-query platform charge hiding inside it, because a hidden constant would have to differ between the two same-context rows and it does not.

One caution comes out of the same arithmetic. The constant those two rows share is smaller than a 300-token answer would cost at the output rate used for the judging figures above, so the serving column is priced on different generator settings from the sweep's. The decomposition still closes on its own terms, and the two sets of rates should not be carried across to each other.

The winner costs three times as much to serve and is 700 ms slower, of which 220 ms is the extra cross-encoder work at depth 100, 310 ms is the rewrite round trip to a language model before retrieval even begins, and the rest is the longer prompt. That last term is a remainder rather than a measurement. Two mechanisms were measured and the third absorbed whatever was left, so the breakdown reproduces the 700 ms by construction and the only claim it supports is that the two measured mechanisms account for most of the gap — a little over three quarters of it — and that nothing else large is hiding behind the prompt term. It says nothing about how much of the gap the prompt length actually causes.

Two comparisons the table cannot support are worth naming. The current row and the runner-up carry the same context length but differ on three axes at once, chunk size and k and rerank depth, so the latency gap between them cannot be attributed to any one of the three. And every latency here is a median, which says nothing about the tail; a deeper cross-encoder is precisely the kind of component whose slowest queries degrade faster than its median does, so a p50 comparison is no evidence at all about what the worst-served fraction of traffic will see.

If it wins by 3.3 nDCG points that may be an easy trade; if it wins by 0.6 over the runner-up, the runner-up is the configuration you ship and the winner is a footnote. This is why the approval conversation has to name production latency as one of the outputs the sweep must report, before it runs — otherwise the sweep optimises a single scalar and the person reading the result has no basis for the trade.

A sweep's peak is not a capacity plan#

Experiment costing and capacity planning get conflated because they use the same units. Capacity planning asks what steady-state serving needs; this asks what a bounded experiment consumes and when the money is spent. The 90 GB of vector storage is a peak that exists for eleven days and then goes away, which is a very different procurement conversation from 90 GB that has to exist forever.

The estimate is also only as good as its worst-behaved assumption, and the one usually named is that all 360 trials cost the same to judge. They do not: context length is the product of k and chunk size, and both axes span a wide range, so the grid runs from three of the smallest chunks at one corner to twenty of the largest at the other. That is better than a twentyfold spread, a good deal wider than the modest one it is tempting to assume, and the k=20 with 1,024-token chunks corner sends 20,000 tokens to both the generator and the judge on every one of 500 queries.

What that spread does and does not break is worth separating, because the obvious conclusion is not quite right. It does not by itself invalidate an estimate built on an average, since the charge per scored query is a straight line in context length and the average trial therefore prices the whole sweep exactly — provided the average used is the grid's and not one convenient example's. The example priced earlier is smaller than the grid's average, and that, rather than the variation as such, is why the headline figure runs low. What makes the failure so reliable is that the average anybody actually reaches for is a representative-looking trial rather than a computed one, and a representative-looking trial is chosen for being easy to describe. Costing at the mean and discovering the grid is dominated by its most expensive corner is a common way to overshoot the estimate by 60%. Costing at the maximum and multiplying is how proposals get rejected that should have been approved.

None of this arithmetic decides whether to run the sweep. A $4,700 experiment on a system serving two million queries a month is trivially worth it if it moves quality at all, and a $4,700 experiment on an internal tool with forty users is not, and no amount of arithmetic about embedding throughput decides that. What it does is put the decision on quantities that move when the cost moves: reindex count against scored-query count, an embedding bill against a judging bill thirty times larger, elapsed time that parallelism compresses against elapsed time it cannot, and a serving cost that starts the day the experiment ends.

Approving the spend is the easy part of the decision, though, because you can at least check the arithmetic. The harder thing is that this sweep is going to produce a winner, and the winner is going to be recorded as a row of parameter values in a file — and a row of parameter values is not enough information to ever run that experiment again.