The bill is not the dimension ratio, at 50M chunks least of all

What a retrieval system costs to store is a separate question from what it costs to run and from how well it retrieves. A line item, throughout this chapter, is one named component of the bytes a vector store holds — the embedding itself, the text kept beside it, the metadata used for filtering, the graph the index walks — each with its own size per stored chunk and its own reason for growing. Nobody is billed per line item; a store charges for capacity and does not care which of your bytes fills it. The reason to break the bytes out anyway is that every proposal to cut the bill is a proposal against one line, and the only way to know what one line is worth is to know what fraction of the whole it is.

The arithmetic that goes wrong is the one everybody does first. Halving a vector's dimensions halves the vector, and a ratio of two to one on the vectors reads as a bill halved. Take a cluster of nine nodes with 256 GiB of RAM each, holding one collection of fifty million chunks embedded at 1024 dimensions in float32: halving the dimensions removes 102.4 GB per replica, which across three replicas is 307 GB. Whether 307 GB is half the bill, a fifth of it or nothing at all depends entirely on figures that are not in that sentence, and every figure below is read off one such cluster's own accounts.

Where 560 gigabytes per replica actually goes#

That cluster — Halvorsen's — held one collection, three replicas, nine nodes with 256 GiB of RAM each, and 2,304 GiB of provisioned capacity at an effective $4.08 per GB-month. Everything in the collection sat in that capacity, and the vectors were a minority of it.

Read the bytes-per-chunk column downwards: 4,096 for the vector, 2,800 for the stored text, 3,400 for the context header, 600 for the metadata, 264 for the graph and 40 for the bookkeeping. Those six add to exactly 11,200, which is the total row. There is no residual and no "other" line. A column that closes on its own total is worth more than any single row in it, because closing is what licenses subtraction: every gigabyte resident on the cluster has been attributed to a named component, so a saving claimed against one line can be checked against the whole rather than taken on trust. A table with a miscellaneous row cannot do that, since a proposal to remove one of its named lines might be removing bytes already counted inside the residual.

The other two columns are that first one multiplied out, and one row is enough to rebuild the rest. Four thousand and ninety-six bytes a chunk times fifty million chunks is two hundred and four billion eight hundred million bytes, and a gigabyte is a thousand million bytes, so that is 204.8 GB — the vector row's total per replica. Every other entry in that column is the same multiplication with a different first factor, which is why 2,800 bytes of stored text comes to 140.0 GB and 264 bytes of graph comes to 13.2 GB. The share column is then each of those totals divided by 560.0, the total per replica: 204.8 over 560 is 36.6%, and 13.2 over 560 is 2.4%. A table you can reconstruct this way is one you can defend in a budget meeting rather than repeat.

The context header is the item most teams do not expect to see there, and it is the one growing fastest. It exists because the chunk alone is not enough for the model to write a grounded answer, so the ingest pipeline attaches the document title, the section hierarchy, the policy's effective dates and a two-sentence summary of the preceding chunk. It is duplicated across every chunk of a document, so a 400-chunk policy wording carries the same 3.4 KB four hundred times, and it is larger than the chunk text itself for the many chunks that are short.

Getting that table is a measurement rather than an estimate, and it is worth an hour. Most stores expose a per-segment or per-field size breakdown — _cat/indices with store and per-field disk usage in Elasticsearch, the collection info endpoint plus segment files in Qdrant, pg_total_relation_size against pg_column_size samples in pgvector. Where the store will not tell you, sampling ten thousand points and measuring the serialised size of each component gives numbers within a few percent. The estimate that goes into a budget conversation should come from one of those, because the first table anyone writes from memory omits at least two rows.

Chunking decisions land in this table too, which is why they belong in the same conversation. Halvorsen chunked at 700 tokens with 120 tokens of overlap, so 17% of the stored text is stored twice, and the overlap also inflates the chunk count that every other row is multiplied by. Moving to 900-token chunks with 100 tokens of overlap would have taken the corpus — the fifty million stored chunks of their document set — from 50M chunks to 39M, removing 22% from every row at once, including the vectors. Whether it is a good idea is a retrieval question with its own evaluation, and it is a larger lever than any compression in this chapter.

Against 560 GB, the 102.4 GB from halving the dimensions is 18.3% of a replica: a hundred and two point four divided by five hundred and sixty. The dimension ratio said 50% and the line items say 18%, and the difference is entirely the rows that are not the vector.

Replicas, headroom and the capacity the invoice is written on#

Replication is the first of the multipliers nobody costs. It multiplies every row, which is obvious once written down and is regularly left out of the first estimate because engineers think in terms of the dataset rather than the deployment. Three replicas of 560 GB is 1,680 GB, and that is the whole collection as it actually sits on the nodes.

Now price it, because the answer looks wrong and is not. At $4.08 per GB-month, 1,680 GB comes to a little under six thousand nine hundred dollars a month, and the invoice was $9,400. That gap is not an error in the line-item table, and running it down is how you find out what the bill is charged against. Multiply the provisioned capacity instead — 2,304 GiB at $4.08 per GB-month — and the answer is about $9,400, the invoice to within rounding. And 2,304 is nine nodes of 256 GiB. The cluster is billed for the capacity it was given, not for the bytes it filled.

That reading puts a condition on the arithmetic in the previous section. Removing 102.4 GB per replica is 18.3% of a replica, and 18.3% of $9,400 is $1,720 a month — but only if the provisioned capacity comes down by the same 18.3%. On a cluster billed against what it was provisioned with, deleting bytes changes nothing on the invoice until somebody reduces the provision: retires nodes, moves to a smaller instance shape, or renegotiates the reservation. The saving is real and it is not automatic, and the step that converts bytes into dollars is an operational one rather than an indexing one.

The remaining headroom between 1,680 GB and 2,304 GiB is not slack either. Some of it is the reindex. Halvorsen built into a new collection and swapped an alias, for the reasons chapter 2 gave, where that swap is what stops queries from searching a half-migrated collection whose vectors are not mutually comparable. During any migration, then, the cluster holds two full copies, so a change that shrinks the collection still needs the capacity for old and new at once. The peak requirement during the migration that saves you 307 GB is the old 1,680 GB plus whatever the new collection takes, and the old copy cannot be dropped until the new one is serving traffic. Teams that size a cluster with 20% headroom discover this the week they try to use their savings.

HNSW construction also wants transient memory beyond the final structure, roughly 1.4 to 1.6 times the graph size while the layers are being assembled, and the build runs at whatever parallelism you configured. Snapshots are the multiplier that turns out not to matter much in dollars: fourteen days of daily snapshots to object storage came to about 2.1 TB after deduplication, which at $0.023 per GB-month is $48. They matter for restore time rather than cost, and a smaller collection restores proportionally faster, which is worth something on the day it is needed and nothing on the invoice.

The same arithmetic written as a function is how the options get generated, and how to read what it prints is worth a sentence.

CHUNKS = 50_000_000
REPLICAS = 3
DOLLARS_PER_GB_MONTH = 4.08

def monthly(vector_bytes: int, payload_bytes: int, graph_bytes: int = 264) -> float:
    per_chunk = vector_bytes + payload_bytes + graph_bytes + 40
    gb = CHUNKS * per_chunk / 1e9 * REPLICAS
    return gb * DOLLARS_PER_GB_MONTH

monthly(4096, 6800)            # 9,381  -- current
monthly(2048, 6800)            # 8,127  -- dims halved
monthly(1024, 6800)            # 7,500  -- int8
monthly(4096,  600)            # 5,588  -- text and header moved out
monthly(1024,  600)            #  3,707 -- both

Read that listing as a differencing instrument rather than as a source of absolute figures. Each call differs from the current one in exactly one argument, so the gap between two results is the saving from that one change: the current line down to the dims-halved line is the truncation saving, down to the int8 line is the int8 saving, and down to the text-and-header line is the payload saving. Those three gaps are three rows of the options table below, computed the other way round, and the comment on each line labels which argument moved.

The absolute figures sit at the scale of the invoice rather than at the scale of the resident bytes, and the section above is why they can: the cluster is billed against provisioned capacity, so the figure worth quoting to a finance partner is the one the invoice is written on. monthly(1024, 600) — both changes applied together — is the floor this collection could reach without touching the chunking or the model.

The options in order of risk#

Ordering by risk rather than by size is what turns an estimate into a plan, because the cheapest change to reverse should be attempted first.

Every figure in the monthly column is four factors multiplied together, and working one row in full makes the other four readable. Take the first: moving the chunk text and the context header out removes 6,200 bytes a chunk, which is the 2,800 bytes of chunk text plus the 3,400 bytes of context header from the line-item table. Six thousand two hundred bytes times fifty million chunks is three hundred and ten gigabytes per replica; times three replicas is 930 GB across the cluster; and 930 GB at $4.08 per GB-month is $3,794 a month. Bytes removed per chunk, chunk count, replica count, price per GB-month — every other row is those same four factors with a different first one.

Where each first factor comes from is the part worth checking rather than accepting. 2,048 is what halving 1024 float32 dimensions removes, since 512 coordinates at four bytes each is half of 4,096. 3,072 is what int8 removes, since a coordinate stored in one byte rather than four leaves 1,024 bytes where 4,096 were. 2,944 is the net figure for a binary index with int8 rescoring, and net is carrying weight there: the binary code is one hundred and twenty-eight bytes a document, and the int8 rescore vector measured in chapter 5, where a second stage reorders the candidates a binary first stage found, is another 1,024, so the pair costs a little over eleven hundred bytes where the float32 vector cost 4,096, and 4,096 minus those two is 2,944. The second row is the same arithmetic with the multiplier moved instead of the first factor changed: dropping to two replicas removes all 11,200 bytes a chunk once, rather than one component of them three times, which is 560 GB and $2,285.

The first row is the largest single saving in the table and it is the one with no retrieval consequence at all, because the text and the context header are never used for ranking. They are payload the search returns so the answer step does not have to make a second round trip. Moving them into the Postgres instance that already holds the canonical documents costs one indexed lookup on the ids the search returned, which Halvorsen measured at 12 ms p95 for a batch of 200 — inside the 340 ms budget the reranker already consumed. It also removes a consistency problem, since the text was being stored twice and could disagree after a document was corrected.

The second row is a decision about availability rather than about retrieval, which means it belongs to a different set of stakeholders and should be raised as such rather than quietly taken.

Those first two rows settle the whole exercise, and settling it is one addition. Three thousand seven hundred and ninety-four dollars plus two thousand two hundred and eighty-five is $6,079 a month, against a target of a $4,700 reduction. Each row is costed against the current configuration on the assumption that no other row has happened, which is the normal way to write such a table and is not how two of its rows compose: apply both and the payload is being removed from two replicas rather than three, so the combined figure is a little over four thousand eight hundred dollars. It clears $4,700 either way, and that is the finding — the two changes with no retrieval risk whatsoever were sufficient on their own, which makes every compression in this topic optional for this bill.

Every row also has a one-off cost that the monthly column does not show, and the shape of that cost is what orders the plan. Truncation is free to apply, since the prefix can be sliced from vectors already stored and no document needs re-embedding. Quantisation is nearly free for the same reason, once the calibration sample is drawn. Both are compute jobs over vectors you already hold, and both are reversible by rebuilding from originals that never left. Changing the chunking, or changing the model, means re-embedding fifty million chunks, which at current API rates for a 1024-dimension model and an average of 700 tokens per chunk is around $455 of inference plus two days of pipeline time. That asymmetry — a compute job on one side, a re-embed of the whole corpus on the other — is a good reason to exhaust the changes that operate on existing vectors before touching the ones that require regenerating them.

Rows three through five are the subject of this entire topic, and they sit below the two boring ones because each carries a quality risk that requires the evaluation apparatus the earlier chapters built. Doing them first is the common sequence and the wrong one: it puts the changes that can degrade search ahead of the changes that cannot, in service of savings that are smaller.

Why "just move to int8" does not clear the target#

Four-fold compression of the vectors removes 3,072 bytes per chunk, which is 27% of the 11,200-byte total and $1,881 of a $9,400 bill. It does not reach $4,700 and no combination of the compression rows reaches $4,700 on its own, because the compressible portion of the bill is 36.6% of it.

The ratio people quote and the ratio they are billed on are two different divisions, and doing both explicitly is what stops the gap being surprising. Int8 is a genuine 4x on the vector: 4,096 bytes become 1,024. What it does to a stored chunk is remove 3,072 of the 11,200 bytes, leaving a little over eight thousand. Divide the original per-chunk total by that remainder — eleven thousand two hundred over eight thousand one hundred and twenty-eight — and the answer is about 1.37. A technique that is genuinely 4x on the thing it compresses is 1.37x on the thing you are billed for, and the two numbers describe the same change.

The vector row's share caps the whole exercise, whatever technique is chosen. It is 36.6% of the bytes, so a compression that reduced every vector to nothing would remove 36.6% of the bill and not one point more, and every real scheme lands short of that because it leaves something behind — 1,024 bytes for int8, a hundred and twenty-eight for a binary code. The division to do first is the line item's share of the total, not its compression ratio.

There is a worse version of the misestimate, and it is not about ratios at all. It is the belief that enabling quantisation reduces the collection. A configuration flag that turns on int8 is a defence that stays constant while the thing it has to discriminate varies: it reports the same quantisation enabled whether the original float32 vectors were dropped or are still resident beside the int8 copy. In several stores the default is to keep them. Qdrant's scalar quantisation stores the int8 copy alongside the float32 originals by default, precisely so that the rescoring stage from chapter 5 has something to rescore with. Elasticsearch's int8_hnsw behaves the same way for the same reason.

The line-item arithmetic is the discriminator the flag is not. An int8 copy of a 1024-dimension vector is 1,024 bytes a chunk, which across fifty million chunks is 51.2 GB per replica of new data. Dropping the float32 originals would remove 3,072 bytes a chunk, or 153.6 GB per replica. Enabling it without also configuring the originals onto disk or out of the collection gives you the first of those without the second, so it increases the collection by 51.2 GB per replica rather than decreasing it by 153.6 — a change of direction, not of magnitude. What settles which of the two happened is the collection size measured before and after, never the setting.

The default is defensible, which is why it is dangerous. Rescoring against the originals is what was buying the recall, so a store that silently discarded them on a compression flag would be trading quality for bytes without being asked. The setting that produces the saving is a second setting, and it has a quality consequence rather than being free: with the originals on disk, every rescored candidate costs a read, and with them gone the rescoring stage has only the int8 copy to reorder with — the configuration the table's risk column assumes when it says the originals must be dropped and quotes recall@10 −1%. Reading the release notes for the specific version you are running is the only way to know which default applies.

The graph is the other row that does not move. Its 264 bytes per chunk are a function of M and the number of points, and it is unaffected by whether the vectors it indexes are four bytes wide or one bit. Both of the shares that follow are read the same way, as the graph's bytes over graph-plus-vector resident bytes, which is also how chapter 5 reads the same 6% when it calls the graph a small part of a float index's footprint. Against 4,096 bytes of float32 vector, 264 bytes of graph is 6% of the two together; against 1,024 bytes of int8, the same 264 bytes is 20% of the two together. After binarisation the graph is twice the vectors, at which point lowering M from 32 to 16 saves more than any further work on the vectors would.

Clusters where the vectors really are the bill#

Halvorsen's cluster prices every byte identically because everything is resident, and a deployment that separates memory-resident vectors from disk-resident payload has a completely different shape. There the vectors can be a third of the bytes and 95% of the dollars, and compressing them really does halve the bill — the same 4x on the same vectors, against a total the vectors dominate. The size of the lever is set by a division you have to do on your own invoice: dollars per byte per line item, read off what you are actually charged rather than assumed from someone else's architecture.

The provisioning reading flips the same way. A store that bills for bytes stored rather than for capacity provisioned pays the saving out automatically, and on one of those the $1,720 arrives the month after the compression lands with no operational step at all. Which of the two you are on is settled by comparing the invoice against the resident bytes for a month you already have: if they agree, deleting bytes moves the bill, and if the invoice is larger and steady while the collection grows, a capacity change has to be scheduled alongside the index change.

The per-chunk figures are averages across fifty million chunks, which is what capacity planning needs and stops being enough as soon as an option removes bytes from only some chunks. Dropping the context header from chunks that already carry their own headings saves 3,400 bytes on the affected fraction and nothing elsewhere, so costing it against the mean overstates it.

A saving is always a fraction of something, and most of the work is deciding what. The dimension ratio divides the vector by the vector and gets 2x or 4x, which is true and is a statement about a data structure. The bill divides by the line-item table, multiplies by the replicas, and then checks the result against the capacity the invoice is written on — three steps that take a 4x down to 1.37x and then, on a cluster billed by provision, to nothing at all until a node is retired.

So the order of operations matters more than the choice of compression. Measure the line items until the column closes on its own total, multiply by the deployment rather than the dataset, find out what the invoice is charged against, and only then rank the options, cheapest to reverse first and riskiest to retrieval last. Halvorsen's target was met by two changes that touched no vector at all, and the compression work the rest of this topic is about went ahead anyway, on its own merits, with six weeks to prove it.