int8 and binary change the distance kernel, not merely the byte count

Quantisation replaces a coordinate held as a 32-bit float with one drawn from a small fixed set of levels, and three things define it. A scale is the width, in float units, of one level. A zero point is the integer level that stands for a float value of zero. A rounding rule settles where a value falling between two levels goes, which throughout this chapter is round-to-nearest. Put together, they say that a coordinate is stored as the integer nearest to that coordinate divided by the scale, and read back as that integer multiplied by the scale again. Push the idea to its limit and one bit per coordinate is left: keep the sign of each coordinate, discard its size, and the comparison between two documents becomes Hamming distance, the count of bit positions at which two codes disagree.

A corpus — the documents you have embedded and stored, one vector each — is read out of memory on every query, so its size sets the storage bill and often the latency as well. Against that, int8 quarters the bytes and binary cuts them to a thirty-second, which is why the idea is attractive enough to reach for before anyone has worked out what it costs. Neither saving arrives the way a compression ratio implies. The scale has to be estimated from your own coordinates rather than assumed, the sign threshold has to sit where your own coordinates divide rather than at zero, and the distance function has to be reimplemented against the new bytes. A store that accepts the smaller vectors and quietly keeps computing float distances over them will return ten plausible neighbours per query and raise nothing.

A scale is a claim about your vector distribution#

Choosing a scale is choosing which interval of float values the integer levels will cover, and the tempting choice is the one that covers everything. Take the smallest and largest coordinate in a sample, map that interval onto the levels available, and nothing saturates at either endpoint, so nothing has been thrown away. The reasoning breaks at its second step. The levels are a fixed budget, spread evenly across whatever interval you hand them, so widening the interval to admit a handful of extreme coordinates makes every level coarser for all the rest. Clipping an outlier damages the ordering of a few thousand coordinates; coarsening the levels damages the ordering of every coordinate there is.

The estimate that survives contact with real embeddings therefore clips rather than covers: pick the 0.1st and 99.9th percentile of the coordinate distribution instead of the extremes, accept that a few thousand coordinates saturate at the ends, and spend the 256 levels on the range where the mass actually is.

import numpy as np

def calibrate(sample: np.ndarray, lo_pct: float = 0.1, hi_pct: float = 99.9) -> np.ndarray:
    """sample: (n, dims) float32. Returns a per-dimension symmetric scale."""
    lo = np.percentile(sample, lo_pct, axis=0)
    hi = np.percentile(sample, hi_pct, axis=0)
    bound = np.maximum(np.abs(lo), np.abs(hi))          # symmetric: zero point stays at 0
    return (bound / 127.0).astype(np.float32)           # (dims,)


def quantise(v: np.ndarray, scale: np.ndarray) -> np.ndarray:
    return np.clip(np.rint(v / scale), -127, 127).astype(np.int8)


def dequantise(q: np.ndarray, scale: np.ndarray) -> np.ndarray:
    return q.astype(np.float32) * scale

Two choices in that function are doing most of the work. Per-dimension scales matter because embedding coordinates are not identically distributed: at Halvorsen Mutual, an insurer whose internal search covers fifty million document chunks at 1024 dimensions, the per-dimension standard deviations across the corpus ranged from 0.008 to 0.19, a factor of twenty-four. A single global scale has to be wide enough for the broadest of those dimensions, which leaves a narrow dimension occupying only a small fraction of the levels and the rest of its budget permanently unused — so one global scale wastes most of the resolution on the narrow dimensions. Symmetric quantisation — zero point fixed at zero, signed int8, scale only — matters for a reason that shows up in the distance function rather than in the accuracy, and the next section is where that reason becomes visible.

Calibration is also a versioned artefact rather than a constant. The scales are fit on a sample of your corpus at a moment in time, and every vector quantised with one set of scales is incomparable with vectors quantised under another, because the same integer means a different float value under each. A backfill that recalibrates halfway through produces an index whose two halves disagree, which is the same class of failure as the truncation mismatch from chapter 2, where a query and its documents were cut and re-normalised under different conventions, and it is just as silent. Store the scale array with the index, hash it, and refuse to write vectors quantised under a different hash.

The kernel has to know what the bytes mean#

Under symmetric quantisation a float coordinate is approximated by s·q — one scale s, times one small integer q that is all the storage the coordinate gets. Write that for both sides of a comparison. A query has integer coordinates q_i and scale s_x; a stored document has integer coordinates r_i and scale s_y. Their dot product is the sum over every dimension i of (s_x q_i)(s_y r_i). Both scales are the same number in every one of those terms, so both come out of the sum as common factors, and the dot product of two quantised vectors is s_x · s_y · Σ q_i r_i.

That factorisation is why symmetric quantisation is worth insisting on. The two scales are properties of the index, fixed when it was calibrated and identical for every candidate a query is ranked against, so the only quantity that varies from one document to the next is the integer sum Σ q_i r_i. Both scales are positive, and multiplying every score by the same positive constant leaves the order of those scores untouched, so an implementation that ranks by the raw integer sum and never reconstructs a float ranks correctly.

Now let the zero point be non-zero, which is what an unsigned or asymmetric scheme does. Each coordinate is then approximated by a scale times an integer offset by a zero point, and the quantity inside the sum becomes Σ(q_i − z_x)(r_i − z_y), where z_x is the query's zero point and z_y the document's. There are four terms in that product where symmetric quantisation left one, and expanding the brackets shows what they are: the integer dot product you wanted, plus three cross terms. The three are a term in z_x z_y that is the same number for every document, a term −z_y · Σ q_i that depends only on the query's own coordinates, and a term −z_x · Σ r_i that depends only on the document's.

The first three are harmless. A constant, and a quantity fixed by the query, shift every candidate's score by the same amount and cannot reorder anything. The fourth is the poisonous one. −z_x · Σ r_i is the query's zero point multiplied by the sum of the document's own integer coordinates, and it does not depend on the query's direction at all — a document with a large coordinate sum gains the same bias whatever is asked of it. An implementation that computes the raw integer dot product and calls it a score has silently added a per-document bias proportional to that sum. It ranks, it returns results, and it is wrong in a way that correlates with document properties.

The accumulator width follows from arithmetic small enough to check by hand. Each factor in a signed 8-bit product has a largest magnitude of 127, so products in an int8 dot product reach 127 × 127 = 16,129. A 1024-dimension vector contributes 1024 such products, and 1024 of them sum to at most 16.5 million. A 32-bit signed integer holds values into the low billions, so an int32 accumulator is sufficient with room to spare and no saturation logic is needed anywhere in the loop. That headroom is what allows a single hardware instruction to accumulate int8 products into int32 lanes without any per-step checking: the corresponding instruction on recent x86 is VPDPBUSD from the AVX-512 VNNI extension, which multiplies four int8 pairs and accumulates into int32 per lane.

There is a third option that most engineers meet only after they have built the symmetric one, which is to quantise the documents and leave the query in float. The query vector is a single array held in cache for the whole search, so storing it as 4 KB of float32 costs nothing at fifty million documents, and comparing a float query against an int8 document recovers most of the rounding error that would otherwise be introduced twice. FAISS and ScaNN both offer this asymmetric path, and on Halvorsen's corpus it bought 0.6 points of recall@10 over symmetric int8 at the cost of a slower kernel, since a float-by-int8 product cannot use VPDPBUSD and falls back to conversion plus FMA, the fused multiply-add that floating-point dot products are built from. Whether that is worth it depends on whether the search is bound by distance computation or by memory traffic, which is a property of the index size relative to cache rather than of the format.

Binary is a different function entirely. Taking the sign of each coordinate turns a 1024-dimension vector into 128 bytes — one bit per coordinate, eight bits to the byte — and similarity becomes Hamming distance, the count of bit positions at which two codes disagree, computed as XOR followed by POPCNT, or VPOPCNTDQ where available. The connection back to cosine is the random-hyperplane result: for vectors whose coordinates are roughly isotropic, meaning spread with no direction favoured over any other, the expected fraction of differing sign bits is θ/π, where θ is the angle between the two original float vectors and π is a half-turn. Read that from right to left and it says the disagreement rate estimates an angle expressed as a fraction of a half-turn, which is the form the following three steps undo.

Take a Hamming distance of 180 out of 1024 bits. Dividing the differing bits by the total gives a fraction of about 0.176, which is the estimate of θ divided by π. Multiplying by π turns that fraction of a half-turn back into an angle: π × 0.176 = 0.552 radians. Taking the cosine of that angle gives about 0.85, which is the cosine similarity the two float vectors would have had. Each step has one job — the division produces a rate, the multiplication converts a rate into an angle, the cosine converts an angle into the similarity the rest of the system expects.

Two conditions on that chain are worth carrying away with it. The relation is an estimator with variance rather than an identity, so two document pairs sitting at exactly the same true angle will produce different Hamming distances, and the closer two candidates are to each other the more often the estimate puts them in the wrong order; that variance is what chapter 5, where a second stage rescores the candidates a binary index returns, is about. The isotropy condition is the other, and it is not free: chapter 1, where anisotropy and a handful of outlier dimensions were what made an untrained slice unsafe, described exactly the property this relation needs and real embedding spaces do not have. The estimate is usable in practice and it is leaning on an assumption your vectors only approximately satisfy.

def to_bits(v: np.ndarray, centre: np.ndarray) -> np.ndarray:
    """v: (n, 1024) float32, centre: (1024,) corpus mean. Returns (n, 128) uint8."""
    return np.packbits(v > centre, axis=-1)


def hamming(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    return np.unpackbits(a ^ b, axis=-1).sum(axis=-1)

The corpus mean is the threshold, not zero#

Thresholding at zero assumes each coordinate is centred there, and embedding coordinates are not. Before centring, 61 of Halvorsen's 1024 dimensions produced a 1-bit for more than 97% of documents, because the underlying coordinate had a mean well above zero. Sixty-one divided by 1024 is a little under six percent, so six percent of every stored code was a bit that almost never varied.

A near-constant bit contributes nothing, and the reason is in the definition of the distance. Hamming distance counts the positions at which two codes disagree, so a position holding the same value in almost every document also holds the same value as almost every query code at that position, whatever the query is. It contributes zero to almost every comparison, and a term that is zero in almost every comparison cannot separate one document from another. Those 61 bits were six percent of the index doing no work.

Centring fixes that by moving the threshold rather than by changing the vectors' meaning. Subtracting the corpus mean before taking the sign puts each coordinate's decision point where that coordinate's mass actually divides, so roughly half the documents land on each side of it and the bit starts carrying close to a full bit of information. On Halvorsen's index that move brought the minimum per-bit entropy up from 0.06 to 0.71 bits and raised binary-stage recall@100 by nine points. That nine-point figure and the eight-point gap between the two binary rows in the table below are two different measurements — recall@100 for the binary stage on its own, against recall@10 expressed as a fraction of exact float — and both are correct.

The centre vector is another versioned artefact that both sides must share, and the query has to be centred with the same vector the documents were. This is the same discipline as the truncation and normalisation conventions from chapter 2, where a prefix and the query it is compared against have to be cut and rescaled identically, and it fails the same way when only one side is updated.

By this point the pipeline has four transformations that must be applied in an agreed order, and the order is a chain of dependencies rather than a list. Truncate first, because everything downstream is fit on whatever width you settled on: a scale, a mean and an evaluation all describe one specific dimension count and none of them transfer to another. Normalise second, so that the vectors going into calibration are on a common scale. Doing this the other way round, calibrating on unnormalised prefixes, would inherit the per-document magnitude variance from chapter 2, where slicing a unit-norm vector left every prefix with a length of its own, straight into the scale estimate — the percentiles would then be measuring how long the documents happened to be as much as where their coordinates sit. Centre third, using a mean computed over already-normalised prefixes. Quantise last, since it is the only step that needs both the width and the two fitted artefacts.

Reversing centring and normalisation in particular is self-defeating, and the reason is worth spelling out. Subtracting a mean from a unit vector produces a vector that is no longer unit length, so normalising afterwards rescales every document by a different amount, which moves the coordinates the mean was computed from, which makes the mean you centred by wrong for the vectors you ended up with. The mean becomes a moving target, and no single pass converges on it. Writing that order down once, as a single function that both the indexing job and the query path import, is the cheapest insurance available in this entire topic.

Most vector databases now ship these formats natively, which changes what you implement without changing what you have to verify. Qdrant exposes scalar and binary quantisation as collection settings with a configurable quantile for the clipping and an always_ram flag for the quantised copy. Elasticsearch and Lucene offer int8_hnsw and bbq_hnsw index types, the latter being a better-quantised binary scheme with its own rescoring built in. pgvector added halfvec for float16, and bit with hamming_ops for binary. Using the built-in is almost always right, and it moves the risk rather than removing it: the calibration quantile, the centring convention and whether the store keeps the original vectors are now defaults chosen by someone who has never seen your corpus, and the only way to find out what they did is to read the release notes and measure.

Bytes, kernels and what each format actually buys#

Two rows in that table share a byte count and differ by twenty-five points of recall, and two more share a byte count and differ by eight. The compression ratio predicts the storage line and nothing else.

Latency is its own question, and the answer depends on whether the kernel exists. A 1024-dimension float32 dot product takes roughly 120 ns on a modern core; the int8 VNNI path runs it in about 35 ns and a 1024-bit Hamming distance in about 6 ns. An HNSW search visiting two thousand nodes therefore spends 240 µs, 70 µs or 12 µs on distance computation respectively — before the graph traversal, the memory latency of chasing neighbour lists, and the priority-queue bookkeeping, none of which shrink at all. Where the library has no integer kernel, it dequantises each stored vector to float before comparing, which costs about 210 ns per comparison and makes the int8 index measurably slower than the float one it replaced while still saving the disk.

Checking against something that cannot be wrong#

Every claim in this chapter is checkable in an afternoon with one procedure: take five hundred queries, compute exact float32 similarities against the full corpus by brute force, keep the true top-100 per query, and measure what the compressed index returns against that. Brute force over fifty million 1024-dimension float vectors is 204.8 GB of streaming per query, four bytes for every one of the components involved, which is impractical live and entirely practical offline on a 2% sample of documents plus the known-relevant ones. The point of the exact baseline is that it has no tuning parameters and no failure modes, so a discrepancy is unambiguously the compressed path's fault.

Comparing against the previous production index instead is the tempting shortcut, and it is the defect worth naming, because it is a defence that stays constant while the thing it is meant to discriminate varies. The production index is itself approximate — HNSW with a finite ef_search, the parameter bounding how much of the graph a single query is allowed to explore, misses documents — so overlap against it can be high because the compressed index is faithful, or high because the compressed index is making the same mistakes the graph already makes. Those two situations are the ones the measurement exists to tell apart, and it returns a similar number for both.

Halvorsen's uncentred binary index measured 0.94 overlap against their HNSW float index and 0.79 against exact search, and the fifteen-point difference was the part that mattered: it is precisely the region where the two approximations agree with each other and disagree with the truth, which is the region a comparison against production cannot see by construction. Against the exact baseline that region has nowhere to hide, because exact search has no ef_search, no graph and no calibration of its own to blame.

The two experiments in the scenario above are the same two measurements the table reports, written as ratios. The well-calibrated index scored 0.818 against a float baseline of 0.826, and 0.818 divided by 0.826 is 0.99, which is the per-dimension clipped scale row. The badly calibrated one scored 0.61 against the same baseline, and 0.61 divided by 0.826 is 0.74, which is the global min/max row. That identity is what licenses reading the table as evidence: those rows are not published benchmark figures imported from a paper about int8, they are one corpus's own two runs at the same format, on the same bytes, through the same kernel. The twenty-five-point difference between them is therefore a claim about a calibration choice and about nothing else, which is exactly what makes the headline true — the compression ratio predicts the storage line, and the scale, the threshold and the kernel predict the rest.

The binary row of that table is where the interesting engineering is, because 0.87 of float recall is a serious loss and the published claims for binary embeddings are far better than that. The claims are honest, and they are not describing an index like this one.