A clean 2D plot has already thrown the distances away

Dimensionality reduction is the operation that turns a corpus of 1,536-component vectors into something a human can look at, and every use of it in a retrieval project splits into two jobs that are routinely confused. One job is compression for production: store fewer numbers per document, accept some loss, keep the ranking mostly intact. The other is a picture for people: put every document somewhere on a page so a reader can see structure. The two are performed by different algorithms with different objectives, and only the first one is answerable to whether search still works.

The picture is the one that causes trouble, because it is persuasive and it is cheap. Six coloured blobs separated by clean white space, captioned "embedding space is well structured", retires the question of whether it is — and the question was going to be answered by a labelled query set that costs a day of somebody's time. What follows establishes why no arrangement of a 1,536-dimensional corpus on a page can carry that claim, what to measure instead, and which feature of such a plot is genuinely worth reacting to. The last of those is the reversal: the property that makes the slide look good is the property that should worry a retrieval engineer.

No projection to two dimensions can keep the distances#

A world map is the familiar version of this, offered as an analogy and not as a mechanism. The Mercator projection preserves angles everywhere, which is what makes it useful for navigation, and it pays for that by distorting area so severely that Greenland appears comparable in size to Africa while being about fourteen times smaller. No flat map of a sphere preserves distance, area and angle at once, and cartographers choose which to sacrifice according to what the map is for. Two things about the comparison are exact and one is not. Exact: something must be given up, and which thing is a choice. Also exact: a reader who does not know which property was sacrificed will read the map for the property it destroyed. Not exact, and this is where the analogy stops helping: flattening a sphere loses one dimension, and the plot on that slide loses 1,534. A map projection is also a fixed formula that runs in both directions, while the algorithms below are fitted, stochastic, and in one case cannot place a new point at all. The choice of what to sacrifice was made by a library's default parameters rather than by a cartographer with a purpose.

There is a bound on how much you can compress before distances break, and it is worth substituting into rather than citing. The Johnson–Lindenstrauss lemma says that projecting n points into k dimensions can preserve all pairwise distances within a factor of 1 ± ε provided k is around 8 ln(n) / ε². Put a documentation corpus of 50,000 chunks and a tolerance of 10% into it. The natural logarithm of fifty thousand is a shade under eleven. The tolerance enters as ε², and a tenth squared is a hundredth, so the division by ε² is a multiplication by a hundred. Eight times eleven is eighty-eight, and a hundred times that is getting on for nine thousand — the roughly 8,700 dimensions the lemma asks for.

That figure is larger than the 1,536 you started with, which is the lemma's polite way of saying this corpus is already compressed. It also closes off an argument before anyone makes it: you cannot appeal to Johnson–Lindenstrauss to license a further reduction, because the bound was already violated by the embedding model itself. Whatever guarantees the 1,536-dimensional representation carries, they come from what the encoder was trained to do rather than from a distance-preservation theorem. Two dimensions preserves nothing of the kind, and it is not a matter of degree — the requested width and the delivered width differ by more than three orders of magnitude. The distances on that slide are whatever the algorithm found convenient.

Both popular algorithms are explicit about this. t-SNE and UMAP — t-distributed stochastic neighbour embedding and uniform manifold approximation and projection, the two dimensionality-reduction methods behind almost every picture of this kind — optimise a local neighbourhood criterion, arranging points so that each one's near neighbours stay near, and they place no constraint at all on the distance between one cluster and another. That is a statement about their loss functions rather than an empirical observation: a quantity absent from an objective is a quantity the optimiser is free to set to anything, and inter-cluster distance is absent from both. Cluster sizes and the gaps between clusters in a t-SNE plot therefore carry no information, a point the algorithm's own documentation makes at length, and a reader looking at six well-separated blobs is reading exactly the two features the method does not compute.

The strongest demonstration is what these algorithms do to data with no structure whatsoever:

import numpy as np, umap

rng = np.random.default_rng(0)
X = rng.normal(size=(5000, 1536))
X /= np.linalg.norm(X, axis=1, keepdims=True)   # isotropic noise on the sphere

Z = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit_transform(X)

Five thousand independent Gaussian vectors, with no clusters in them by construction, produce a picture of a dozen or so well-defined islands with white space between them. The chain that produces them is short enough to follow. The algorithm's job is to find a two-dimensional arrangement where near neighbours stay near. In 1,536 dimensions with isotropic noise, every point's fifteen nearest neighbours are essentially arbitrary — the distances to its nearest and its five-hundredth-nearest neighbour differ by very little, so which fifteen win is decided by rounding rather than by structure. The algorithm nonetheless has to place those fifteen close on the page. Doing that for every point simultaneously is impossible in two dimensions, because the arbitrary neighbour sets conflict, so the optimiser settles for the best compromise available: pull mutually-agreeing groups into tight clumps and let everything else fall away. Islands are what a two-dimensional compromise looks like. A plot that looks clustered on noise cannot be used as evidence of clustering on data.

This gives a cheap control worth running whenever such a plot is going anywhere important: generate a matched sample of random unit vectors with the same count and dimension, run it through the identical pipeline and parameters, and put the two images side by side. Structure that is genuinely in the corpus survives the comparison, since the noise panel will show islands of similar size but no relationship at all between an island and any label you colour by. The n_neighbors and min_dist parameters change the number and tightness of the blobs in both panels, which is itself informative about how much of the shape belongs to the settings.

Measure what the reduction destroyed#

The honest way to use a reduction is to quantify what it cost. Nearest-neighbour overlap does that directly: take the k nearest neighbours of each point in the original space, take the k nearest in the reduced space, and report the mean size of the intersection over k.

from sklearn.neighbors import NearestNeighbors

def knn_ids(M, k=10):
    nn = NearestNeighbors(n_neighbors=k + 1, metric="cosine").fit(M)
    return nn.kneighbors(M, return_distance=False)[:, 1:]

def overlap(A, B, k=10):
    a, b = knn_ids(A, k), knn_ids(B, k)
    return float(np.mean([len(set(x) & set(y)) / k for x, y in zip(a, b)]))

print(overlap(V, Z_umap))     # 0.11
print(overlap(V, V_pca256))   # 0.83

Nine per cent of each chunk's true neighbourhood survives the trip to two dimensions, and the picture still looks orderly, because looking orderly and preserving neighbourhoods are different objectives. The t-SNE row cannot even be evaluated end to end, since t-SNE has no transform for points it was not fitted on, so a query cannot be projected into the same plot without refitting the whole thing.

The last column of that table carries no independent information, and checking that is what makes the other two columns trustworthy. Fifty thousand chunks at 1,536 float32 components is fifty thousand times 1,536 times four bytes, which is the 307 MB in the first row. Every other entry is that figure scaled by how many bytes per vector the row keeps: 256 dimensions is a sixth of 1536, and a sixth of 307 is the 51 quoted; 64 is a twenty-fourth, and a twenty-fourth of 307 is the 13; int8 keeps every dimension at one byte instead of four, so a quarter of 307 is the 77; two dimensions out of 1536 is the 0.4. No row carries a residual, which means storage is settled by arithmetic before anything is measured, and the only quantities in the table that had to be observed are overlap and recall.

Those two are worth reading against each other rather than down. Overlap falls from 1.00 to 0.83 and recall falls by two points; overlap falls from 0.83 to 0.51 and recall falls by fifteen; overlap collapses to 0.11 and recall is still 0.24, which is far above the nothing that a scrambled neighbour list might suggest. So the two columns are not the same measurement at different scales. Overlap asks whether the exact ten nearest neighbours are the same ten, which a small perturbation breaks; recall asks only whether one particular passage is somewhere in the top ten, which survives a great deal of reshuffling. Overlap is therefore the harsher instrument and the earlier warning, and it overstates the damage to retrieval — useful as a cheap screen, misleading if quoted as though it were a retrieval number.

A difference between the methods matters for anything beyond a picture. PCA — principal component analysis, which keeps the directions along which the corpus varies most and discards the rest — is a linear map, so the same matrix that reduced the corpus reduces an incoming query, and the result is deterministic and cheap. UMAP fits a model that can transform new points but does so approximately and with parameters that change the layout, and t-SNE has no out-of-sample transform at all. For a reduction that lives in the serving path, the ability to project a query with the same transform the corpus used is a hard requirement, which rules out most of what produces attractive pictures.

The middle rows are the ones with practical value. PCA to 256 dimensions costs two points of recall for a sixfold reduction in index size, and int8 quantisation costs almost nothing for a fourfold reduction, which are trades worth making deliberately and worth measuring exactly this way. The measurement procedure is identical whether the reduction is for a plot or for production; only the answer differs.

What to substitute in a review deck follows from the same table. A labelled query set — 120 queries taken from real traffic, each with the passages that answer them marked by someone who knows the product — gives recall@10 of 0.78 with a bootstrap 95% interval of ±0.07, that interval obtained by resampling the query set with replacement and re-scoring, alongside the polarity-correct@1 figure from the previous chapter. That is four numbers with error bars, it costs a day of labelling, and it answers the question a plot gets captioned with.

The labelling is the part people try to skip, and the shortcuts available are worse than they look. Generating synthetic queries from each chunk with a language model produces a set on which retrieval scores far too well, because the query was written from the passage and inherits its vocabulary; measured on this corpus, synthetic queries gave recall@10 of 0.94 against 0.78 for real traffic. That gap runs the wrong way to be harmless: the shortcut does not add noise, it adds optimism, so it will report an improvement for any change that increases vocabulary overlap and stay silent about every change that matters to a user who phrases things differently. Real queries with real judgements are what makes the number movable and comparable across changes, and 120 of them is enough to detect a five-point regression, which is the size of change any of the decisions in this topic will produce.

The thing in the plot that should worry you#

Endorsing such a plot as weak evidence is the wrong concession, because the feature that makes it attractive is a warning sign. Six document types separating cleanly means the largest sources of variation in the space are format, register and boilerplate rather than subject matter. If every one of those types discusses the same product, a well-behaved space would show topical structure cutting across the type colours, with the runbook about failover sitting near the API reference page about failover.

The consequence is measurable and it is bad for retrieval:

same = mean_cosine_within_group(V, doc_type)     # 0.71
cross = mean_cosine_across_groups(V, doc_type)   # 0.38
q_type = nearest_group(embed(queries))           # support-ticket style
print((q_type == "support_ticket").mean())       # 0.86

Queries written in the register of a support ticket land in the support-ticket region, and 86% of them retrieved a ticket as their top hit even when the answer lived in the API reference. The gap between a within-type cosine of 0.71 and a cross-type 0.38 is what makes that happen, and the arithmetic is worth stating as a handicap rather than as a gap. A document of the query's own type starts, on average, a third of the cosine scale ahead of a document of any other type. Relevance has to make up that entire deficit before a cross-type document can even tie. A query therefore has to be dramatically more relevant to a document of another type to outrank a mediocre match of its own type, and the clean white space on the slide is a picture of that handicap.

Two other shapes in such a plot are worth reading. A very dense knot of points, tighter than any real cluster, is usually boilerplate — cookie banners, navigation, licence headers, the same document footer chunked forty thousand times — and those chunks will either flood results or occupy index space for nothing. Points scattered far from any cluster tend to be extraction failures: tables flattened into columns of numbers, pages where the PDF parser emitted ligature garbage, chunks that are a single heading. Both are ingestion defects with a specific fix, and the plot found them.

That is what these plots are for. Coloured by source, ingestion date or document type, they are a corpus hygiene tool that surfaces duplicated ingestion runs, a missing document type, and a chunker that produced ten thousand near-identical fragments. Each of those is a question about which points exist, which is a question a projection can answer, since a duplicated ingestion run is visible as a doubled clump however the distances were mangled. Retrieval quality is a question about how points are ordered against a query, and ordering is precisely what the projection discarded.

Where a picture of a space is genuinely misleading#

There is a further trap in the middle rows of that table. Reducing dimensions for storage and then plotting the reduced vectors gives a plot that agrees with production, and it is still the plot's local structure you are looking at rather than its distances. Consistency with production does not upgrade the evidence. The plot of a PCA-256 corpus is a plot of the same 1,534-dimension loss applied to a corpus that has already lost a bit; agreeing with the index means the two share a preprocessing step, which says nothing about whether either one preserves what a ranking needs.

The noise control has a limit of its own, and it is worth naming because the control is the strongest tool in this chapter. It establishes that clumping is not evidence, and it cannot establish that clumping is not structure. A corpus with real clusters in it will also produce islands, and the two panels are told apart by whether an island corresponds to a label rather than by how crisp it looks. So the control converts an unfalsifiable picture into a falsifiable one; it does not convert it into a measurement, and the measurement is still the overlap number and the labelled query set.

The deeper limitation is that every number in this chapter describes one corpus under one model. Overlap@10 of 0.83 for PCA-256 is a property of these vectors; the same reduction over a different encoder's output can behave differently, because how much of the variance sits in the leading components depends on the model. A model whose corpus varies along a few strong directions loses little at 256 dimensions, and one whose variance is spread evenly loses a great deal, and nothing about the two is visible from a model card.

Nothing here transfers by assumption, which is a specific instance of a more general fact about every measurement so far — the cosines, the thresholds, the coordinate correlations, the recall figures. Each is calibrated to one model and one index, and the day someone swaps the model is the day that stops being an abstract caveat.