The encoder was trained to pull paraphrases together and never shown a negation

Two embedding models given the same corpus can disagree about which documents are near each other, and neither of them is wrong. The geometry each produces is the outcome of a training run, and a training run has two inputs that decide everything about the result: a loss function that says what "near" is supposed to be rewarded for, and a collection of text pairs that says which texts are supposed to be near. Both are published for every model discussed here. Reading them turns model selection, and most of the surprises that follow it, from taste into prediction.

An encoder, for the rest of this chapter, is the half of a transformer that turns a sequence of tokens into vectors, with no text-generating half attached. The dominant recipe for training one is contrastive: take a batch of pairs that are supposed to be related, encode both sides, and adjust the weights so that each item's partner scores higher against it than every other item in the batch does. The word carries the whole design. Nothing in the procedure ever tells the model what a sentence means in isolation. It only ever tells the model which of several candidates is the right partner, so what the model learns is whatever discriminates the right partner from the wrong ones — and the wrong ones are chosen by whoever built the batch.

What the embedding space was optimised to preserve#

The loss is usually InfoNCE, which is a softmax cross-entropy over similarities with the correct partner as the label:

import torch
import torch.nn.functional as F

def info_nce(anchors, positives, temperature=0.05):
    a = F.normalize(anchors, dim=-1)     # (B, d)
    p = F.normalize(positives, dim=-1)   # (B, d)
    logits = (a @ p.T) / temperature     # (B, B) similarity matrix
    labels = torch.arange(a.size(0), device=a.device)
    return F.cross_entropy(logits, labels)

The whole geometry follows from those four lines, and each line is worth reading for what it commits the model to. F.normalize rescales every vector to length one, so the products that follow are cosines rather than raw dot products — the magnitude of a vector is divided out before the loss ever sees it, and a quantity a loss cannot see is a quantity gradient descent has no reason to arrange. a @ p.T is a matrix multiplication in which the entry at row i, column j is the cosine between the i-th anchor and the j-th positive, so the matrix holds every pairing of every item in the batch against every other. labels = torch.arange(...) says that for row i the correct column is i, which is exactly the statement that the true partner of an anchor is the positive that arrived with it. And cross_entropy takes a softmax across each row and penalises the negative log of the probability it assigns to that correct column.

Follow what minimising that quantity requires. The softmax across a row divides the exponential of the diagonal entry by the sum of the exponentials of all the entries in that row, so the probability can be raised in two ways: push the diagonal entry up, or push the other entries down. Gradient descent does both at once, on every row, at every step. So each step raises the cosine of each true pair and lowers the cosine of each item against the other B−1 items that happened to share its batch.

Counting those two effects is what explains the resulting space. The similarity matrix has as many entries as the square of the batch size, and only as many of them as the batch size itself lie on the diagonal. Every remaining entry is treated as a pair that ought to be pushed apart. With a batch size of 8,192, which is typical for these models and is why they are trained on hardware with a lot of memory, every step pushes each text away from 8,191 randomly drawn other texts and towards exactly one. The gradient budget is spent on separation in that ratio, and separation from a randomly drawn other text is achieved by learning what a text is about.

The temperature of 0.05 in that snippet is worth noticing, because dividing by a small number sharpens the softmax and makes the loss dominated by the hardest negatives in the batch. Dividing by five hundredths is multiplying by twenty, so a cosine gap of a hundredth between two candidates becomes a logit gap of a fifth before the exponential is taken, and the exponential turns that into a noticeably larger share of the row's probability mass. The row's loss is then almost entirely decided by whichever off-diagonal entry sits closest to the diagonal one, and the many entries far below it contribute almost nothing. Models trained at low temperature end up with scores squeezed into a narrow band near the top of the range, which is one reason a cosine of 0.71 means something entirely different from one model to another.

Where the pairs come from, and what each source teaches#

The loss says the two sides of a pair should end up close. What ends up close in practice is therefore decided by whoever assembled the pairs. These collections are large, public and specific: several hundred million pairs, scraped and filtered, of a handful of recognisable shapes.

The third column and the fourth column are the same fact stated twice, and the fourth is the one nobody reads. A property appears in the third column when some pair in the source differed along it and the loss was therefore forced to represent it; a property appears in the fourth when the source is silent about it, so no gradient ever touched it in either direction. Run down the fourth column and notice which kind of thing keeps appearing: correctness, whether the answer is right, direction of a claim, contradiction. Every one of those is a relation between two texts rather than a topic either text is about. The column is not a list of oversights. It is a list of the things a pair of texts drawn from the open web is never labelled for, because labelling them requires a judgement that scraping cannot produce, and a pipeline that harvests hundreds of millions of pairs cannot afford a judgement per pair.

That structure predicts where a model will disappoint before anyone measures it. A model trained on a corpus heavy in web question-and-answer text does well on help-centre prose, where a user's informal phrasing has to reach a written article, because that is precisely the shape of a StackExchange pair. The same model does poorly on a corpus of stack traces and function names, where the useful signal is exact identifier overlap, because no pair in its training data ever taught it that TimeoutError and TimeoutException are different things worth keeping apart. The chain runs: the pair source contained no example whose two sides differed only by an identifier → the loss was never penalised for placing two similar identifiers together → the geometry has no direction that separates them → and a nearest-neighbour search, which can only consult that geometry, cannot recover the distinction. The first link is a decision somebody made about what to scrape. Every link after it is forced.

The NLI row carries the caveat that matters most for the chapters ahead. Some sentence encoders were trained with a triplet objective on SNLI and MNLI, collections of sentence pairs each hand-labelled as entailment, contradiction or neither, where the contradiction sentence of a premise is used as an explicit negative, so it is inaccurate to say that no training pair anywhere has ever tried to separate a statement from something incompatible with it. What is accurate is that this signal is a small minority of the data in current retrieval models, that a contradiction in NLI is usually a different scenario rather than a syntactic negation of the same sentence, and that published evaluations on negation-focused benchmarks find these models perform close to chance on telling a sentence from its own negation. The gradient spent nearly all of its budget on topic, so topic is what the geometry encodes at high resolution.

The negatives are drawn at random, which makes the task easy#

The other half of the objective deserves as much attention as the pairs, because in-batch negatives are whatever else happened to be sampled into the batch. Telling a help-centre article about refunds apart from 8,191 arbitrary web paragraphs is not a demanding discrimination, and a model can drive that loss most of the way down while remaining unable to tell the refund article from the returns article next to it. The loss cannot notice the difference, because the returns article was not in the batch, and if it had been it would have been one row of eight thousand rather than the row the answer depended on.

Training pipelines address this with mined hard negatives: run a first-pass retriever over the corpus, take documents that rank highly for a query but are labelled irrelevant, and add them to the batch as explicit negatives. Reported gains are large — the original dense-retrieval work on Natural Questions saw top-20 accuracy move by several points from hard negatives alone, and every strong model since includes some form of mining. The mining is done by relevance judgements, though, so it surfaces the confusions that a labelled relevance dataset happens to contain. A pair of sentences that differ by one function word, share every content word, and would both be judged relevant to the same query is not the kind of negative this process produces, because no annotator ever marked one of them irrelevant to the other.

The consequence is a space where the resolution is wildly uneven. Distinguishing topics is something the model was drilled on across hundreds of millions of examples with escalating difficulty; distinguishing two sentences within a topic that differ in their claim is something it was drilled on approximately never. Both distinctions look equally reasonable to ask of a similarity score, and only one of them is supported.

What this replaced, and what it did not#

Before contrastive sentence encoders, dense representations came from counting. The distributional hypothesis — Firth's line that you shall know a word by the company it keeps — was implemented in word2vec and GloVe as a shallow objective over co-occurrence within a window, producing one static vector per word type. That gave the field the arithmetic the vector topic opened with, and it gave it a hard ceiling: bank had one vector covering the river and the institution, and a sentence vector was whatever you got by averaging its words, which discards order entirely.

Contextual encoders removed the first limitation, and initially made the second one worse. Taking BERT's raw pooled output as a sentence vector produces a space where nearly everything is close to everything else, because the masked-language-model objective it was pretrained with, predicting words blanked out of a sentence, never required two sentences to be far apart; measured cosine between random sentence pairs from a general corpus sits around 0.6 to 0.7 rather than near zero. The reason is the same reason as everything else in this chapter: to predict a blanked-out word the model needs a good representation of the sentence it sits in, and nothing in that task compares two sentences at all, so no pressure ever existed to spread them out. Sentence-BERT in 2019 showed that a comparatively small contrastive fine-tune on top of a pretrained encoder fixed this, and that recipe — pretrain on language, fine-tune on pairs — is what every model in the table above is doing.

The thing it replaced least is lexical retrieval. BM25 scores a query against a document by term overlap weighted by rarity, it has no training data and no notion of paraphrase, and on corpora full of identifiers, product codes, error strings and names it still beats general-purpose dense retrieval regularly. The pattern that most production systems have converged on is running both and merging the ranked lists, which is an admission that the two methods fail on different inputs rather than one being a strictly better version of the other.

Predicting what the space will and will not separate#

The useful consequence of knowing the objective is a question you can ask before running any experiment: was there a training pair whose two sides differed only in the property I am relying on? If the answer is no, the space almost certainly does not encode that property at a scale that survives a nearest-neighbour search.

Run that test over a few properties and the pattern is consistent. Paraphrase survives, because paraphrase pairs are the bulk of the data. Cross-lingual matching survives in multilingual models, because translation pairs were included deliberately for exactly that reason. Query-to-document asymmetry survives in models trained with instruction prefixes such as E5's query: and passage:, where the same text embedded under the two prefixes lands in measurably different places. Numeric magnitude does not survive, because no pair source systematically contrasts "retry after 30 seconds" with "retry after 300 seconds". Temporal order does not survive. Argument roles — who indemnifies whom — do not survive, which the vector arithmetic topic already demonstrated with a contract clause and its reversal.

The test has a limit worth stating, because it is a screen rather than a proof. A property with no dedicated pair source can still be partly encoded as a side effect of one that has: dates are numerals, and numerals differ as tokens, so a document mentioning one year does sit slightly away from the same document mentioning another. What the test predicts is not that the signal is absent but that it is small relative to topic, and "small relative to topic" is exactly the condition under which a nearest-neighbour search over a real corpus loses it, because every candidate in the top of that ranking already shares the topic. The screen fails in the other direction too, for the one class of model where the caller supplies the discrimination at query time rather than the training set supplying it in advance.

This is a claim about the embedding model, not about what a language model can do. The same underlying transformer, prompted generatively, will tell you correctly that one sentence contradicts another; the embedding is a single fixed point selected by a loss that never had a channel for that relation, and one vector cannot carry a relation that only exists between two texts. That last sentence is the hard floor under everything in this topic, and it is structural rather than a matter of training budget: a relation takes two arguments, and a stored vector has been computed from one text with no access to the other. Instruction-following embedding models are the live attempt to loosen this, letting the caller state the task so the space rearranges per query, and there is active work on using decoder LLMs as embedding models through in-context examples rather than fine-tuning, surveyed in LLMs are Also Effective Embedding Models. Those shift what a single vector can be asked to represent; none of them make one vector into a comparison operator.

So the geometry is a record of a training procedure, and the procedure is public. The loss decides that magnitude is invisible and that separation is rewarded in proportion to how many random texts share a batch; the pair sources decide which properties get a direction at all, and their silence on relations between texts is not an accident of any one dataset but a consequence of what can be harvested at that scale. A model card and a dataset listing therefore answer questions that would otherwise cost a week of experiments, including which corpus a model will fail on and why a larger version of it will fail the same way.

Chapter 1 measured a paraphrase at 0.887 without saying why, and the objective is the why. The next question is where in the vector that similarity actually lives, and the tempting answer — that particular coordinates hold particular features — is worth testing rather than assuming.