Late chunking changes the vectors and nothing else

Late chunking is a change to when the embedding model runs, and not to where a document gets cut. Every strategy in this topic so far decides boundaries first and hands each resulting chunk to the encoder on its own. Late chunking keeps whichever boundary rule was already chosen and moves the encoder in front of it: the model reads the whole document, and the cutting happens to the model's output rather than to the text. That single reordering is the entire idea, and everything the technique buys and everything it fails to buy follows from it.

What the reordering answers is anaphora — the habit a document has of naming something once and then referring back to it as "it", "the unit" or "the above". A person reading the document straight through resolves those references without noticing they are doing it, because the name is a few pages back and still in mind. A chunk cut out of the middle of that document has no few-pages-back. The name is not in it, and a vector computed from the chunk alone therefore has no way to encode which product, policy or component the chunk is talking about. The clearest way to see the cost is one page of one datasheet, sitting in an index of 84,000 chunks built from a corpus of vendor documentation.

A chunk that cannot name its own subject#

The datasheet names the product once. The cover section says "Digidim 458/DALI dimming driver", and for the next nine pages the document says "the driver", "the unit", or "it". Query that index for "Digidim 458 dimming curve" and the results come back like this:

The chunk that answers the question is at rank 912 out of 84,000, and the chunk that ranks is a parts list. Nothing has malfunctioned. The query supplies four pieces of vocabulary — the brand name, the model number, and the words for the behaviour being asked about — and the two chunks divide them cleanly between them. Chunk 1 holds the brand name and the model number and says nothing about dimming behaviour. Chunk 12 holds the dimming behaviour and, of the query's vocabulary, contains only "dimming curve". Half of a corpus of dimming drivers is documents about dimming curves, so that shared vocabulary separates chunk 12 from almost nothing. The two discriminating tokens are the ones it lacks.

Read the table as the distance between its two rows rather than as two scores. The cosine gap is a little under three tenths; the rank gap is just over nine hundred places. Those two numbers describe the same pair of chunks, and their sizes are wildly different, because 84,000 chunks are packed into a narrow band of similarity to any one query. A tenth of cosine is not a tenth of the index. That asymmetry is worth carrying forward: in a dense index, a small movement in score is a large movement in rank, and reading a cosine delta as though it were a retrieval delta will mislead in both directions.

No boundary placement fixes this, and the reason is a limit on what a splitter is able to do at all. A splitter's only power is to decide which of the document's existing words end up in which chunk. It cannot introduce a word the author did not write. The product name is written once, on the cover, nine pages earlier. Any chunk that contains chunk 12's sentence and also contains the cover would have to span nine pages, which is not a retrieval unit. So cut the document at headings, at similarity dips, or every 300 tokens: chunk 12 still says "the driver", because that is what the text says. The failure is upstream of every dial the previous six chapters offered.

Embedding first, cutting second#

Every strategy so far produces chunks and then embeds each one in isolation, which works in the obvious order. The encoder receives chunk 12's 190 tokens, computes a representation for each of them with reference to all the others, and then mean-pools them: it averages the 190 token vectors component by component to produce the single vector that gets stored. The step worth stating is the one that makes the failure inevitable. An encoder is a function of the tokens handed to it, and a function's output is determined by its input. Anything the document established on page 1 was never in that input. It therefore cannot be anywhere in the output, whatever the model knows about dimming drivers in general.

Late chunking inverts the order. Tokenize the whole document, run one forward pass over all of it with a model whose context window is long enough to hold it, and keep the per-token output representations rather than pooling immediately. The property that makes this worth doing is self-attention, which computes each token's representation with reference to every other token in the input. Because that reference is to every other token and not to a neighbourhood, the representation of the word "driver" at position 4,100 has attended to position 40, where the model saw "Digidim 458/DALI". Only then do you apply the chunk boundaries — to the token sequence, not to the text — and mean-pool each span separately.

Two spans are enough to see the whole difference. Take a document, cut it in half, and compute both ways. Independent embedding runs two forward passes, each over its own half, and averages each pass's own token vectors. Late chunking runs one forward pass over the whole sequence and takes two averages over disjoint halves of that pass's output. The averaging arithmetic is identical in the two cases — the same number of vectors, added and divided the same way. What differs is what each token vector had already looked at by the time it was averaged.

The implementation is short, because it is ordinary pooling applied to spans instead of to the whole sequence.

import torch
from transformers import AutoModel, AutoTokenizer

name = "jinaai/jina-embeddings-v2-base-en"           # 8,192-token context
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModel.from_pretrained(name, trust_remote_code=True).eval()

def late_chunk(text: str, spans: list[tuple[int, int]]) -> torch.Tensor:
    enc = tok(text, return_tensors="pt", truncation=True, max_length=8192)
    with torch.no_grad():
        hidden = model(**enc).last_hidden_state[0]     # (n_tokens, 768)
    vecs = [hidden[a:b].mean(dim=0) for a, b in spans]
    return torch.nn.functional.normalize(torch.stack(vecs), dim=-1)

Four things in that function carry the argument. hidden is the model's output before any pooling, one row per token and 768 numbers wide, 768 being this model's representation width. spans is a list of (a, b) token offsets, one pair per chunk, marking where each chunk starts and stops inside that same token sequence. hidden[a:b].mean(dim=0) takes the rows between those two offsets and averages them down to a single row, which is the mean-pooling step applied to a span rather than to everything. And torch.nn.functional.normalize rescales each resulting vector to unit length, so that a dot product against a query reads as a cosine.

spans is the part worth being precise about, because it is an input. Those offsets are produced by whatever splitter was already chosen. Late chunking is not a splitting strategy and does not compete with the ones in chapters 1 through 6; you still decide where the boundaries go, by heading, by similarity, or by counting to 480. That is the first thing about this technique that is easy to get wrong, and the name invites the error: something called late chunking sounds like a rival to heading-aware chunking, and it consumes heading-aware chunking's output instead.

Follow the consequences of spans being an input and the chapter's title falls out with nothing left over. The technique takes a document and a set of boundaries and returns a set of vectors. The boundaries are what it was given, so they do not move. The text of each chunk is the text those boundaries delimit, so it is byte-identical to what the same boundaries produce under ordinary embedding. Every payload field computed from that text — token counts, breadcrumbs, section ids — is computed from the same bytes, so it is unchanged too. What the generator reads when a chunk is retrieved is unchanged, and the citation is unchanged. The chunk records differ in the float arrays and in nothing else.

That accounting is worth having because of what it licenses. When exactly one thing changes, an A/B comparison between the two indexes needs no new relevance labels, no re-check of citation rendering, and no re-parse: the served text is the same text, so any difference in the evaluation is a difference in ranking. It also bounds the upside. A failure that is not a ranking failure cannot be repaired by this, and the last section of this chapter is a list of the ones that are not.

On the datasheet, chunk 12's cosine against "Digidim 458 dimming curve" moves from 0.508 to 0.686 and its rank from 912 to 4. The cosine moved by a little under two tenths and the rank by just over nine hundred places — the same lopsided relationship as before, now running the other way. The text of chunk 12 still does not contain the product name. Its vector now does, in the only sense a vector contains anything.

There is an analogy for this, and it is worth stating as an analogy rather than letting it pass as a description. A proofreader who has read the whole manual knows which unit "the unit" is on page nine; a proofreader handed page nine alone does not. Late chunking is the first proofreader. Where the picture breaks is at the output: the proofreader who knows could write the product name into the sentence, and late chunking cannot and does not. The words on page nine are exactly as they were. Only the internal representation of them is better informed, which is why this technique moves ranks and never moves text.

The idea was published in 2024 as Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, and it is worth reading for the ablations rather than the headline. An ablation is a run with one component removed or changed, which is what tells you where a reported gain actually came from: the gains concentrate on datasets with long documents and referential text, and several benchmark subsets move by less than a point. That distribution of the gain matches the two conditions below, which is a small piece of external agreement with the argument this chapter is making.

Two mechanical details cause most of the trouble when this gets built, and both are silent. Boundaries have to be expressed as token offsets into the same tokenization the model saw. A splitter that returns character offsets needs return_offsets_mapping=True and a translation step, and an off-by-one there shifts every chunk's pooling window without producing an error, because hidden[a:b] is a perfectly legal slice at any offsets that are in range.

The second detail is a place where a defence looks like it is holding and is not. The pooling has to match what the model was trained with. The models this technique is usually applied to are mean-pooled, so mean-pooling spans is consistent with training. A model trained with CLS pooling — where one designated leading token is trained to carry the representation of the whole input, and the other positions are never asked to stand for anything — has no per-token representation that means what you want. The tempting part is that the hidden states are right there, the slice runs, the shapes are correct, and vectors come out. What comes out is quietly off-distribution and scores worse than the naive baseline. Shape-checking is a defence against a missing tensor, and this is not a missing tensor.

Where this is heading is worth a sentence, because it changes the arithmetic rather than the idea. Embedding models with 32,000-token windows now exist at prices comparable to 512-token models of two years ago, and the overflow row in the table below shrinks every time one ships. The technique's dependence on a context window is a cost curve rather than a ceiling, which is a reason to expect it to become ordinary and not a reason to adopt it before it is cheap on your corpus.

Document length and genuine context-dependence#

Two conditions have to hold together, and a corpus can fail either one while looking like a good candidate.

The document has to fit the model's context window, because the whole mechanism is one forward pass over the whole document. On a 200,000-page vendor PDF corpus, the length distribution decides how much of the corpus actually gets the treatment.

Roughly 14 percent of documents overflow. That figure is a direct measurement, and the table brackets it without any measuring at all: the document at the 75th percentile is 7,400 tokens and fits, the document at the 90th is 14,200 and does not, so the length at which documents start overflowing falls somewhere between those two percentiles, and the share that overflows therefore lies between a tenth and a quarter. The measured 14 percent sits inside that band, which is the useful thing to know — two instruments that share no step agree, so the figure is a reading rather than a guess. The band is as far as the table can take you on its own. Filling it in by interpolating between two quantiles would be inventing the shape of a distribution nobody observed between them.

What happens to the overflowing documents is a decision you have to make rather than a detail the library handles. Truncating at 8,192 reproduces chapter 3's failure — where an encoder ceiling means only a chunk's opening is indexed — at document scale, with everything past the window unindexed. The usual alternative is to break the document into macro-windows of 8,192 tokens and late-chunk within each, which works, and reintroduces the original problem at a coarser granularity: a chunk in macro-window 2 has attended to macro-window 2 and knows nothing of the cover page in macro-window 1.

The window is the sharpest example in this topic of a defence that stays constant while the thing it has to handle varies. The window is 8,192 tokens for every document, whatever the document is. Document length on this corpus runs from 3,100 tokens at the median to 61,000 at the 99th percentile — a twentyfold spread. A fixed defence against a varying quantity fails on exactly the subset where the quantity is largest, and it is tempting to read that as acceptable, because the failing subset is a minority. On this corpus the long documents are also the ones with the most cross-references and the most anaphora. The fallback applies precisely where the technique was most needed, so the coverage number and the value number point in opposite directions and should never be averaged into one.

The second condition is that the chunks are genuinely context-dependent. Late chunking gives a chunk information from the rest of its document, and that is worth something only when the rest of the document contains something the chunk needs — an entity named once and then pronominalised, an implicit subject, a term whose meaning is scoped to a section, a table whose column headings are three chunks away. On a corpus of self-contained specification sections, where every section restates its subject in its first sentence because that is the house style, there is nothing for the attention to add. The vectors change slightly, the recall moves by fractions of a point, and the bill is a full re-embed.

That bill is larger than "the same tokens, once", which is the assumption that makes late chunking sound free. Attention is quadratic in sequence length: it compares every token with every other token, so the work in a pass over a sequence grows with the square of that sequence's length. Sixteen passes of 500 tokens each do sixteen times five-hundred-squared comparisons; one pass over the same 8,000 tokens does eight-thousand-squared, which is sixteen times more. The measured penalty is substantially smaller than that quadratic prediction, because attention is only part of a transformer's work and the rest of it grows in proportion to length rather than to length squared. Measured on one A10G, a mid-range 24 GB inference GPU, with the model above, a single 8,000-token forward pass costs about 4.4× the wall-clock of sixteen 500-token passes over the same text.

The four-hour embedding pass over the 90,000-document knowledge base becomes something closer to eleven hours. Eleven over four is a multiple under three, comfortably below 4.4, and it has to be: the 4.4 figure was measured at the full window length, and the median document on the corpus above is a little over a third of the window. Read 4.4× as the ceiling on the slowdown rather than as the average one. That is before counting the ingest rework: the pipeline has to hold whole documents in memory through embedding, track token offsets rather than character offsets through splitting, and handle the overflow policy. Three weeks of engineering for that is a fair estimate rather than a padded one.

Cross-references, bad parses, and what the recall number is made of#

Late chunking addresses one thing — a chunk's vector being ignorant of its document. Several nearby problems survive it intact, and being clear about which is which keeps it from being sold as a general upgrade.

It does not change what the generator reads. The cross-reference case from chapter 4 — where the chunk that ranks says the deadline is defined in section 4.2 and the chunk holding the figure sits well down the list — is the sharp illustration. The chunk holding "the window is 60 days" has now attended to "Claims must be filed within the window defined in section 4.2", so its vector picks up the filing vocabulary and its rank improves, which is real and useful. The chunk that comes back at rank 1 is still a sentence that points at section 4.2 without stating the number. If the 60-day chunk lands at rank 6 with k=8 the answer is now available in context; if it lands at rank 11 nothing has changed. Late chunking improves the odds on that failure. The ingest-time reference resolution from chapter 4, which rewrites the pointer to carry its target's text, removes it.

It does not fix a bad parse, a wrong boundary, or an oversized chunk. A chunk that straddles two procedures because a layout model invented a heading is still a chunk that straddles two procedures, now with a document-aware vector describing the straddle. The vector has become an accurate description of a chunk that should not exist.

And it does not, on its own, tell you whether recall of 0.61 on an index is a chunking problem. That question sits underneath every proposal in this topic and it has no chunking answer. Recall of 0.61 is equally consistent with boundaries in the wrong place, with an embedding model that does not understand the domain's vocabulary, with a query set full of questions the corpus does not answer, and with an evaluation label set that marks the wrong documents relevant. Those causes have nothing to do with each other, and no metric separates them. A metric counts what was missed, and these buckets are distinguished by why a thing was missed, which is a property no count carries. That is why the hand reading in the case above is not a shortcut taken for lack of tooling. It is the only instrument that reports the quantity the budget decision actually needs.

Late chunking is a change of one variable, and the whole of its behaviour follows from which variable that is. Boundaries in, vectors out: the served text, the payload fields and the citation are exactly what the same boundary rule produced before, and a chunk's vector is now pooled from tokens that had the rest of the document available while they were being computed. That buys one specific thing, which is a vector that knows what its chunk is about when the chunk's own words do not say. Where a corpus is full of documents that name a product once and then say "the unit", it is worth real points. Where sections restate their own subject, it moves recall by fractions and charges a full re-embed for the privilege.

The two conditions are checkable before anything is built. Measure the length distribution against the model's window to find out how much of the corpus is even eligible, and read twenty failures to find out how many of them are the kind of failure this repairs. If both come back favourable, the remaining question is only whether the same missing information can be supplied more cheaply — by prefixing each chunk with its document title and section heading before it is embedded, so the entity name sits inside the text the encoder reads. That costs no new model, no context-window arithmetic and no ingest rework, and the next chapter measures it against everything here.