Every chunking rule is a bet about where a fact ends

Chunking is the step that cuts a document into the smaller pieces that actually get indexed, and each piece — a chunk — is embedded into its own vector, stored as its own record, and returned on its own when a query matches it. What comes back from a search is therefore never the document; it is one of the fragments somebody decided to cut it into. In a working system you meet chunking in the ingest path as one function sitting between parsing and embedding, usually configured with two numbers, a size and an overlap, and those two numbers are the whole of what most teams ever decide about it.

It is worth being precise about why the step is compulsory, because the reason is usually given as one sentence about limits and it is really three separate pressures. An embedding model accepts a fixed amount of text and drops whatever runs past that without raising an error, which is the silent truncation of topic 8.7, so a unit longer than the limit is indexed by its opening alone while still being served in full. A generator has a context window, and a forty-page manual would fill it on its own, so even a system that indexed whole manuals could not hand one to the model. And a single vector averaged over a forty-page manual is too vague a description to match a question about one paragraph inside it. That third pressure is the one that survives everything: an encoder with no input limit at all would remove the truncation and leave the vagueness untouched, so even an unlimited encoder would not remove the need to cut.

A chunking rule decides where a retrievable unit stops, and every such rule is a bet about where facts end in your documents. Fixed windows bet that facts end every n tokens. Recursive splitting bets that they end at paragraph breaks, and failing that at sentence breaks. Heading-aware splitting bets that the author's section boundaries are the fact boundaries. Semantic chunking bets that a fact ends where the text stops being about the same thing, which it measures as a drop in similarity between neighbouring sentences. These are five answers to one question, and they differ because they disagree about what a fact is.

Where does a retrievable unit stop?#

One page of a company expense policy runs 1,180 tokens — a token being the unit an encoder counts in, roughly a short word or a piece of a long one, which is why token offsets and character offsets never line up. It opens with a short preamble about who the policy covers, moves into section 3 "Submitting a claim" with two subsections, presents a table of approval thresholds under section 4, and closes with a paragraph on disputed claims. Nothing about the page is unusual; it is the kind of document that makes up most of a company's internal corpus. Put it through five different splitters and you get five different sets of boundaries, and only two of them keep the sentence "Claims must be filed within 60 days of the transaction date" in the same chunk as the heading that says which claims it applies to.

Here is the page with token offsets, so the boundaries below have somewhere to land. The preamble runs to token 95. Section 3's introduction runs from 96 to 240. Subsection 3.1, "Deadlines", runs from 241 to 430, and the 60-day sentence sits inside it at tokens 298 to 331. Subsection 3.2, "Receipts", runs from 431 to 690. Section 4 with its threshold table occupies 691 to 1010, and the closing paragraph on disputed claims runs to 1180.

Two conventions in that table are worth settling before reading it, because they look like disagreements and are not. A boundary is sometimes written as the last token of the unit that ends there and sometimes as the first token of the unit that begins there, which is why the heading-aware row lists 430 and 690 while the semantic row lists 431 and 691. Those are the same two cuts named from opposite sides. And the strategy named "Semantic, 90th percentile" cuts where similarity is in the lowest decile, which is the same instruction stated the other way round: the cut lands where dissimilarity is in the top tenth, and dissimilarity is one minus similarity, so the ninetieth percentile of the one is the tenth percentile of the other.

The fixed-512 cut at token 512 falls in the middle of a sentence about receipt legibility, and the cut at 1024 falls inside the approval-threshold table, splitting the header row away from three of its five rows. Neither of those is a fact boundary in any sense an author would recognise; they are the consequence of a page that happens to be 1,180 tokens long. The recursive splitter never cuts mid-sentence, because it only falls back to sentence granularity when a single paragraph exceeds the ceiling, and no paragraph on this page does. Heading-aware produces the smallest chunks and the cleanest ones, at the cost of a chunk of 95 tokens for the preamble that will almost never be the right answer to anything.

The instructive way to read that table is down the chunk-count column, as gaps rather than as levels. Three, three, three, seventeen, six, five: the three strategies that arrive at the same count are the ones that ignore the document and divide its length by a size, and they agree with each other for a reason that has nothing to do with the page. The jump from six to seventeen is a change of granularity rather than a change of accuracy — the sentence-window splitter is not finding more boundaries in the document, it is declining to have boundaries at all and sliding a frame instead. The gap that carries real information is the small one, between the heading-aware six and the semantic five, because those two are the only strategies claiming to have located the author's own divisions, and one of them found one fewer.

That missing division is worth chasing, because it is where the two strategies can be checked against each other. The semantic splitter cuts after tokens 95, 430, 690 and 1010, writing the middle two from the far side as 431 and 691. The heading-aware splitter cuts after 95, 240, 430, 690 and 1010. Every one of the semantic cuts is also a heading cut — four of the five, matched exactly, with no semantic cut left over anywhere else on the page. That is a closing account rather than a rough agreement, and it licenses a claim the similarity numbers on their own could not support: a splitter that never looks at a heading recovers the author's section boundaries, because well-written prose changes subject where its author put a heading, and a similarity-based cutter finds the same place.

The residual is the fifth heading cut, at 240, and it is not noise. Token 240 is where section 3's introduction ends and subsection 3.1 "Deadlines" begins, and a section introduction is written to lead into its first subsection, so the two sides of that boundary are about the same subject in a way the two sides of the other four are not. The similarity drop there is real but too shallow to reach the lowest decile, so the cut is not made. Read the last column of the table and that single missed cut is exactly what puts the 60-day sentence into a semantic chunk carrying section 3's introduction with it. One unmatched boundary, one consequence, and they are the same fact seen twice.

The sentence-window splitter, which produces 17 chunks from one page, puts the 60-day sentence in a chunk that contains nothing else about deadlines, expenses or claims beyond the two sentences either side of it. That density buys precision and costs context, which is a trade with real consequences rather than a matter of taste. A chunk holding one sentence and its neighbours is a very sharp match for a query that quotes it and a very poor answer to a query that needs to know which claims the deadline applies to.

What overlap actually buys#

Overlap is usually described as "so facts do not get cut in half", which is close enough to be misleading. It does not prevent the cut. The cut happens exactly where it was going to happen; overlap adds a second chunk that starts before it, so the region around the boundary appears twice in the index.

Take a fixed 300-token window over the same page, which puts a boundary at token 300, in the middle of the 60-day sentence. Chunk 1 ends with "Claims must be filed within 60" and chunk 2 begins with "days of the transaction date." Neither chunk can answer the question, and the first one is worse than useless because it looks relevant and terminates before the number.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("BAAI/bge-base-en-v1.5")
ids = tok(page, add_special_tokens=False)["input_ids"]

def fixed(ids, size, overlap):
    step = size - overlap
    return [ids[i:i + size] for i in range(0, len(ids), step)]

for overlap in (0, 60):
    chunks = fixed(ids, 300, overlap)
    hit = [n for n, c in enumerate(chunks)
           if "60 days of the transaction date" in tok.decode(c)]
    print(f"overlap={overlap:3d}  chunks={len(chunks)}  sentence intact in {hit}")

overlap=  0  chunks=4  sentence intact in []
overlap= 60  chunks=5  sentence intact in [1]

Both printed lines can be recovered from the offsets above without running anything, and doing that once is what makes the mechanism stick. The step variable in the function is the distance between the starts of consecutive chunks, and it is the window size minus the overlap. With no overlap the step is the full 300, so the chunks tile the page end to end, and a 1,180-token page takes four of them — three full windows and a short remainder — which is the first printed count. The 60-day sentence occupies 298 to 331, and the boundary at 300 falls inside that span, so no single chunk holds it end to end and the list of hits is empty. Subtracting an overlap of 60 from a window of 300 leaves a step of 240, which is smaller, so covering the same page takes more chunks: five rather than four, the second printed count. The hit is reported as 1 rather than 2 because a Python list is numbered from zero, and the chunk that gains the sentence is the one immediately after the first.

With 60 tokens of overlap, chunk 1 still ends mid-sentence, and chunk 2 now starts at token 240 rather than 300, so it contains the sentence whole. The fact was not saved from being cut; a neighbour was created that happens to contain it. That is the actual mechanism, and stating it that way makes the sizing arithmetic obvious. Overlap has to be larger than the fact you are trying to preserve, or it does nothing for that fact. Sixty tokens of overlap protects a sentence; it does not protect a two-sentence definition, and it certainly does not protect a five-row table that was cut in the middle.

The condition can be made exact, and it is worth doing because it turns a rule of thumb into something checkable. A fact begins somewhere before a boundary and ends somewhere after it. The chunk that follows the boundary starts one overlap back from it, so that chunk contains the fact whole exactly when the overlap reaches back past the fact's first token. The distance from the boundary back to the fact's first token is at most the length of the fact, and it is exactly the length of the fact in the worst case, where the boundary lands on the fact's final token. An overlap at least as long as the longest fact you care about therefore protects every one of them, and an overlap shorter than a given fact protects that fact only when the cut happens to fall near its end.

The wrong belief that forms here is easy to hold, and it is tempting for a good reason. Overlap is configured once, as a single number, and it does visibly rescue the sentence it was tested on, so it reads as a defence that has been installed. What makes it fail is that the number stays fixed while the thing it has to protect does not: the facts in a corpus range from a clause to a five-row table, and a defence held constant across that range stops discriminating somewhere in the middle. The sentence that gets rescued in a demo and the table that does not are protected by the same setting, and nothing in the output distinguishes them.

The cost is straightforward. Overlap of v on windows of size s multiplies your chunk count by roughly s / (s - v). At 512 with 64 overlap that is 1.14×, which nobody notices. At 300 with 100 overlap it is 1.5×, which is 50 percent more vectors, 50 percent more embedding spend at ingest, and 50 percent more index. It also means the same sentence is now in two records, so a query matching it can return both, and a top-k of 5 — k being the number of chunks the retriever hands on for each query — can spend two slots on near-duplicate text. Deduplicating retrieved chunks by source-document offset before they reach the context window is a small piece of plumbing that most pipelines skip and then wonder where the redundancy came from.

That multiplier is not a rule of thumb, and deriving it takes one line. Without overlap, covering a document of a given length takes that length divided by s chunks, because each chunk advances the full window. With overlap each chunk advances only s - v, so covering the same length takes that length divided by s - v. Dividing the second count by the first cancels the document length entirely, which is why the multiplier does not depend on how long your documents are, and what survives the cancellation is s / (s - v). Putting the two configurations through it recovers both printed figures: a step of 448 under a window of 512 gives the first, and a step of 200 under a window of 300 gives the second. The second is larger for a reason worth naming — the overlap is a bigger fraction of the smaller window — so the tax is set by the ratio of overlap to size rather than by the overlap on its own.

Two decisions hide inside that arithmetic and are worth separating, because only one of them is yours. The encoder's input limit is a requirement: it is a property of the model you chose and no configuration changes it. The window size and the overlap are design decisions taken underneath that requirement, and they trade index size against the length of fact you are willing to lose. A team that writes down "512" without recording which of the two it is has lost the ability to revisit it, because the number now looks like a constraint.

Semantic chunking moves the decision to the model#

The four structural strategies all read the document's surface: counting tokens, finding \n\n, matching a heading pattern. Semantic chunking reads its meaning instead, by embedding each sentence or each small sliding window of sentences, taking the cosine similarity between consecutive embeddings, and cutting wherever that similarity falls below a threshold.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
sents = split_sentences(page)                       # 41 sentences on this page
vecs = model.encode(sents, normalize_embeddings=True)

sims = (vecs[:-1] * vecs[1:]).sum(axis=1)           # 40 consecutive similarities
cut_at = np.percentile(sims, 10)                    # 0.612 on this page
breaks = [i + 1 for i, s in enumerate(sims) if s < cut_at]
print(breaks)                                       # [3, 13, 21, 33]

Every count in that block follows from the one before it, and the chain is short enough to check by eye. There are 41 sentences, and a similarity is taken between each sentence and the next, so there is one fewer similarity than there are sentences: 40 of them, which is what the comment on that line says. Slicing the vectors two ways is how that pairing is expressed — vecs[:-1] is every sentence except the last and vecs[1:] is every sentence except the first, so multiplying them position by position lines each sentence up against its successor. Because the vectors were normalised at encoding time, each of those element-wise products summed along a row is already a cosine similarity, with no division by lengths left to do. The tenth percentile of 40 numbers has four numbers below it, so four cuts are made, and four cuts on one page produce five chunks — which is the chunk count the table gives for this strategy. Nothing in that chain is a coincidence; the chunk count is the percentile choice arriving downstream.

The four breakpoints correspond to token offsets 95, 431, 691 and 1010. The similarity between the last preamble sentence and the first sentence of section 3 comes out at 0.58; the median similarity between consecutive sentences within a subsection is 0.79. That gap is the signal the strategy runs on, and it is real, but its size varies enormously by corpus. On this policy page the tenth percentile is 0.612. On a corpus of terse API reference lines, consecutive sentences share so much boilerplate that the tenth percentile sits near 0.88 and the same absolute threshold of 0.612 would produce one chunk per document.

Which is the first thing worth being blunt about: the threshold is the entire design of semantic chunking, and it is a property of your corpus rather than a default you can copy from a tutorial. Chapter 6 takes that apart along with what the approach costs at ingest, because the cost is where the argument about it usually ends up.

Parse quality bounds chunking quality#

Chunking operates on whatever the parsing step from 9.3 produced. If the PDF extractor dropped a table's cell boundaries and emitted eleven numbers in a row with no delimiters, no splitter recovers the table; the best splitter available will simply put a clean boundary around unusable text. If the extractor interleaved a two-column layout so that sentences alternate between two unrelated arguments, semantic chunking will cut at nearly every sentence, because consecutive sentences genuinely are dissimilar. Chunking quality is bounded above by parse quality, and a chunking experiment run on a badly parsed corpus measures the parser.

The two-column case is worth a second look, because it is the one place where a semantic splitter reports its own input problem and nobody reads the report. Cutting at nearly every sentence is exactly what the rule is supposed to do when consecutive sentences are about different things, and on interleaved text they genuinely are. The splitter is correct and the output is useless. A chunk-count that jumps by an order of magnitude on one source format is therefore a parser alarm rather than a chunking result, and it is available before any retrieval metric is computed.

Indexing units, context units and retrieval knobs#

It is also worth separating chunking from the two things it is most often confused with. Chunking decides what gets indexed as a unit; it does not decide what gets sent to the model. A pipeline can retrieve a 200-token chunk and then expand it to its full parent section before generation, in which case the chunk size governs matching and something else entirely governs context. And chunking is not retrieval tuning: k, the similarity metric, hybrid weighting and reranking — the pass where a second and slower model rescores the chunks the retriever returned and reorders them — all sit downstream and all move the same headline numbers, which is exactly why a chunking result that was measured while those moved says nothing.

Three ways, one harness#

The reason to fix this vocabulary early is that the only useful thing you can do with it is comparison. There is no published answer to "what is the right chunk size", because the question is malformed: the right size is a function of your documents' structure, your embedding model's input limit, the length of the questions people ask, and how much context the generator needs to answer without hedging. Every one of those varies by corpus, and none of them is knowable from a blog post.

What is knowable is which of three configurations does better on your corpus, and getting that requires chunking the same corpus three ways and measuring retrieval for each. The word doing the work is same. One corpus, parsed once, with the parse output cached so all three runs consume identical text. One query set, written before you see any results, with the answer spans marked so you can tell whether a retrieved chunk contains the answer or merely sits near it. One embedding model. One index with the same distance metric, the same HNSW build parameters — HNSW is the navigable neighbour graph most vector indexes are searched through, and its build settings decide which neighbours a search can reach at all — and the same k. One evaluator, run the same way on all three outputs.

Under those conditions the difference between run A and run B is attributable to where the boundaries fall, because nothing else moved. That sounds procedural to the point of being obvious, and it is the step teams skip most reliably, because a chunking experiment tends to arrive bundled with the other improvements someone wanted to make anyway — a newer embedding model that came out last month, a larger k that seemed worth trying while the index was being rebuilt anyway, a re-parse with the extractor upgrade.

Every one of the five strategies is a bet, and the bets are what the differences on that one page of expense policy are made of: that facts end every n tokens, or at paragraph breaks, or at the author's headings, or where the prose changes subject. A boundary in the wrong place does not corrupt the text; it removes a fact from the index while leaving the text in place, which is why the damage is invisible from anywhere except a retrieval measurement. Overlap softens that in one direction only, by adding a neighbour long enough to hold the fact whole, and the length of the fact decides whether it works.

The five strategies in the table above are the vocabulary. Every claim in the rest of this topic — that raising chunk size made recall worse, that a chunk at rank 14 is unreachable, that late chunking changes only the vectors — is a claim about one of them, measured against another. The immediate problem is that measuring one against another is easy to do badly, and a chunking comparison with three variables moved is the most common way a team convinces itself it has learned something about boundaries.