A fixed neighbourhood expands predictably; a section does not

Two retrieval designs match a small piece of text and return a bigger one. Sentence-window retrieval embeds every sentence on its own and, when a sentence is the best match for a query, returns that sentence together with a fixed number of sentences either side of it. Parent-document retrieval embeds small child chunks and, when a child is the best match, returns the structural unit that encloses it — the section under a heading, a page, a clause. Most descriptions of the pair stop there, at match small, return big, and at that level of detail they are the same technique with a knob set differently.

Reading them that way is a fair reading of the description and a wrong reading of the behaviour, and the cost of the mistake lands on whoever has to keep the system inside a token budget. What separates the two designs is where the size of the returned unit comes from. Sentence-window computes the returned span from the position of the match: two sentences before, two after, wherever that lands. Parent-document reads the returned span off the document's structure: whatever heading, section or page encloses the match. Those coincide only on documents whose structural units happen to be small and even, and a body of documents on which retrieval is hard is exactly one where they are not.

The returned unit comes from position or from structure#

Read the two branches as two different functions of the same hit. The window branch is a function of the hit's index in a sentence array and nothing else, so its output size is determined at configuration time: window radius times mean sentence length, with a variance that comes only from how long the author's sentences are. The parent branch is a function of how the document was written, and its output size is a property of a body of documents that you inherit rather than choose.

The two branches are usually argued about as though they answered different requirements, and they do not. The requirement is identical in both: a matched fragment is too small to answer from, so the generator needs more text around it. Everything after that sentence is a design decision. Choosing to derive the extra text from a position in an array is one decision, and the property it buys — a size the system sets rather than discovers — follows from that choice and not from the requirement. Choosing to derive it from the document's own structure is the other decision, and the property it buys is a unit whose boundaries somebody intended. Separating the two matters because a requirement is not negotiable and a design decision is, and a team that has fused them will defend an inherited size distribution as though the complaint demanded it.

A range query on a sorted array and a foreign-key lookup differ in exactly that respect. arr[i-2:i+3] returns five elements no matter what is in the array. SELECT * FROM sections WHERE id = ? returns whatever that row happens to hold, and if some rows hold 200 tokens and others hold 6,400, the query plan is fine and the prompt is not.

That pairing is an analogy rather than a claim about databases, and it breaks in two places, both of which turn up again as real effects later in this chapter. Array elements are uniform in size, so a five-element slice is a fixed number of bytes as well as a fixed count of elements; five sentences are a fixed count and a variable number of tokens, and the gap between those two kinds of fixedness is where a window's residual variance lives. The second break is at the ends of the array, where a slice quietly returns fewer elements than were asked for. That is not a flaw in the picture. It is what the window code below actually does.

Bounded by construction#

The implementation that goes with the window branch is small, because the expansion does not need a document store or a parent link. Store the neighbourhood on the chunk's metadata at index time, embed only the sentence, and swap the text back in after retrieval and reranking. LlamaIndex, one of the two frameworks most RAG pipelines are built on, ships this as a node parser plus a MetadataReplacementPostProcessor, and the two-component shape is worth copying whatever framework you are in: the replacement happens after scoring, so the reranker sees the same tight sentence the retriever matched.

import re

def build_sentence_windows(doc_text, doc_id, radius=2):
    sents = re.split(r"(?<=[.!?])\s+", doc_text.strip())
    nodes = []
    for i, s in enumerate(sents):
        lo, hi = max(0, i - radius), min(len(sents), i + radius + 1)
        nodes.append({
            "id": f"{doc_id}:{i}",
            "embed_text": s,                       # what gets vectorised
            "window_text": " ".join(sents[lo:hi]), # what the generator sees
        })
    return nodes

def replace_with_windows(hits):
    """Run after retrieval AND after reranking, never before."""
    for h in hits:
        h.text = h.metadata["window_text"]
    return hits

The score-then-expand ordering established in the previous chapter, where the expansion is applied after the reranker has scored the small unit the retriever matched, is easier to violate here, because most frameworks apply postprocessors as a single list and the replacement is just another entry in it. A cross-encoder scoring a five-sentence window is scoring four sentences the retriever never considered, and the effect is larger than with parents: the matched sentence is a fifth of the window, so a distinctive one-sentence match can be scored down to nothing by four neighbouring sentences of narrative.

The claim that a window's size is decided at configuration time is worth checking rather than accepting, and the check is a multiplication of two numbers. The first is how many sentences come back, which is twice the radius plus one — the matched sentence, plus radius sentences before it and radius after. At a radius of two that is five sentences for every hit anywhere in the collection, and that count has no variance at all. It is not an average across documents; it is the same integer every time, because the loop that produces it counts positions in an array and reads nothing about the text. The second number is how many tokens a sentence runs to, and that one is a property of whoever wrote the documents. Multiply the two — sentences per window, times tokens per sentence — and the product is the expected size of a returned unit.

Reading the radius sweep as a product#

The radius is the only real tunable and it sweeps quickly, since re-running it requires no re-embedding — the vectors are of the sentences, and only the stored window text changes. On a corpus of 12,000 customer call transcripts, with k=8 and the LLM judge from chapter 1, a second model shown the question, the produced answer and a hand-written reference and asked whether the two agree:

The second column is that product written out, and it can be checked row by row. At radius zero the unit is a single sentence and the table gives 22 tokens, so 22 is what a sentence costs on this material. Radius one returns three sentences, which should cost three times 22, and the table gives 66. Radius four returns nine sentences, which should cost nine times 22, and the table gives 198. Three of the five rows land on the product exactly.

The other two miss it by a few tokens, in opposite directions, and both misses have a cause worth naming rather than rounding away. Radius two returns five sentences, so the product predicts a little over 108 and the table gives 108 — slightly under. The reason is in the code above: max(0, i - radius) and min(len(sents), i + radius + 1) clamp the window at the ends of a document, so a sentence near the start or the finish gets a shorter window than the radius asked for, and those short windows pull the median down. That is the second place the array analogy broke, arriving as a two-token effect. Radius eight returns seventeen sentences and the table gives 380, slightly over seventeen times 22. The cause there is the opposite one. Twenty-two tokens is the median sentence, and sentence lengths are right-skewed: most are short, a few are very long. The median of a sum of many draws from a skewed distribution drifts towards that distribution's mean, which sits above its median, so summing one sentence keeps you at the median and summing seventeen does not.

Neither residual is large, and their smallness is the substantive result. A window's size is predictable to within a couple of tokens from two numbers, one of which is set in a configuration file and the other of which can be measured once on the material and reused. That predictability shows up again in the third column, and again in the table further down that measures the size of what actually reaches the prompt.

The third column is the same product taken one step further, multiplied by the number of passages sent. Every row closes exactly: 22 × 8 is 176, 66 × 8 is 528, 108 × 8 is 864, 198 × 8 is 1,584 and 380 × 8 is 3,040, and those five products are the five entries in the prompt-tokens column with no residual anywhere. A ratio that holds on one row is a coincidence. A ratio that holds on five independent rows, across units ranging from one sentence to seventeen, is what licenses writing prompt size as a formula — per-unit tokens times k — rather than as a curve somebody fitted to measurements. It is the difference between a number you can put in a capacity plan and a number you have to re-measure every time a parameter moves.

The curve in the last column has the same shape as the k curve and for the same reason, which is worth noticing because it means radius and k are partially interchangeable: both buy context and both eventually buy dilution, and a pipeline at radius 4 with k=8 is sending about as much text as radius 2 with k=15. That equivalence is the formula being used in both directions — 1,584 tokens against 108 times 15, which come out within a few dozen tokens of each other. They are not equivalent in what that text contains — one spends the budget on more of the same neighbourhood, the other on more places in the collection — and which trade is better depends on whether your failures are missing-context or missing-document.

Asymmetric radii are underused and cheap to try. In transcribed dialogue the sentence that gives an answer its meaning is usually the question before it, so a radius of three before and one after outperformed a symmetric radius of two on the same set by 1.4 points. That comparison is cleaner than it looks, and the formula is why. Three before, one after and the matched sentence itself is five sentences; two before, two after and the match is also five sentences. The two configurations return the same count of sentences and therefore the same token cost, so the 1.4 points cannot be a reward for sending more text. It is entirely a statement about which sentences were sent. In procedural documentation the dependency runs the other way, since a step's consequences follow it. The parameter costs nothing to change and nobody changes it.

Adjacent hits create the one bookkeeping problem the technique has. When sentences 47 and 49 both match, their radius-2 windows overlap in four sentences, and a naive pipeline sends that overlap to the generator twice. Duplicated text is not merely wasted budget — repeated evidence reads to the model as corroboration, which is exactly the false signal the later chapter on diversity in a result set, where several retrieved passages give one account of the same cause, is about. Merging overlapping spans into a single unit before prompting is a few lines over sorted sentence ranges, and on the transcript corpus it cut prompt tokens by 18% at k=8 with no accuracy change, because adjacent matches are common in dialogue where a topic runs for a minute. Note that the merge also breaks the exact closure above: once overlapping windows are collapsed, the prompt is no longer k times a unit, and the 18% is the size of that break.

What a parent inherits instead#

Nothing in the previous section survives the move to parents, and it is worth being precise about which step fails first. The count fails. A window returns twice-the-radius-plus-one sentences because a loop was told to; a parent returns however many tokens that parent happens to hold, and there is no factor in the expression that a configuration file controls. Both numbers in the product are inherited, so there is no product to write down — there is a distribution, and the only way to learn its shape is to measure it.

Measuring it is what the following table does. It reports what actually reaches the prompt on the same corpus of 12,000 customer call transcripts, at k=8:

The window column repeats the closure exactly — 108 in the first row, 864 in the fourth, the same multiplication by eight that held across the radius sweep. The parent column does not close, and the failure is informative. Multiplying the median parent by eight overshoots the tabulated median prompt by more than a whole parent's worth. Two causes are available and both are real: several of the eight hits can be children of the same parent, and a parent is sent once, so the prompt often holds fewer than eight distinct units; and the median of a sum of skewed draws is not the number of draws times the median of one draw, which is the same effect that lifted the radius-eight row. Nothing in the table says which cause dominates. What the table does say is that the arithmetic which made the window column predictable is simply unavailable on the parent column, so any figure about parent prompt sizes has to be measured on the sum rather than derived from the unit.

The zero in the bottom-left cell deserves a second look, because it is a stronger statement than an observation of zero. The largest returned unit a radius-2 window produced anywhere in this corpus is 390 tokens, and eight of those is a little over three thousand — comfortably inside a 16k budget, with room to spare. No query in that pipeline can exceed the budget, whatever the material does, as long as the maximum unit size holds. The cell is a consequence of the row two lines above it rather than an independent count, and it would still read zero on a corpus nobody has measured.

The parent column reaches the opposite conclusion from figures that look, at first glance, as though they should reach the same one. The largest single parent is 11,200 tokens, which is also inside a 16k budget, and yet 4.2% of queries exceed that budget. There is no contradiction: the budget bounds a prompt, and a prompt is a sum of up to eight units. Two large parents landing in one result set clear 16k between them without either one being unusual. A per-unit cap does not bound a prompt, and a team that checks the largest document against the context window has checked the wrong quantity.

Both columns can be read with a single rule, which is the useful thing to take away from the table. Subtract the p99 prompt row from the budget. Where that difference is positive, the last row is at or near zero; where it is negative, the last row is not. The window column leaves a wide positive margin and reports zero. The parent column's p99 prompt is 24,600, half as much again as the budget, and it reports 4.2%. The rule holds on two columns that share no machinery — different index, different expansion step, different size distribution — which is what makes it a rule about prompts rather than an observation about one pipeline.

The p99 row is therefore the operational content of this chapter. A window pipeline has a prompt size you can compute in advance and put in a capacity plan. A parent pipeline has a long right tail that fires on whichever queries happen to hit the largest sections, and those are frequently the most important documents, because the longest page is usually the one everybody edits. Truncation logic then decides what to drop under budget pressure, and truncation logic that fires on 4% of queries is a source of failures that never appears in an offline evaluation run at median size.

That last sentence names the stumble this chapter exists to prevent, and it is worth stating in its general form because it recurs everywhere a tail matters. The reassuring measurement — a median prompt of 4,100 tokens against a 16k budget — is identical in the world where the tail fires on one query in twenty-four and the world where it never fires at all. A median cannot tell those two apart, because a median is by construction insensitive to what the largest few per cent of its inputs are doing. Believing the median has cleared the budget is believing that a defence works when the quantity it was supposed to discriminate is one the defence never reads. The same shape appears whenever a team reports that the system ran for a quarter without falling over: absence of a report is what both worlds produce, and only an instrument that reads the tail can separate them.

Which corpus each one wins on#

Call transcripts are the clearest case for windows. A sales call has no headings, no sections and no natural units below the call itself; a 45-minute transcript is 6,000 tokens of flowing dialogue. Parent-document retrieval on that corpus has exactly two granularities available — the utterance and the whole call — so expansion means sending 6,000 tokens for a hit on one line. A window of two sentences either side returns the question that prompted the answer and the follow-up that qualified it, which is precisely the missing context, at 110 tokens. Depositions, meeting notes, news articles, novels and any transcribed speech behave the same way, since what surrounds a sentence in a linear narrative is what makes it interpretable.

That description also settles what the parent column of the table above was measuring, and it is worth being explicit because the column header could mislead. If the corpus has no sections, then the "section parents" a hierarchical splitter produced on it are not units any author wrote — they are blocks the splitter invented at whatever size the text allowed. Their median of 610 tokens and their maximum of 11,200 are a joint property of the splitter's settings and the material, and their internal coherence is not guaranteed by anything at all. On a corpus that does have real structure, the same column would report a different spread and a different kind of unit, so the specific figures transfer no further than the pipeline that produced them. The shape — a median far below a maximum, with no configured factor anywhere — is what transfers.

Structured reference material goes the other way, and the reason is that its units of meaning are frequently not sentences at all. On the API reference from the previous chapter, a hit on the sentence "Returns 409 if the idempotency key has already been used" needs the parameter table above it and the list of other error codes below, and neither is a sentence. A radius-2 window cuts a Markdown table in the middle of row three and hands the generator four columns without a header, which is worse than the fragment it replaced. The structural unit exists, it is small and coherent, and taking it whole is what the corpus wants.

The wiki row is the honest one. When the structural units are bad, a window is the lower-risk default, and it is not a good answer — it is a bounded one. A window on a sprawling page returns 110 tokens of the correct area and no guarantee that the relevant definition is inside the window, so the failure mode becomes missing context rather than diluted context. Which of those two failures you prefer depends on whether your generator is more likely to hallucinate a plausible answer from partial evidence — to state something fluent that the passages it was given do not support — or to say it does not know, and that is worth measuring on your own model rather than assuming.

Where a bounded span is not enough#

The bound that makes windows safe is the same bound that makes them insufficient when evidence is genuinely distributed. A question whose answer requires a definition on line 6 and a condition on line 71 of the same document is unanswerable from a five-sentence span, and no radius that fixes it is small enough to keep the property this chapter is about — a radius wide enough to bridge 65 sentences is not a window, it is the document. Parent-document retrieval handles that case when the two lines are in the same section, which is a real advantage and the reason the technique exists.

Windows also assume sentence segmentation is meaningful, which fails on code blocks, log lines, tabular data and languages where the regex-and-punctuation approach above breaks down. A window over a stack trace returns five lines of a stack trace, and the five preceding lines of a JSON payload are not context for the sixth.

That failure is worth following one step further, because it does not merely degrade the technique — it removes the guarantee the technique was chosen for. The bound on prompt size came from multiplying a fixed sentence count by a typical sentence length, and the second factor is only meaningful if sentences are a real unit in the material. Where the splitter finds no punctuation it returns enormous pseudo-sentences, and the product that was supposed to be bounded inherits their size instead. So the first thing to check before adopting windows is the distribution of sentence lengths in the corpus: if the p95 sentence is 400 tokens because half the corpus is unpunctuated logs, the bounded expansion is not bounded. The check is cheap and it is the same measurement that gave the 22 in the radius sweep, taken at a high percentile rather than at the median.

The size guarantee is conditional in one further way that is easy to state and easy to forget. It holds for the retrieved units and says nothing about anything else in the prompt. A system template, a conversation history and a long question all consume the same budget, and the ceiling derived above bounds only the passages. A pipeline that sizes its context window against that ceiling alone has computed a true number about the wrong quantity.

Both techniques share one property that the next chapter takes apart. They expand every hit, unconditionally, whether or not the evidence for that query was actually fragmented. A query whose answer sits entirely within one 200-token child still gets the full 610-token parent, and a query whose answer is one sentence still gets five. That is a lot of tokens spent on queries that did not need the help, and it invites a rule that expands only when the retrieval result itself suggests the evidence was split.

The reason that rule needs a chapter of its own is that it makes the expansion conditional, and a conditional strategy breaks the arithmetic this chapter has been relying on. Everything above worked because the same transformation applied to every query, so a per-unit figure multiplied cleanly into a per-prompt figure and an evaluation compared two systems that differed everywhere. A rule that fires on some queries and not others produces two systems that are identical on most of their inputs, and the standard way of measuring the difference between them stops being able to see it.