Rank 14 is not a near miss, and no chunk size reaches it

A cross-reference is a sentence that names where the rest of an answer lives instead of containing it: "within the window defined in section 4.2". A person reading a policy resolves one of these without noticing, by turning to the section named. A retrieval system does not turn the page. It embeds each chunk of text once, scores every chunk against the query, and hands the highest few to a language model, so a pointer and its target are two independent candidates competing on their own merits.

Three terms carry the argument. Top-k is how many of the highest-scoring chunks are put into the model's context window, and k is that count. Cosine is the similarity score the retriever sorts by, higher meaning more alike. Rank is a chunk's position in that sorted list, which is a consequence of the score rather than a quantity of its own — a distinction that turns out to decide whether this failure is fixable by turning a dial. What makes the failure worth a chapter is that every signal a reviewer checks comes back correct while the answer is wrong.

A trace where nothing looks broken#

A policy corpus, chunked at 120 tokens with no overlap, answers the query "how long do I have to file an expense claim?" like this:

Top-k is 8, so chunk 14 is not in the context window. The model receives eight chunks, one of which states with complete fluency that claims must be filed within a window, and none of which says how long the window is. It answers 30 because 30 days is what expense policies usually say.

The causal chain there has no weak link and is worth stepping through, because each step is the sort of thing that gets skipped as obvious. The retriever ranks by similarity between the query's vector and each chunk's vector. A vector summarises the words that went into the chunk, so a chunk written in the query's vocabulary scores well and a chunk written in some other vocabulary does not. The pointer chunk is written in the query's vocabulary — claims, filed — which is why it wins. The referent chunk is written in the vocabulary of the answer — window, transaction date — which the person asking never used. So the ordering the retriever produces is exactly the ordering it was asked to produce, and the chunk carrying the fact is missing from the context window as a correct consequence of a correctly working component.

Why 14 and not 9#

The instinct on seeing rank 14 with k=8 is that this was close, and that k=20 would have caught it. Look at the score column instead of the rank column.

Between rank 9 and rank 60 the score falls by 0.070, and between rank 12 and rank 20 it falls by 0.020. Those two subtractions are worth converting into a rate, because a rate is what makes the rank column readable. Fifty-one places separate rank 9 from rank 60, and seventy thousandths of cosine spread across fifty-one places is a little under one and a half thousandths a place. Compare that with the top of the list, where rank 1 sits at 0.812 and rank 2 at 0.784: one place, twenty-eight thousandths. A single place at the head of the list costs about twenty times what a single place costs in the region where the referent is sitting, and further out it flattens again — eighty places from rank 20 to rank 100 cost forty thousandths, around half a thousandth each.

The intuition to take from that is about what rank measures. Rank is a count of how many chunks happen to be above you, and near the top of the list very few chunks are packed into a given band of score, so each place is expensive. Down in the tail several hundred chunks are packed within a few hundredths of each other, so places are cheap and a chunk's exact ordinal position is decided by differences smaller than the noise in the query itself. The ordinal number 14 looks like it is on the same scale as the ordinal number 8. The scores say it is not.

Making that precise closes the argument about k. To reach the context window by scoring its way in, the chunk holding the number has to beat the score at rank 8, which is 0.703 against its own 0.664 — thirty-nine thousandths of cosine. The entire spread of this tail, from 0.698 at rank 9 down to 0.611 at rank 100, is eighty-seven thousandths. The lift required is therefore close to half the whole dynamic range of everything below the cut, which is not a nudge; it is asking the chunk to become a different kind of match.

Raising k does not supply any of that lift, and the reason is worth stating plainly rather than leaving implicit in the tables. k is where the list is cut. It is not an input to the scorer, so no value of k changes a single number in the score column. Raising it from 8 to 20 moves the cut line past a chunk that has stayed exactly where it was, which works only for as long as the chunk stays there.

The lottery a rephrasing wins#

Rephrase the question three ways and watch what happens to it:

The chunk moves between rank 9 and rank 140 across four phrasings of the same question, because the chunk has almost nothing in common with any of them. Its content words are window, 60 days, transaction date. It never says file, claim, expense, deadline or submit. Its score against these queries is essentially its background similarity to anything written in the register of a corporate policy, which is why its position is a lottery.

Reaching for a larger k against a lottery like that one is the strongest form this mistake takes, and it is worth naming as a shape rather than as an anecdote, because it recurs. A cut at k=20 is a constant. The quantity it has to discriminate — where this chunk lands — is not constant; it varies by more than a hundred places across four ways of asking one question. A defence that holds still while the thing it must separate moves around it does not fail loudly. It succeeds on whichever cases were in front of you when you chose the constant, which is why shipping k=20 as the fix means shipping a system that answers the deadline question correctly when it is asked one particular way.

There is a second reason k=20 is not the fix, which holds even for the phrasings where the chunk does make the cut. The context window would then contain a fluent sentence at rank 1 saying the deadline is defined in section 4.2, and somewhere around rank 14 a chunk beginning "4.2 The window is 60 days". Answering correctly requires the model to notice that the rank-1 sentence is a pointer, recognise "4.2" in a different chunk as its target, and prefer the second over the first. Models do this sometimes. They do it far less often when the pointer is early and fluent and the target is buried among nineteen other chunks, and no part of the pipeline has ever been measured on whether it does it. Betting the fix on unmeasured multi-hop behaviour — a multi-hop answer being one the model can only assemble by using a first retrieved chunk to find what it needs in a second — is how a system passes review and keeps giving people the wrong number.

No chunk size spans this#

The tempting reading is that this is the mirror image of the runbook problem from chapter 3, where a chunk longer than the encoder's token limit was indexed by its opening alone — that chunk was too big, this one is too small, so sweep the size and find the middle. The sweep will terminate, it will produce a size at which the runbook stops truncating, and the deadline question will still be answered from a pointer.

The reason is geometric, and it can be settled with a subtraction rather than with an experiment. In this document, section 3.1 sits at token 250 and section 4.2 sits at token 3,340. A chunk containing both must therefore span the distance between those two offsets, which is a chunk of at least 3,100 tokens. Set that against the encoder limit measured in chapter 3 and the two figures meet: 3,100 tokens is over that ceiling by a factor of six, so the smallest chunk that could hold both halves of this fact is six times the largest chunk the encoder will read. The interval of chunk sizes that both spans the reference and stays indexable is empty, and it is empty by a wide margin rather than by a hair.

That emptiness is the most useful thing in the chapter, because of what it licenses skipping. The span came from the document's own token offsets and the ceiling came from a probe against the embedding model, and those two measurements share no step, no instrument and no assumption. When two independent measurements bracket a range and leave nothing inside it, no sweep over that range can find anything, and the experiment can be declined before it is run. A chunk of 3,100 tokens would in any case be a chunk consisting mostly of unrelated material, so even the impossible option is not the option anyone wanted.

Overlap does not help either, and for a reason that is about locality rather than size. Overlap extends a chunk by tens of tokens into its immediate neighbour, and the target here is thirty chunks away. Both dials operate on locality, and a cross-reference is the document's way of pointing somewhere non-local. Enlarging a window and sliding a window are the only two things those dials do, and neither of them reaches a place the document chose because it was far away.

The gap between the two rows is the one worth reading, and it is not a gap in severity. Both failures lose the same thing, which is a fact the corpus contains and the model never sees. They differ in whether a dial reaches the cause. Row A is caused by a length, and length is what chunk size sets, so a dial reaches it. Row B is caused by a distance the author of the document chose, and no chunking parameter is a statement about distances between sections. Reaching for the same dial twice is what makes the second failure look unfixable when it is merely unfixable there.

Row A is also the failure your infrastructure can see. After chapter 3, a single query over the n_tokens column reports how many chunks exceed the encoder limit, and the number is either zero or it is a bug with an address. Row B produces no anomaly anywhere: the chunk is 120 tokens, well under any limit; the retrieval returns a high-scoring hit at rank 1; the generator produces a confident answer with a correct citation; latency is normal; no error is logged. It surfaces as a person saying the bot told them 30 days. That asymmetry is worth holding onto, because monitoring built around the failure you have already fixed will keep reporting green while the other one runs.

Making the referent reachable#

Two repairs work, and they sit at different ends of the pipeline.

Resolving cross-references at ingest inlines the target into the chunk that points at it, so the pointer chunk becomes self-contained and its vector picks up the vocabulary of the answer.

import re

REF = re.compile(r"\b(?:section|clause|§)\s*(\d+(?:\.\d+)+)\b", re.I)

def resolve(chunk: str, sections: dict[str, str], budget: int = 60) -> str:
    seen, out = set(), chunk
    for num in REF.findall(chunk):
        target = sections.get(num)
        if target and num not in seen:
            seen.add(num)
            out += f"\n\n[Section {num}] {first_n_tokens(target, budget)}"
    return out

What that repair changes is the scoring, and the mechanism is the one the trace table already demonstrated, run backwards. The chunk's vector summarises the words in the chunk. Appending the referent's opening puts the answer's vocabulary — window, transaction date — into a chunk that already carried the query's vocabulary. A single chunk now shares words with both halves of the question, so it is the chunk that wins, and the fact arrives with it rather than being pointed at.

Run over a corpus of policy documents, the pattern matched 3,114 references across 12,000 documents and resolved 2,706 of them, an 87 percent hit rate; the remainder were prose references such as "the deadline described above" and "Appendix C" that the pattern does not cover. That percentage is worth reading alongside its residual rather than on its own. Resolved references subtracted from matched references leave the ones still pointing at nothing, and that difference runs to a few hundred — a number that does not shrink because the headline rate is high, and one that has to be tracked in its own right if anybody is to know when it reaches zero.

After resolution the rank-1 chunk reads "Claims must be filed within the window defined in section 4.2. [Section 4.2] The window is 60 days from the transaction date." — one chunk, both halves of the fact, and the generator has nothing to chase. The cost is about 15 percent more tokens in the affected chunks and a second pass over the parsed corpus that runs in minutes.

The retrieval-side repair keeps the small chunks for matching and expands what gets served. Index at 120 tokens, store each chunk's parent section id, and after retrieval replace each hit with its full parent section, deduplicated. Matching stays precise, the served text carries its surroundings, and section 3.1 arrives whole — although note that this still does not deliver section 4.2, because the parent of the pointer is not the parent of the referent. Parent expansion solves the problem of a fact split across adjacent chunks; it does not solve a reference across sections. Applying it and assuming the cross-reference case came along with it is the most common way this repair gets over-credited.

Where these repairs stop#

Cross-references are not a quirk of one policy. They are how documents that were written to be read by a person who can turn a page avoid repeating themselves, which makes them dense in exactly the corpora that end up in retrieval systems: contracts, statutes, insurance policies, compliance manuals, API specifications with a shared error-code appendix. A quick count on this corpus found that 22 percent of documents contain at least one numbered section reference, and the documents that contain them are disproportionately the ones people ask questions about, because a document dense enough to need internal references is a document dense enough to be confusing.

That percentage and the reference count quoted above are a check on each other, and passing the check is what licenses quoting either. Twenty-two percent of twelve thousand documents is rather more than two thousand documents carrying at least one numbered reference, and 3,114 references spread across those documents comes to a little over one apiece. A count of references divided by a count of documents that each hold at least one cannot come out below one. Had it done so, one of the two figures would have been measured over a different set of documents, and neither would be safe to quote.

The ingest-side repair inherits the limits of its own pattern, and the pattern is a regular expression over numbered references. A reference phrased as "the deadline described above" carries no number for it to capture, and one phrased as "Appendix C" carries a label the expression does not recognise as a section number. Those are not rare shapes in prose, and they are the reason the hit rate is a percentage rather than a completion.

The measurement that tells you a fix worked rather than moved the problem has three parts. The first is span containment on the cross-referencing query class specifically, since that class is small enough to be invisible in an aggregate. The second is span containment on everything else, to catch the regression where inlined text dilutes vectors that were fine. The third is a count of unresolved references remaining in the corpus, which is the only number that tells you the size of the residual rather than whether today's example is fixed — and it is the same residual the hit rate above already implies, now tracked rather than inferred.

Both of these repairs lean on something the documents supply: a section numbering scheme, a parent-section id, a heading hierarchy the parser recovered. That is a dependency worth examining directly, because a strategy built on document structure inherits whatever the structure signal is actually worth.

What this chapter asks you to change is a reading habit before it is a configuration. A rank is a report on how many chunks outscored something, and in the flat region of a score distribution that count is decided by differences smaller than the difference between two ways of phrasing the same question. Read the score column, ask how much lift the chunk would need to cross the cut, and compare that lift against the spread of the whole tail. When the lift is a large fraction of the spread, the chunk is not near the cut in any sense that a bigger k addresses.

The deeper point is about what chunking can and cannot reach. Chunk size and overlap both set the size of a local window, and a cross-reference exists precisely because a document needed to say something about a place that is not local. The repairs that work therefore change what a chunk contains or what a hit delivers, rather than how wide the window is: inline the referent at ingest so the pointer chunk carries the answer's vocabulary, or expand a hit to its parent section at retrieval and stay clear-eyed that this reaches adjacent text and not a referenced section elsewhere. Then measure the affected query class on its own, because a class this small disappears into an aggregate, and count what remains unresolved, because that count is the only honest description of how much of the problem is left.