Choosing a chunk cutoff

The previous chapter established the shape of the curve without locating its peak. Too few chunks and the evidence is not in the prompt at all, which recall measures directly. Too many and the evidence is present and outcompeted by passages that resemble it. Between those there is a maximum, and finding it is a sweep rather than an argument.

That maximum is not an empirical accident, and it is worth seeing why it has to be there before running anything, because the same argument says what moves it. Two quantities change as k grows and they change in different shapes. Adding one more chunk can only help on the queries whose answer was not already in the list, and every chunk added shrinks that population — the first few catch the easy queries, and each further chunk is left with a harder residue to rescue. So the benefit of widening arrives with diminishing returns. The cost does not. A near-miss chunk is pasted into the prompt on every query, including all the ones that were already being answered correctly, so interference accumulates roughly in step with the number of chunks instead of tailing off.

Add a term that flattens to a term that keeps climbing and the sum rises, turns over and falls. That is an interior maximum, and it is why the curve has a peak rather than a plateau you can sit anywhere on or a slope worth riding to the end. The same decomposition says where the peak will sit, which is the part that does not transfer from someone else's write-up, because both terms are properties of your own document collection. How quickly the benefit term flattens depends on how often your answers are ranked below the first few chunks. How steeply the cost term climbs depends on how much near-duplicate material the collection contains. A store of superseded policy revisions and regional variants has a steep cost term and an early peak; a store of unique, well-separated documents has a shallow one and a late peak.

This chapter spends its length on a single integer because that integer is usually set once, by whoever wrote the retrieval call, and then inherited by everything downstream. The sweep that locates it costs a few thousand model calls, and it produces two further findings on the way: that the peak differs by query type, so one global number is a decision to lose points on one population or the other, and that the peak moves again when you change the order the chunks are pasted in. A reranker — a second, more expensive scoring pass over the first-stage candidates, which reads the query and each candidate together instead of comparing precomputed vectors — changes the whole curve, which is why the productive sequence is to rerank first and cut second.

Sweeping k on data you did not tune on#

The measurement is the distractor harness of the previous chapter with one change: the gold chunk is no longer pinned into the prompt. Whatever the retriever returns at each cutoff is what the model sees, which means the sweep captures both failure modes at once — missing evidence at small k and interference at large k — and the peak falls where they balance.

def evaluate_cutoff(queries, k, rerank=False, order="desc"):
    correct = recalled = 0
    for q in queries:
        cands = index.search(q.text, n=100)
        if rerank:
            cands = reranker.rank(q.text, cands)          # precision pass
        chunks = arrange(cands[:k], order)                 # see below
        recalled += any(c.id == q.gold_id for c in chunks)
        correct  += judge(ask(assemble(q.text, chunks)), q)
    return {"k": k, "rerank": rerank, "order": order,
            "recall": recalled / len(queries),
            "accuracy": correct / len(queries)}

held_out = load("eval/held_out_480.jsonl")   # never used for tuning
grid = [evaluate_cutoff(held_out, k, rerank=r)
        for r in (False, True)
        for k in (2, 4, 6, 8, 12, 20, 50)]

Reporting recall alongside accuracy at every point is what makes the output interpretable rather than a single mysterious hump. Below the peak, recall and accuracy rise together and the message is "you are missing evidence". Above it, recall keeps rising while accuracy falls, and the message is the interference the previous chapter measured. The k where those two diverge is the answer.

The recall column is the benefit term, measured. The step from k=2 to k=4 buys more recall than the step from k=20 to k=50 does, and the second step adds far more chunks than the first — each additional chunk is competing for a smaller pool of queries that the chunks above it did not already serve. That is the shape the opening argument predicted, and it is also why the accuracy columns can turn over while the recall column is still climbing. By the time k reaches the teens the benefit term has nearly flattened, and the cost term has not flattened at all.

The reranked column peaks earlier and higher, which is the whole argument for the sequence in this chapter. A reranker raises the density of useful chunks near the top of the list, so a given accuracy is reachable with fewer of them; six reranked chunks beat twenty unreranked ones by 9.9 points while costing roughly a third of the input tokens. Reranking without then cutting captures a fraction of the benefit — the reranked column at k=20 is only 3.5 points above the unreranked one — because a better ordering of twenty chunks still puts nineteen non-answers in front of the model.

Earlier and higher are two separate effects and they come out of two different terms. Reranking acts on the benefit term: at any given k, a list ordered by a model that read the query and the chunk together is likelier to have the answer inside the cutoff than a list ordered by comparing vectors that were computed before the query existed. So the benefit term is higher everywhere and flattens sooner, which lifts the peak and moves it left. Reranking does not touch the cost term at all, because that term counts chunks and a reordering changes no counts. That is the reason the reranked column recovers so little at k=20: the improvement is confined to the term that has already stopped mattering by the time the cutoff is that wide.

This table is also the constructive half of the answer to an engineer who wants to retrieve fifty chunks so that nothing is missed. Their worry is that the top 20 sometimes misses the answer, and the recall column confirms it: recall@20 is 91.3% and recall@50 is 94.6%, so roughly one query in thirty does have its answer in the tail. The disagreement is only about what to do with those thirty candidates. Pasting them into the prompt trades 3.3 points of recall for 8 points of accuracy at the same cutoff. Retrieving them, scoring them with a reranker and then keeping six gives the tail a chance to be promoted on merit while the prompt gets shorter, which is the configuration that wins the table. Retrieve wide and generate narrow are compatible positions once the two stages are separated.

The same two terms dispose of the strongest version of the widening proposal, which is not fifty chunks but all of them: move to a model with a very large window and pass the whole collection, so that nothing can possibly be missed. It is tempting for a reason worth respecting rather than dismissing. A context window is one of the few numbers in this area that a provider states exactly and enforces exactly, a larger one genuinely removes a hard constraint teams have spent real effort working around, and it requires no retrieval engineering at all. None of that is wrong.

What is wrong is the inference drawn from it. A window governs which requests are accepted, not which parts of an accepted request get used, and nothing in an API reports the second. Read against the two terms, passing everything is the far right-hand end of the sweep. The benefit term is exhausted, because a collection large enough to need an enormous window is one where the answer was already inside the first few chunks for almost every query. The cost term is at its maximum, because every near-miss in the collection is now in the prompt on every query, including the queries that were already correct. Both accuracy columns above are falling from k=8 onward and are still falling at the widest cutoff measured, and passing everything is a much larger step in that same direction.

The condition under which this bites is worth stating, because it is not universal. It requires a collection that contains near-misses. A service whose entire document set is small, unique and non-overlapping has a flat cost term and nothing to tune, and for that service a large window really is close to free. Collections that motivate retrieval in the first place are rarely like that, which is what the previous chapter's tour of ranks 21 to 50 was about.

Choosing the peak by eye invites overfitting to a set of 480 queries, where the standard error on a proportion around 0.8 is about 1.8 points. The difference between k=6 at 80.9% and k=8 at 80.4% is not a difference. A defensible rule is to take the smallest k whose accuracy is within one standard error of the maximum, which here selects 6 and has the useful property of breaking ties towards the cheaper configuration. It is the same convention used for choosing regularisation strength by cross-validation, and for the same reason: when the curve is flat near its top, prefer the simpler end of the flat region.

Two facts underneath that number are worth having in hand, because together they decide how large a held-out set has to be. The standard error of a proportion is built out of two quantities: how uncertain the outcome of a single query is, and how many queries were averaged. A score near 0.8 means each query is a coin that lands right about four times in five, and dividing that per-query uncertainty by the number of queries and taking a square root gives the spread of the average. On a set of 480 queries, that comes to about 1.8 points.

The square root is the half that governs planning. The spread falls with the square root of the item count rather than with the count, so quadrupling a held-out set halves the wobble and buys nothing beyond that — the same arithmetic the position probe in the first chapter of this topic runs on. The other half is that a comparison between two cutoffs is not one measurement but the difference of two, each wandering on its own, and a difference of two independent measurements wanders further than either alone, by a factor of the square root of two. Half a point between k=6 and k=8 is therefore not a small effect measured precisely. It is nothing, reported to one decimal place.

That is what makes the one-standard-error rule a decision rather than a habit. Every k inside the band is indistinguishable from the best on the evidence you collected, so the sweep cannot order them, and a tie the measured criterion cannot break has to be broken on criteria it did not measure — fewer tokens, lower latency, and less of the interference the previous chapter priced, all pointing the same way. What the rule protects against is subtler than overspending. Reading a peak off a curve by eye fits the cutoff to the noise in one evaluation set, so whichever k drew the luckier sample of queries wins, and that win does not reproduce on the next set. A rule that declines to choose inside the noise band cannot be captured that way.

One number serving several query types#

Splitting the same sweep by query type shows that the aggregate peak is an average over populations with different optima.

Single-fact lookups peak at 6 and lose ten points by 20, because there is one right chunk and everything else is competition. Aggregation questions — how many incidents did each region log in Q3, which of our contracts renew before March — are still climbing at 12 and only flatten around 16, since the evidence genuinely lives in many chunks and starving the prompt removes some of it. A global cutoff of 6 costs the aggregation queries 8.4 points; a global cutoff of 12 costs the lookups 5.7 points. Running one number for both is a decision to lose one of those, and it should be a decision rather than an accident.

Routing on query type is not exotic to implement: a small classifier or an intent field you already have on the request selects a k per class, and the sweep gives you the per-class value. Whether it is worth it depends on the mix, and the same table tells you — if 80% of your traffic is lookups, a global 6 costs you very little in aggregate, and if the mix is even it costs several points.

Ordering moves the peak as well as the height#

Because identical evidence scores differently at different positions in a prompt — highest at the ends, lowest in the middle — where a chunk lands inside the block matters, and the cutoff sweep interacts with it. Three arrangements of the same reranked top-k, on the same held-out set:

The edges arrangement is the one that follows directly from a U-shaped position curve: put the two chunks you are most confident about where the model reads most reliably, and let the middle hold the ones you are least sure of. It buys 1.3 points here and, more interestingly, moves the optimum from 6 to 8 — with the strongest evidence protected at both ends, the marginal chunk does less damage, so a slightly larger prompt is affordable. That coupling is why cutoff and ordering have to be swept together rather than tuned one after the other; a k chosen under descending order is not the k you want under a different arrangement.

Both of those effects were predictable before the sweep was run, from the account of position bias in the first chapter of this topic: the ends of a prompt beat the middle because material at either end has an advantage the middle lacks, and the middle is worst because it is crowded by its own nearest competitors. If that is what is happening, then which chunk occupies the crowded region is a choice rather than a consequence, and the cheapest chunks to put there are the ones you trust least. That predicts the gain in height. It also predicts the shift in the optimum, and it does so through the cost term: a marginal chunk added to a list whose two strongest pieces of evidence are pinned at the ends arrives in the region where the model weighs material least heavily, and the evidence it competes with there is the weak evidence rather than the answer. The cost term climbs more slowly, so its crossing point with the benefit term moves right.

The gains here are small and they are the least stable numbers in this topic. The right conclusion is not "use edges", it is that ordering is a free parameter with a measurable effect, and that a system pasting chunks in reranker order has made a choice by default rather than by evidence.

Where a tuned cutoff drifts#

The sweep optimises one parameter against one evaluation set, and it inherits every limitation of that set. A cutoff tuned on a held-out sample of last quarter's queries is calibrated to last quarter's query mix, and it will drift when the product ships a feature that changes what people ask. Re-running the sweep quarterly, and after any model change, is the maintenance cost of having tuned it at all.

It also does not address a failure that the numbers in this chapter quietly contain. Reranking and cutting improved accuracy substantially and did not restore it to the level the same model reaches on short prompts with the same evidence. Some of the remaining gap is retrieval — recall@6 is 81.2%, so a fifth of queries never see their answer — but not all of it, and the distractor sweep of the previous chapter pointed the same direction by showing accuracy falling even with the gold chunk pinned in place.

A chunk cutoff is one of the few parameters in a retrieval-augmented system where the cheap setting and the accurate setting coincide, and the only way to know which value that is on your data is to sweep it with both numbers on the axis. What the sweep also buys is an argument that ends: the engineer who wanted fifty chunks and the engineer who wanted six are both holding a legitimate concern, and the table settles it in an afternoon rather than in a quarter of alternating changes.

If chunks that are topically confusable are only part of the story, then something else is degrading as the prompt grows, and separating the two requires an experiment in which nothing about the evidence changes at all — no retrieval, no near-misses, the same document and the same question, with the prompt made longer by material that could not possibly be mistaken for an answer. That experiment is the next chapter, and it is the one that decides whether a sprint belongs on the retriever or somewhere else entirely.