Retrieve into the window, then cache what you retrieved

Arguments about long context tend to be conducted between two designs, and the design that usually wins is neither of them. On one side is chunk retrieval: score fragments of documents against the query, keep the best few thousand tokens of them, send those. On the other is stuffing: send everything and let the model sort it out. Both are points on a single axis — how many tokens of a corpus reach the prompt — and treating that axis as having only two settings hides everything between them.

Sweeping the setting rather than choosing between its ends produces a curve with an interior maximum, meaning the best score comes from a value in the middle of the range rather than from either extreme of it. Chapter 4 explains why one has to exist: sending more raises the chance the answer is present and raises the number of passages competing with it, and the first of those effects runs out while the second does not. The measurements below all come from one corpus at one query volume, so the shape they show transfers further than the particular budget that wins.

Reading a budget sweep by its gaps#

The measurement is an evaluation run at five different context budgets over one 1.2-million-token corpus, with the same questions and the same grader throughout, so that the only thing varying between rows is how much of the corpus reaches the prompt.

Accuracy climbs from six thousand tokens to about fifty thousand, flattens, and then falls — the fall being the dilution and position effects of chapter 4 arriving once there is enough irrelevant material for them to act on. The best configuration sends 4% of the corpus, beats both endpoints on quality, comes in at a sixth of the stuffed cost, and lands its p95 within two seconds of the pure retrieval design. There is nothing exotic in it: it is retrieval, with the selection performed at document granularity instead of chunk granularity and a top-k of twelve documents rather than eight fragments. Later chapters call that middle design document selection: retrieval whose unit is a whole document rather than a fragment of one, and whose budget is tens of thousands of tokens rather than a few thousand.

Two of those claims are the table restated rather than measured. Forty-eight thousand tokens out of 1.2 million is 4% exactly, so "sends 4% of the corpus" is the first two columns of one row divided by each other. And $112 against $720 is close enough to one part in six for "a sixth of the stuffed cost" to be a fair rounding of the last column. Neither adds information beyond the row it came from, which is worth knowing before either is quoted in a design document as a separate result.

Divide the cost column by the token column before reading anything else, since that separates the price of a token from the number of them. The top-8 and top-24 rows come out at exactly the same price per thousand tokens, so whatever discount either is getting, the other is getting the same one. The whole-corpus row's price per thousand tokens is exactly a tenth of that — and a tenth is the multiplier a cached read earns, which is also the floor: no cached read is billed below it.

That floor turns the coincidence into a proof. If the chunk rows were themselves discounted, then a tenth of their already-discounted rate would put the whole-corpus row underneath the floor, which the provider does not price. The chunk rows are therefore paying the undiscounted rate, and the whole-corpus row is paying the cached-read multiplier against it, exactly and with no residual. What that licenses is reading the cost column as three numbers multiplied together — daily queries, tokens sent, and a cache multiplier — and nothing else. What it does not license is assuming the same of the two document-selection rows, which share a third price of their own, between the other two. That rate is lower than per-session caching alone produces, which says the column is carrying reuse across sessions as well. How much is a measurement rather than an inference.

Now read the remaining columns by their gaps rather than their levels. The cost column has a near-coincidence in it: the top-24-chunks row and the top-12-documents row sit four dollars a day apart while the second sends nearly three times the tokens of the first, so at this volume the same money buys either more fragments or fewer whole documents. The p95 column repays the same treatment for a different reason. Every row pays the same generation time, since the answers are the same length, so that constant cancels out of every difference and each gap is prefill alone. The gap from the top-8 row to the top-12-documents row is under two seconds for forty-odd thousand extra tokens. The gap from there to the whole corpus is more than four times as large for nearly thirty times as many extra tokens. Prefill is therefore far cheaper per token at the top of the table than at the bottom, which is the cached read of the whole-corpus row showing up in the clock as well as in the bill.

Selecting whole documents rather than fragments#

Chunk-level retrieval optimises for precision in a small budget, which is the right optimisation when the budget is five thousand tokens and the wrong one when it is fifty thousand. A 600-token chunk removes the section heading above it, the definitions clause the section refers to, and the exception three paragraphs down that reverses the rule. That damage was worth accepting when the alternative was not fitting; with a fifty-thousand-token budget it is not.

Raising the chunk count instead is the obvious alternative and the table shows what it buys: top-24 chunks reaches 70.8%, four and a half points above top-8 and two and a half below top-12 documents at a similar token count. The gap between the 18K-token and 48K-token rows is therefore not mostly about volume, since both are small relative to the window; it is about whether the evidence arrives with its surrounding structure intact. Ranks nine through twenty-four are also the weakest chunks the ranker has, so a large part of what a bigger k adds is near-miss material that competes with the correct passage.

The claim about surrounding structure is a causal chain, and it is short enough to check link by link. A chunk boundary is drawn by a splitter that reads character counts, not meaning, so it falls wherever the budget runs out rather than where the argument does. A rule and its exception are usually in different paragraphs, so a boundary between them is likely rather than unlucky. A model given the rule without the exception has no signal that anything is missing, because a chunk carries no marker saying where it was cut. Therefore it answers confidently from half a rule. Widening the unit to the whole document removes the boundary rather than moving it, which is why the fix is a change of granularity and not a change of chunk size.

The implementation keeps the same index and changes the aggregation step. Score chunks as before, then aggregate chunk scores up to their parent document, rank documents, and emit whole documents until the token budget is spent.

def select_documents(query: str, budget_tokens: int = 48_000) -> list[Document]:
    hits = vector_index.search(embed(query), top_k=200)
    scores: dict[str, float] = {}
    for h in hits:                      # max-pool chunk scores per parent doc
        scores[h.doc_id] = max(scores.get(h.doc_id, 0.0), h.score)
    ranked = sorted(scores, key=scores.get, reverse=True)

    selected, used = [], 0
    for doc_id in ranked:
        doc = corpus[doc_id]
        if used + doc.n_tokens > budget_tokens:
            continue                    # skip, do not truncate
        selected.append(doc)
        used += doc.n_tokens
    return sorted(selected, key=lambda d: d.doc_id)   # canonical order

Two lines in that function are load-bearing beyond the obvious. Skipping an oversized document rather than truncating it is the fallback discipline that chapter 7 takes apart in detail; a half document in the window is a document whose second half the model cannot know is missing. And returning the selection in sorted document-ID order rather than relevance order is what makes the next section possible.

Both of those are design decisions rather than requirements, and the difference is worth marking because the two get argued as though they were the same kind of thing. The requirement is that the prompt fit the budget. Skipping and truncating both satisfy it. Ordering by relevance and ordering by document ID both satisfy it. What the two choices buy is a property the requirement never asked for: an honest window in the first case, and a reproducible byte sequence in the second.

The subset varies per query, which is exactly what a cache cannot tolerate#

Everything chapter 2 established about prefix stability applies here with the sign reversed. The whole point of this design is that the content of the window depends on the query, and content that depends on the query cannot be part of a prefix shared across queries. There are three ways to place it, and they have genuinely different economics.

The selected documents can go after a stable shared prefix — instructions, tenant policy, a persona — in which case the cache covers a couple of thousand tokens and the fifty thousand tokens you actually care about are paid at full rate on every request. That is $144 a day at this volume rather than $112, so it is not a catastrophe; it is simply the cache doing nothing useful.

They can be keyed per document set, which is what the sorted return value enables. Assemble the selection in canonical order, hash the document IDs, and the same set produces byte-identical bytes no matter which query surfaced it, so any later request selecting the same twelve documents matches the entry. Whether that pays depends entirely on how often sets repeat, which is a measurement rather than an assumption. On this corpus, sets repeated across sessions 31% of the time — support questions cluster hard on a few policies — and within a session, where the set is pinned after the first query, repetition is near total.

They can be pinned for the session, which is the version the team shipped: select once on the first question, hold the set for the conversation, and re-select only when a query scores poorly against the pinned set. That converts a per-query problem into a per-session one, and the arithmetic is worth writing out because it is much less favourable than chapter 2's.

A session of four queries with one topic shift costs two writes against two reads: the first query builds an entry, the two follow-ups read it, and the topic shift throws it away and builds another. In multiplier terms the average request costs (1.25 + 1.25 + 0.1 + 0.1) / 4 = 0.675 of an uncached read, against 0.1 for chapter 3's shared corpus prefix that 2,000 daily queries all matched. A per-session cache is roughly seven times less effective per token than a shared one, because it amortises a write over a handful of requests instead of thousands. It still pays — 0.675 is better than 1.0 — and it is nowhere near the discount that the phrase "we cache the context" usually implies.

The write fraction sets the multiplier#

That average generalises, and generalising it removes the need to re-derive a number every time a caching scheme is proposed. Two prices appear in it. A cache write costs 1.25 of what the same span would cost uncached, because the provider computes the state and stores it. A cache read costs 0.1 of the same span. Every request pays one price or the other and nothing else, so the average request costs the read price on the requests that read and the write price on the requests that write. Stated as a rule: the multiplier is the read price, plus the difference between the two prices, multiplied by the fraction of requests that write.

That is a straight line in the write fraction. At a write fraction of nothing it gives the read price, 0.1. At a write fraction of everything it gives the write price, 1.25. At a write fraction of one half — which is what the diagram shows, two writes across four queries — it gives 0.675, which is the number the explicit average produced. The line reproduces the worked case exactly, and it does so without referring to the size of the prefix anywhere, because both prices are already expressed as multiples of what that same prefix costs uncached.

Chapter 3's shared corpus prefix sits at the other end of the same line. One write against 2,000 daily queries makes the write fraction almost nothing, so the line returns 0.1, which is what that chapter reports. The factor of roughly seven between the two designs is therefore a ratio of write fractions and has nothing to do with what is being cached, which model is serving it, or how large the prefix is. Two schemes with the same write fraction have the same multiplier whatever else differs between them.

The two dollar figures in the diagram are the same statement in currency, and checking that they agree is a habit worth having. A write of forty-eight thousand tokens is drawn at $0.180 and a read of the same span at $0.014. Divide the first by 1.25 to recover what those tokens would have cost uncached, then take 0.1 of that, and the result is the read price to the tenth of a cent the diagram rounds to. Two multipliers stated in prose and two prices drawn on a diagram close on one implied uncached price for the same span. When a costing does not close like that, one of the two is describing a different span, and finding out which is usually faster than re-running anything.

The line also says which quantity to instrument, and it is not the one most dashboards carry. A hit rate looks healthy under per-session pinning, because a long conversation reads the same entry over and over and a hit rate counts those reads. The write fraction is the reciprocal side of the same tally, and it is what the multiplier depends on. A workload of many short sessions and one of few long sessions can report the same hit rate and sit at opposite ends of the line, because what changes is how many reads each write is amortised over.

The 31% figure is not something to guess at, and it is cheap to obtain from data you already have. Replay a week of production queries through the selection function offline, hash each canonical document-ID list, and count how many distinct hashes the week produced and how the query volume distributes across them. A workload where 60% of queries land on the top twenty sets has a document-set cache worth building and warming ahead of demand; a workload where every query produces a fresh set has one worth skipping entirely. Running that replay takes an afternoon and settles a design argument that otherwise runs on intuitions about how repetitive support questions are.

The lever that moves a p95 regression#

Prefill time is a function of tokens in the window, and chapter 4 established that neither prompt ordering nor cache warmth escapes that — ordering changes what is billed, warmth changes the constant factor, and both leave the term itself intact. Reducing the number of tokens is the only change that acts on the term directly, which makes this design the answer when a long-context feature is rejected on latency rather than on cost.

The magnitude is worth having in mind. Going from 1.2M tokens to 48K cuts the p95 from 12.4 seconds to 4.6, a factor of 2.7, and it does so while accuracy goes up by three points. No amount of cache tuning on the 1.2M design reaches that, because even a perfect hit rate leaves a cache read of 1.2 million tokens, which chapter 4 measured at roughly a third of the cold read rather than a hundredth. When somebody proposes to fix a latency regression by improving the hit rate, the ceiling on that fix is the warm-read latency, and you can look it up before spending the sprint.

That factor of 2.7 is worth putting next to the token reduction that produced it, because the two are not the same size. Twenty-five times fewer tokens buys under three times less wall-clock time, which says a large part of the p95 is not prefill at all — it is generation, plus whatever fixed cost a request pays before any of the window is read. Prefill is the only term this design moves, and it can only ever move the share of the p95 that prefill occupies. Anybody promising a twenty-five-fold latency win from a twenty-five-fold token cut has priced the fixed part at zero.

Operational weight, and corpora with no interior maximum#

The middle design is not a compromise that gets both sets of benefits. It carries the full operational weight of a retrieval pipeline — an index to build, embeddings to keep current when documents change, a selection step whose failures are silent — and adds cache-key management on top, so it is the most moving parts of the three designs by a clear margin. Choose it because the measurements point there and not because it sounds balanced.

The topic shift in the diagram is the honest part. Any pinning scheme trades cache warmth against selection freshness, and a conversation that wanders costs a write per wander. Setting the re-selection threshold too high pins a stale set and answers question four from documents about expenses; setting it too low re-selects constantly and the cache never pays. It is a threshold you tune on transcripts, and the tuning is worth about 15% of the feature's bill.

Two figures in this chapter are both 0.31 and they measure unrelated things, which is a collision worth noticing before it turns into a wrong inference. The 0.31 in the diagram is a relevance score: how well the fourth query matched the set that was pinned for the first three, and the trigger for rebuilding. The 31% in the section above is a repeat rate: the share of sessions whose selected set had already been seen. Nothing connects them. Raising the re-selection threshold does not change how often sets repeat across sessions, and a corpus whose sets repeat more often does not make any individual query score better against a stale pin.

The middle design is also not universally the interior maximum. The curve at the top of this chapter is a property of this corpus, and a corpus whose documents are long and mutually independent will peak at a different budget, while a task that genuinely requires the whole document — reviewing a contract, reviewing a diff — has no interior maximum at all, because any subset is a subset of the thing under review. Chapter 8 uses exactly that distinction to route.

Which of those a workload is can be read off the same line as everything else here. An interior maximum exists when the competitor count grows with the window; it does not exist when the material being sent is the object of the question rather than evidence about it. A contract review has no competitors, because there is only one contract, so its selection term never declines and nothing pulls the curve back down.

The design in the middle of the curve wins twice and is paid for in a third currency. It sends less than a twentieth of the corpus and beats both endpoints on accuracy, and it cuts the p95 by nearly a factor of three, and both of those follow from tokens in the window being the one term that quality and prefill share. What it does not inherit is the caching discount a shared prefix earns, because the content of its window is chosen per query. The write fraction is the number that says how much of that discount survives: a scheme whose entries are read many times per write keeps most of it, and one that rebuilds on every query keeps none.

What all three designs now share is that their differences are measured rather than argued, which is convenient for this chapter and not yet enough to get a switch approved. A comparison that means something has to hold the right things constant, report the numbers that decide, and be honest about which result would change the decision. Chapter 6 designs it.