Deriving the allocation

An allocation is the set of caps a context budget assigns to its slices — system, tools, retrieval, history, reserve — and it is the artefact a design review actually argues about. Five numbers on a slide look the same whether they were derived or observed, and the difference is entirely in what sits behind them. One possibility is that a slice holds the tokens it happened to be using when somebody wrote the budget down. The other is that below that size the product measurably gets worse, by an amount somebody has recorded.

Only the second survives the follow-up questions, and the questions are always the same three. Where did the number come from. What happens if I take some of it. How will you know when it stops being right. A slice that can answer all three has an observed size, a measured quality delta and a named consequence attached to it.

The allocation, with its evidence#

The last row is the one that gets left off slides and is worth defending explicitly. Topic 2.14 established that a local tokenizer and the provider's own counter can disagree by a few percent on a large prompt, and at the boundary that disagreement is the difference between a request that goes through and a 400. Five hundred tokens on a 32,000-token window is 1.6 percent of capacity held back against a measurement error, and if your assembler counts with the provider's endpoint on every request the slack can be smaller. It should be a line in the budget either way, because an unallocated remainder that nobody named will be claimed by the next person who needs 400 tokens.

How a slice gets a number#

An observed size tells you what a component currently costs and says nothing about what it is worth, so the derivation has two stages.

The p95 sets a floor of plausibility, and it does it by division. Sections on this corpus have a median size of 2,150 tokens, so a retrieval slice of 8,000 tokens holds three of them at typical sizes with room to spare — which is what the sweep below reports for that cap. Sections at their p95 are 2,900 tokens, and three of those come to more than 8,000, so on the one request in twenty where the sections are that large the third section does not fit and is truncated. That is a fact worth knowing before any quality measurement, because it means a cap does not deliver a fixed number of sections. It delivers a number that falls in exactly the traffic where the documents are densest.

The quality delta is what turns a plausible size into an entitlement, and it comes from holding everything else fixed and sweeping one slice against an eval set that does not change:

EVAL = load_cases("support_eval_v4.jsonl")      # 180 cases, frozen since March

def sweep_retrieval(caps: list[int]) -> list[dict]:
    rows = []
    for cap in caps:
        budget = replace(BUDGET, retrieval=cap,
                         history=BUDGET.history + (BUDGET.retrieval - cap))
        scores, latencies, costs = [], [], []
        for case in EVAL:
            req = assemble(case.query, case.session, budget)
            t0 = time.monotonic()
            resp = client.messages.create(**req)
            latencies.append(time.monotonic() - t0)
            costs.append(price(resp.usage))
            scores.append(grade(case, resp))     # grounded-and-correct, 0 or 1
        rows.append({"cap": cap, "grounding": mean(scores),
                     "p95_latency": p95(latencies), "cost": mean(costs)})
    return rows

The line that hands the difference to history is the one doing the experimental work, and it is easy to leave out. Take it out and a run at retrieval=10_000 has 4,000 tokens sitting idle, so what the run measures is retrieval against an empty window — and retrieval will win that comparison at every cap, because more evidence is never worse than unused space. Put the line back and those 4,000 tokens go to history, so the run measures retrieval against the thing that would actually have them. That is the question a design review is asking: not "is retrieval worth having" but "is the next thousand tokens better spent here or there". A sweep that leaves the difference unspent answers a question nobody has. For one support assistant:

cap      chunks   grounding   p95 latency   $/req
 8,000      3       0.62         2.9s       0.031
10,000      4       0.71         3.2s       0.037
12,000      5       0.78         3.4s       0.043
14,000      6       0.86         3.7s       0.049
16,000      7       0.87         4.1s       0.055
18,000      8       0.87         4.4s       0.061

Read that block one row at a time and the shape of the number falls out. Between 12,000 and 14,000 tokens the grounding score goes from 0.78 to 0.86, which is eight points for one extra section. Between 14,000 and 16,000 it goes from 0.86 to 0.87, which is one point for the same extra section, and between 16,000 and 18,000 it does not move at all. A curve that rises steeply and then flattens has a knee — the point where extra tokens stop buying quality — and it is at six sections here.

Both sides of 14,000 are then defended by that knee, and it is worth spelling out both, because a cap defended only from below is a cap somebody will raise. Against 18,000: the seventh and eighth sections cost 4,000 tokens, and the same two rows say they cost 700 ms of p95 latency, since 4.4 seconds less 3.7 is seven tenths of a second, and $0.012 a request, since $0.061 less $0.049 is $0.012. What they buy is one point of grounding. Against 12,000: the step down from six sections to five gives back 2,000 tokens and costs eight points. Same curve, opposite direction, and a reviewer who wants 10,000 now has to argue that a grounding score of 0.71 is acceptable for this feature rather than that 14,000 sounds high.

The size of the difference matters as much as its direction, and an eval set of 180 cases is small enough that this needs stating in the review. A proportion near 0.85 measured over 180 cases carries a standard error of about 2.7 points, which is the amount the score would be expected to move if the same 180 cases were replaced by a fresh 180 drawn the same way — it comes from the score and the sample size and from nothing about retrieval. A difference has to be roughly two of those before it is worth believing, which puts the threshold at about five points — under that, a delta is not distinguishable from resampling the same traffic. The eight-point step from five sections to six clears that bar comfortably. The one-point step from six to seven is well under one standard error, which is why the curve is described as flat after the knee rather than as still rising slowly: an eval of this size could not tell the difference between a one-point gain, no gain, and a one-point loss. A reviewer entitled to push back on 14,000 tokens is equally entitled to push back on any claim built from a difference the eval could not have detected.

History is derived the same way, with one adjustment that changes the answer. Sweeping the history cap across all 180 cases showed almost nothing, and the reason is arithmetic rather than a property of history. Two thirds of the eval's sessions are under six turns, so their history never reaches 5,000 tokens, so raising the cap above 5,000 changes nothing about the prompt those cases produce, so their scores are identical at every cap in the sweep. Two thirds of the cases contributing an unchanging score to a mean is two thirds of the mean held constant, and whatever the remaining third does gets divided by three before it reaches the number anyone reads. A flat curve here is not evidence that the cap does not matter. It is evidence that most of the eval is not sensitive to it, which is a fact about the eval. Splitting the eval by session length gave the real picture: on the 41 cases with more than twelve turns, multi-turn resolution rose from 0.68 at a 5,000-token cap to 0.81 at 9,000 and then to 0.815 at 13,000. Twelve exchanges is where this feature's sessions stop carrying new constraints, which matches what the transcripts show — users state the environment early, describe the symptom in the first few turns, and after that the conversation is mostly the assistant's own prior reasoning, which is the same property that makes history the first slice an eviction ladder gives up.

Where the next thousand tokens should go#

Sweeping each slice separately answers where its curve flattens and leaves a second question open, which is what to do with capacity that is genuinely spare. The 32,000-token window above is fully allocated, and the interesting version of the exercise is the one where a tool-schema audit hands back 3,000 tokens and someone has to decide which slice receives them. The sweeps already contain that answer, because the local slope of each curve at its current cap is the marginal value of the next thousand tokens:

Reserve wins that comparison for a reason worth being explicit about, which is that it is the only slice whose marginal token is not spent on every request. Follow what happens to a thousand tokens in each case. Added to retrieval, they are a thousand more tokens of text in the prompt, so they are prefilled and billed on all 128,000 monthly requests — about $0.003 and 90 ms each, whether or not that request needed them. Added to the reserve, they are a thousand tokens that nothing writes into. The input ceiling drops by a thousand and the space sits empty unless an answer runs long, so it is charged on the answers that use it and on nothing else. One is a cost per request and the other is a cost per long answer, and on this traffic long answers are two percent of the total.

That asymmetry makes the reserve the cheapest place to put slack, and it is also the argument for the second-largest recipient being history. History's gain is concentrated in a segment rather than spread across the mean, and a mean is exactly the wrong instrument for that: 0.012 of improvement occurring on 18 percent of traffic and nowhere else averages to about 0.002 across the whole, which is a true number and a misleading one. Nobody experiences 0.002. The sessions that reach turn thirteen experience 0.012, and the rest experience nothing. The table does not resolve that judgement; it makes the judgement visible enough to be made deliberately rather than by whoever edits the constants first.

Cost and latency deserve the same treatment, which is why the sweep records them alongside quality. A reviewer asking for 10,000 tokens of retrieval is asking for $0.037 per request and a 3.2-second p95 rather than $0.049 and 3.7 seconds, and the honest answer names all three numbers: 0.71 grounding, $0.012 cheaper, half a second faster. The monthly figure is that per-request difference times the traffic — $0.012 saved on each of 128,000 requests is about $1,540 — so the question stops being whether 14,000 is justified and becomes whether this feature can accept a grounding score of 0.71 to save $1,540 a month. That is a decision the room is competent to make, and it is a different decision from the one it was making before the two cost columns were written down, when the only number on the table was the one the person arguing for it happened to have.

Writing it down where it can be argued with#

An allocation that lives in five constants at the top of a module will be edited by whoever next needs room. Keeping it as a record with its evidence attached makes the edit a conversation:

# budget/support_assistant.yaml   — reviewed 2026-02-14, next review 2026-05
model: claude-sonnet-4-5
window: 32000
slices:
  system:    { tokens: 1500, evidence: "measured 1462; ci asserts <=1600" }
  tools:     { tokens: 3000, evidence: "19 tools, 2840 after schema audit PR#4127" }
  retrieval: { tokens: 14000, evidence: "sweep 2026-02-11: knee at 6 sections;
                                         12k -> grounding 0.86->0.78" }
  history:   { tokens: 9000,  evidence: "12 exchanges @ 740 mean; >12-turn
                                         subset flat above 9k" }
  reserve:   { tokens: 4000,  evidence: "output p99 3810 over 500 samples" }
  slack:     { tokens: 500,   evidence: "counter disagreement at boundary" }
alerts:
  eviction_rate: { history: 0.15, retrieval: 0.05 }   # page if exceeded
  length_stop_rate: 0.01

The alerts section is what keeps the record from going stale silently. Eviction rates are the running check on whether the allocation still matches traffic: if the history rung starts firing on 30 percent of requests rather than 15, sessions have got longer and the 9,000-token slice is now wrong, and that shows up as an alert rather than as a slow drift in answer quality that nobody attributes to anything. The length_stop_rate does the same job for the reserve.

The allocation also needs an enforcement point, or it is a slide. The __post_init__ assertion from chapter 2, where the caps became constructor arguments and over-allocation became a startup failure, is what converts these numbers into a contract the codebase keeps: the day a pull request raises retrieval to 16,000 without lowering anything else, the sum of the slices exceeds the window by exactly the amount that was added, the service fails to start in staging with over-allocated by 2000, and the author has to name the slice that pays. The record and the assertion are two halves of one mechanism. The record says what each number is for and the assertion says that the numbers still add up, and a change that breaks either one is caught by the other.

The two edges back into the sweep are the whole point of drawing it. One is an alert, which fires when traffic has moved away from the distribution the caps were derived on, and the other is the calendar, which catches the slower drift that no threshold notices. A budget with neither edge is a measurement that was true once.

Every number here is conditional on the eval set#

An eval set is a sample of a distribution that moves. The 180 cases were drawn from support traffic in one quarter, and if the product ships a feature that generates a new class of question — customers attaching JSON logs is exactly such a case — the sweep's knee can move without anyone re-running it. A frozen eval set is what makes the comparison across caps valid and is also what makes it expire, so the review date in the record matters as much as the numbers.

Aggregate scores also hide segments, which the history sweep demonstrated by accident: across all 180 cases the cap looked irrelevant, and on the 41 long sessions it was worth thirteen points. A slice can be correctly sized for 80 percent of traffic and badly undersized for a segment the mean absorbs, and the segments worth splitting on are usually the ones with different token economics: non-Latin locales, sessions with attachments, queries against the densest documents. Reporting one grounding number per cap is enough for a sweep to find a knee and not enough to promise the allocation serves everyone.

Nor does any of this transfer between features. The same model, the same retriever and the same corpus will want a different split for a summarisation feature than for a troubleshooting assistant, because the summariser's reserve is 12,000 tokens and its history is nearly nothing. A budget is a per-feature artefact, and copying one across is how a feature ends up with a 4,000-token reserve for a task that emits 9,000.

What the evidence column buys, then, is not certainty about the numbers. It is that every one of them can be attacked on its merits by somebody who disagrees, and that the attack has a specific form — re-run the sweep, show a different knee — rather than the form these arguments usually take, which is two people asserting different constants at each other.

The review had a second question after the one about retrieval, and it is the one everybody eventually asks: if the constraint is this expensive to manage, why not move to a model with a 200,000-token window and stop managing it.