Eviction by component priority

Eviction is what a prompt assembler does when the components it has fetched do not fit the budget: something has to give up tokens, and eviction is the rule that decides which. Most assemblers have no such rule. They have a concatenation order and a prompt[:limit], which produces a rule as a side effect — whatever the rendering happened to put last is what gets discarded.

That side effect is why an assistant sometimes ignores its own formatting rules, and why it sometimes behaves as though a tool it has does not exist. Both are the same defect seen through different symptoms, both appear only on long sessions, and neither of them looks like a context problem from the outside.

The tail is not a priority ordering#

Here is the assembler that produces both:

def assemble(system, tool_docs, chunks, history, ceiling):
    prompt = "\n\n".join(
        [c.text for c in chunks]      # evidence first
        + [m.render() for m in history]
        + [system]                    # instructions last, for recency
        + [tool_docs]                 # tool descriptions last of all
    )
    return prompt[: ceiling * 4]      # trim the tail to fit

Topic 4.5 established that instructions placed near the end of a long prompt are followed more reliably than instructions buried at the top, and that ordering is applied correctly here. What the code does not notice is that the truncation rule and the ordering rule now point at the same tokens. Moving the instructions to the end makes the model follow them, and it also makes them the first thing thrown away.

Trimming the tail is a policy in the sense that it is deterministic, and it is not a policy anyone chose, because the ordering it depends on was decided for a completely unrelated reason. Reverse the concatenation to put instructions first and the same code discards recent conversation turns; group tool docs beside the system block and it discards retrieval. Nothing about the trim expresses a judgement about value. It expresses a judgement about string layout.

The test for whether a rule was chosen is whether it changes when the reason for it changes, and this one fails it in both directions. Nobody who moved the system block to the end intended to make it disposable, and nobody who added the slice intended to pick a victim. Each edit is correct against the reason it was made. What joins them is that a Python slice takes the front of a string, so the component the ordering rule promoted is the component the truncation rule demotes, and the two rules never appear in the same diff, the same review or the same file.

Character-based trimming compounds this in two separate ways. The first is that prompt[: ceiling * 4] reintroduces the divide-by-four estimate a real token count exists to replace, and it does so at the worst possible moment: ceiling is a token budget, the slice index is a character offset, and multiplying by four converts between them using exactly the constant chapter 1 measured as wrong by up to 71 percent. On a prompt that is mostly JSON schemas and code the slice keeps far more text than the ceiling allows, so the trim runs and the request is still refused.

The second is that a character offset lands wherever it lands. A token boundary, a word boundary and a JSON structure boundary are all invisible to it, so the cut can fall inside any of them, and a tool schema can survive as an opening brace and half a description. That is worse than losing the schema outright, and the reason is worth being exact about: a tool that is absent is a tool the model does not attempt, while a tool whose definition is truncated is a tool the model believes it has and calls with arguments that were described in the half of the schema that did not survive. The failure moves from a missing capability to a wrong call.

The fix is to separate the two decisions. Once they are separate, eviction operates on components with a stated rank rather than on offsets in a string:

NEVER_REDUCE = ("system", "tools", "reserve")

EVICTION_ORDER = (
    ("history", "drop_oldest_exchange"),   # whole exchanges, topic 2.14
    ("retrieval", "drop_lowest_ranked_chunk"),
    ("retrieval", "truncate_chunk_bodies"),
)

def evict(components, budget):
    for name, action in EVICTION_ORDER:
        while over_budget(components, budget) and can_apply(components[name]):
            APPLY[action](components[name])
            if components[name].tokens <= FLOORS[name]:
                break
    if over_budget(components, budget):
        raise Unfittable(components, budget)   # a branch, not a silent trim
    return components

Two properties make this different from a trim-until-it-fits loop rather than a tidier version of it. The NEVER_REDUCE set is enforced by the absence of any rule that touches those components, so instructions and tool definitions cannot be lost by accident regardless of how the prompt is laid out. That is a stronger guarantee than a check would be, and the difference matters: a check has to be consulted, and the tail slice never consulted anything. Here the components are unreachable because EVICTION_ORDER is the only thing evict iterates over and their names do not appear in it. And Unfittable is raised rather than swallowed, which turns "we could not fit this" into a branch where the request can be decomposed, routed or refused, instead of a quietly damaged prompt.

The ladder, and where it stops#

Eviction has floors as well as an order, since a slice trimmed to nothing is usually worse than a request that does not run. Retrieval reduced from eight chunks to one is a retrieval-augmented assistant answering from one paragraph, and its answers are confidently wrong in a way that is harder to detect than an error.

For a troubleshooting assistant, the floors come from what the feature is for. History's floor is the last three exchanges plus the pinned first user turn, because a troubleshooting session's identity — which cluster, which region, which version — is usually stated once at the beginning and never repeated, and dropping it produces an assistant that answers confidently about the wrong environment. Retrieval's floor is four chunks, measured: below four, grounding — the share of answers whose claims are supported by the evidence actually supplied — fell by eleven points on the fixed eval set, a frozen collection of graded test queries kept identical between runs, and between four and eight chunks it moved by less than two.

Those two retrieval figures are doing different jobs and it is easy to read only the first. Eleven points between three chunks and four says the floor is in the right place. Under two points between four chunks and eight says that above the floor the material is nearly interchangeable, which is what makes retrieval safe to evict from at all — if every chunk had been worth three points the ladder would need a different second rung. A floor is a claim about where a curve falls off a cliff, and it takes both measurements to establish one: the drop below it, and the flatness above it.

Floors and a rank together give an eviction order with a reason attached to each step. History goes first because turns seven through twelve of a troubleshooting session are mostly the model's own prior answers, which the model does not need to re-read in order to continue. Retrieval goes second because chunk eight is by construction the least similar of the eight and is frequently near-duplicate material. Tool definitions never go, and the reason is a difference in kind rather than in degree. Losing a chunk of evidence leaves the model with less to reason from, and it answers less well. Losing a tool schema leaves the model with no representation of that action at all, so it does not answer less well — it answers a different question, usually by describing a manual procedure it has invented because the thing it would have done is not in the request. Both are losses; only one of them is on a scale where more or less makes sense. Instructions never go for a parallel reason: their loss changes the shape of every answer rather than the quality of one.

What the ladder does to a request 4,500 tokens over#

The abstract order is easier to argue about than to picture, so here is one real request from that troubleshooting assistant, on a 32,000-token window with a 4,000-token reserve subtracted, which puts the input ceiling at 28,000. Retrieval had returned eight chunks totalling 15,900 tokens, over its own 14,000-token cap because one chunk was a single oversized runbook section, and the session carried 12,300 tokens of history:

ceiling 28,000  (window 32,000 − reserve 4,000)
start   system 1,462  tools 2,840  retrieval 15,900  history 12,300
        total 32,502   over by 4,502

rung 1  history.drop_oldest_exchange        (floor: pinned turn 1 + last 3 exchanges)
   ex2  −1,180   history 11,120   total 31,322   over by 3,322
   ex3  −  940   history 10,180   total 30,382   over by 2,382
   ex4  −1,510   history  8,670   total 28,872   over by   872
   ex5  −  760   history  7,910   total 28,112   over by   112
   ex6  −1,020   history  6,890   total 27,092   fits, 908 spare

send    4 exchanges dropped · retrieval untouched · floor not reached

Every number in that trace is one of two subtractions, and following them once makes the rest of the chapter's arithmetic readable. The first is the overrun: add the four slices — 1,462 plus 2,840 plus 15,900 plus 12,300 — and the total is 32,502, and 32,502 less the 28,000-token ceiling is the 4,502 the ladder has to find. The second happens on each rung. Dropping exchange two removes 1,180 tokens, so history goes from 12,300 to 11,120 and the total falls by the same 1,180 to 31,322, which is still 3,322 above the ceiling. Nothing else in the request changed, which is why one figure can be subtracted from both columns. Five exchanges later the total is 27,092, below 28,000 for the first time, and the loop stops.

Two things in that trace are worth reading carefully. Retrieval was never touched, even though it was the slice that had overrun its own cap, because history was above its floor and history is rung one; the ladder is an order of sacrifice rather than an attribution of blame, and the component that caused the overrun is not necessarily the component that pays for it. That is deliberate, and it is also the kind of decision that should be visible in the record rather than emergent from the code, because a reasonable team could rank these the other way.

The second is the last line. Going into rung six the request was over by 112 tokens, and the smallest thing the ladder is permitted to remove was a 1,020-token exchange, so it removed 1,020 tokens to recover 112 — and 1,020 minus 112 is the 908 tokens of window left unused on the send line, on a request that had just discarded conversation to make room. The overshoot is not a defect in this trace. It is the size of the eviction unit, and any policy that removes material in units overshoots by up to one whole unit whenever the shortfall is smaller than the unit. The unit you evict in sets how much you overshoot, and evicting whole exchanges is the coarse-grained choice — taken because a half-exchange is a user question with no answer, or an answer with no question, and both confuse the model more than their absence. A finer rung underneath, such as truncating the body of the lowest-ranked chunk to reclaim exactly the remainder, recovers that overshoot at the cost of a rule that can cut a rate table in half.

The ladder itself runs against projected totals computed once from the per-slice counts, with a single authoritative count before the request goes out. Re-counting after every rung reproduces the pre-flight latency of a trim loop exactly, and the arithmetic does not need it: each rung's effect on the total is the size of the unit it removed, which is already known. That is the property the trace above depends on. Because eviction removes whole units whose token counts were measured when the slice was, the new total is the old total minus a number the assembler is already holding, so the six rows of that trace cost one count rather than seven. A trim loop cannot do this because it does not know what its unit is worth until it has re-measured the string it belongs to.

The same technique, three different orders#

Nothing about that order generalises, and that is the point of writing it down per feature rather than shipping it in a library with a default. What generalises is the question the order answers, which is what this feature's answer cannot be produced without.

The code review assistant is the useful contrast, because the material it protects most fiercely is retrieved material of exactly the kind the troubleshooting assistant evicts second. The diff arrives through the same retrieval path as the related files that give it context, and a reviewer's answer without the related files is thinner while a reviewer's answer without the diff is nothing at all, so the diff belongs in NEVER_REDUCE beside the system prompt and the related files become rung one. The contract summariser has almost no history to give up and a 12,000-token reserve it cannot touch, which leaves it with an eviction ladder of one rung and a refusal branch — a legitimate configuration, and one that makes the Unfittable path a normal part of its operation rather than an edge case.

Summarising history is a different mechanism#

The counter-proposal is always the same: instead of dropping the oldest exchanges, summarise them, so nothing the user said is lost. That is a real technique with real benefits, and it does not do the job an eviction policy does.

The size row decides the argument, and the chain is three links. A summariser is a model call. The length of a model call's output is not a parameter, it is an outcome — asking for "about 800 tokens" produces 620 on one session and 1,450 on another, and the only way to find out which is to make the call. A budget has to know a component's size before assembly, because the caps are inputs to fetching rather than checks after it. So a summariser cannot be the thing that bounds a slice: it can only be a thing that usually makes the slice smaller. Something still has to decide what happens when the summarised prompt is too large, even in a system that summarises aggressively, and that something is a deterministic eviction policy. Summarisation lowers how often the policy fires; it does not remove the need for one, which is why it complements the ladder rather than replacing it.

Where summarisation earns its place is on the axis eviction is bad at. Eviction preserves recent turns verbatim and forgets old ones completely, so a session where the user stated a constraint in turn two and the relevant work is happening in turn thirty loses that constraint entirely unless it is pinned. A rolling summary carries the gist of turns one through twenty in 400 tokens and gives the model something in place of nothing. The two compose: summarise on a schedule outside the request path, treat the summary as a pinned, size-capped part of the history slice, and keep the eviction ladder for the case where even that does not fit.

An order of sacrifice is not an allocation#

An eviction policy is a rule about what happens when the budget is exceeded, which makes it a statement about the order of sacrifice rather than about the sizes themselves. A budget that gives retrieval 14,000 tokens and history 9,300 will evict history on long sessions no matter how carefully the ladder is written, and if this feature's sessions genuinely need 15,000 tokens of history, the ladder will faithfully execute a bad allocation on every request. Frequent eviction is a signal that the allocation is wrong, and instrumenting how often each rung fires is the cheapest way to find out — a history rung firing on 40 percent of requests means the history slice is undersized rather than that the policy is working.

Priority is also not relevance. Dropping the lowest-ranked chunk assumes the retriever's ranking is meaningful, and on a corpus where the retriever scores three near-duplicates above the one document that answers the question, the eviction ladder will dutifully preserve the duplicates. The policy inherits whatever ordering quality the ranker has, and no eviction rule compensates for a ranker that puts the answer in position eight.

What the ladder does guarantee is that the loss is chosen. A prompt that arrives at the model without its formatting rules is a prompt that lost an argument nobody had; a prompt that arrives with four exchanges missing lost an argument someone wrote down, gave a floor to, and can be shown the trace of.

The NEVER_REDUCE set is where the next problem hides. Tool definitions are protected from eviction because losing them removes capabilities, which makes them a fixed cost paid in full on every request, whether the request needs any tools or none. On the agent in that bug report the block was 6,800 tokens on a 32,000-token model, and nobody had ever looked at it.