Eviction order under a token budget

A compaction pass that has been made structurally safe still has to choose what goes. Given a set of removable exchanges and a deficit of 4,000 tokens, the API has no opinion, the token counter has no opinion, and whatever the code does by default becomes the policy — which in most implementations is "oldest first", sometimes "largest first", and in neither case anything anyone decided. Both defaults have a characteristic failure: oldest-first removes the instruction the operator gave at turn 2, and largest-first removes the retrieved passage the current question depends on.

This chapter replaces the default with an order that can be written down and reviewed. It ranks removable material on four properties applied in sequence, shows what each one is protecting against, gives the pinned set that the order is not allowed to touch, and covers the alternative to deletion — a short durable record of a large tool result — that recovers most of the tokens without losing the fact that the action happened. It closes on the standing proposal to buy a larger window instead, which deserves a real answer rather than a dismissal.

Four properties, applied in order#

Take a coding agent at step 31 of a refactor whose next request is 4,000 tokens over budget, with this in its transcript:

The three file contents are the block to drop, and each column contributes to the reason. They are the largest single block by a factor of five, so removing them clears the deficit with one decision instead of a cascade of small ones. They are stale, and of the four blocks the likeliest to be wrong, because the agent has edited all three files since it read them — which means the transcript is showing the model text that does not match the working tree, a positive harm rather than a neutral cost. And they are recoverable at the cost of one cheap tool call if the agent turns out to need them, which is not true of the operating constraints or of the few-shot examples, those being the worked specimens of the required patch format carried in the prompt so the model has something to copy the shape of. The system block is the smallest thing in the transcript and the last thing to remove, which is the point at which size stops being the organising principle.

Generalising that decision gives a policy. Rank every removable block by recoverability first, then by staleness, then by how directly it bears on the decision the model is about to make, and only then by size.

Recoverability comes first because it converts a permanent loss into a latency cost. Anything the agent can fetch again with a tool call — file contents, a directory listing, a schema, a search result — is cheap to evict, since the worst case is one extra round trip. Anything that exists only in the transcript is expensive: the user's original instruction, a decision the agent made and its justification, a constraint the operator stated.

The ordering follows from a difference in the shape of the two costs rather than from a preference between them. Evicting recoverable material has a cost bounded above by one round trip, because the worst that can happen is that the agent needs it and fetches it again. Evicting material that exists only in the transcript has no bound at all — nothing in the system can reproduce it, so the cost is whatever the agent does wrong for the remainder of the session, which nobody can price in advance. Spending the bounded costs before the unbounded ones is the only order defensible without knowing which material the model will turn out to need, and not knowing that is the permanent condition here: the eviction happens before the call that would have revealed it.

Staleness comes second because stale material is worse than absent material. A file read fifteen steps ago and edited twice since does not merely waste 6,000 tokens; it actively describes a state of the world that no longer exists, and the model has no way to know which of the two versions in its context is current.

The reason it has no way to know is that a transcript carries no timestamps a model can read. Every block in the array is presented as equally true and equally current, and there is no field saying that the file contents captured at step 16 have since been superseded. Absent material produces a model that does not know something, and a model that does not know something asks or fetches. Stale material produces a model that knows something false, and a model that knows something false acts. The most common concrete instance is the agent that re-applies a change it already made because the transcript shows the file without it, and what is worth noticing there is that the agent is behaving correctly on the evidence in front of it. The context is what was wrong.

Decision-relevance comes third and is the term that keeps recency from becoming the whole policy. The last four exchanges are usually the most relevant, and usually is not always: in a session where the operator said at turn 2 that migrations must never be run against production, that instruction is fifty turns old and is the single most consequential thing in the array.

Size comes last, as a tiebreaker among blocks that are equally recoverable, equally current and equally irrelevant. Sorting by size first is the mistake that produces an agent which drops its own instructions to make room for a log file.

PINNED = {"system", "operator_constraints", "current_task", "last_2_exchanges"}

def evict(blocks: list[Block], deficit: int) -> list[Block]:
    def rank(b: Block) -> tuple:
        return (
            0 if b.recoverable else 1,          # recoverable goes first
            0 if b.stale else 1,                # stale goes first
            b.relevance,                        # 0.0-1.0, low goes first
            -b.tokens,                          # among equals, biggest first
        )
    freed, removed = 0, []
    for b in sorted(blocks, key=rank):
        if freed >= deficit:
            break
        if b.kind in PINNED:
            continue
        removed.append(b)
        freed += b.tokens
    log.info("eviction", freed=freed, deficit=deficit,
             kinds=[b.kind for b in removed])
    return [b for b in blocks if b not in removed]

The four properties are written as a tuple because a tuple sort is what "in order" means in code. Comparing two tuples compares their first elements, and looks at the second only when the first two are equal, and at the third only when the first two pairs are equal — so a tuple whose members are the properties in the order the policy states them is the policy rather than an implementation of it. Two details in it are easy to get backwards. Each of the first two terms is written so that the material you want removed sorts to zero, because sorted ascends and the loop takes blocks from the front. And b.tokens is negated for the same reason: among blocks that tie on the first three properties the policy wants the largest gone first, and an ascending sort on a positive size would hand back the smallest.

The PINNED set is the part to argue about in review, because it is the only place where the policy makes a claim that cannot be renegotiated at runtime, and a pinned set that grows without limit is a budget that cannot be met.

That last phrase names a specific failure rather than a vague one, and the mechanism matters because it is how a compaction pass stops degrading and starts refusing. Pinned material is subtracted from the window before anything else, so it fixes the ceiling on everything competing for the remainder. As the pinned total rises the removable pool shrinks, and at the point where the pinned material plus the current turn exceeds the window there is no set of removals that closes the deficit at all — the pass has nothing left it is allowed to drop and the request is over the limit anyway. Every addition to PINNED should therefore be read as a claim that the thing is worth more than the largest deficit it could one day make unclosable.

The tool result that should have been summarised#

An agent that repeats a tool call it already made — running the same search, fetching the same file — is showing an eviction-order defect rather than a structural one. A whole exchange was removed to free 9,000 tokens, the exchange contained a directory listing the agent still needed, and from the model's position the search simply never happened.

Deleting is not the only way to free the tokens. A tool result is usually a large body with a small essential summary, and keeping the summary preserves the agent's knowledge that the action occurred and what it broadly returned:

def digest(exchange: Exchange) -> dict:
    """Replace a 9,000-token tool exchange with ~60 tokens of durable record."""
    return {
        "role": "user",
        "content": (
            f"[compacted] step {exchange.step}: called {exchange.tool_name}"
            f"({exchange.args_summary}) -> {exchange.result_digest}"
        ),
    }
# e.g. "[compacted] step 16: called read_file(src/billing/ledger.py)
#       -> 412 lines, defines LedgerService, applyCredit, reverseCharge"

That record is a plain message with no tool blocks in it, so it satisfies the structural rules the previous chapter established — a tool_use and its tool_result must appear as a matched set, and a digest carries neither. It is 60 tokens instead of 9,000, and the agent retains the fact that it read the file plus a pointer to what was in it, which is enough to prevent a redundant re-read and enough to prompt a deliberate one when the details matter. Microsoft's Compaction documentation for its agent framework catalogues the named variants of this — sliding window, summarisation, digest — under an explicit token budget, if you want the vocabulary.

Sixty tokens can do that because of what triggers the repetition in the first place. A model repeats a search when nothing in its context indicates the search happened; this is not remembering and forgetting but reading an array, and an array with no record of an action is indistinguishable from an array describing a session where the action was never taken. A line naming the tool, its arguments and a one-line account of what came back is evidence that the action occurred, and evidence is the whole of what the repetition needed. What sixty tokens cannot do is answer a question about the contents, and that is why the pointer is worth as much as the record: an agent that now needs a specific line of the file knows the file exists, knows roughly what is in it, and can read it again on purpose rather than by accident.

Buying a 200k window does not remove the budget#

The standing alternative to all of this is to move to a model with a much larger window and let nothing be dropped, and it does buy something real: sessions that used to compact at turn 37 now compact at turn 300, and the rate of every failure in the previous chapter falls immediately.

What it does not do is install a budget. Retrieval that was capped at 12,000 tokens because that was all there was room for now returns 40,000, since the cap was never a deliberate quality decision and there is no longer anything forcing the question. Nothing decides what matters when the array grows, so the composition of the prompt is once again whatever the components happened to produce.

The two cost cells are one multiplication each against a rate quoted per million tokens. Eighteen thousand tokens is eighteen thousandths of a million, and eighteen thousandths of three dollars is a little over five cents, which is the $0.054. A hundred and twenty thousand tokens is a hundred and twenty thousandths of a million, and that against the same three dollars is the $0.36. The ratio between the two cells is the ratio of the two token counts and nothing else, since the rate is identical on both sides — which is the row's point, because nothing about the larger prompt was chosen and its cost rose by exactly the factor its size did.

The cost line is the one that gets attention and the position line is the one that matters more. Prompt size is a continuous variable that trades against both money and answer quality rather than a wall you either clear or do not, and a policy sentence competing with 120,000 tokens of retrieved material is in a worse position than the same sentence competing with 15,000, regardless of the fact that both fit. The evidence on long-horizon agent runs points the same way — the GenericAgent work on context density found that compressing the working set outperformed expanding it — and the practical version is that a bigger window is worth buying once you have a budget to spend it under, and is an expensive way to defer an incident before that.

Choosing what to remove is the only step in a budget that is about answer quality rather than about arithmetic, and it is the step most often left to whatever the loop happened to do. An order derived from recoverability, staleness, relevance and size, with an explicit pinned set and an eviction log joined to graded outcomes, is a policy that can be argued with and corrected. A transcript trimmed from the front is not, and its characteristic failure — an agent that stops honouring a constraint it was given fifty turns ago — is indistinguishable from a model quality problem until somebody reads the log.

Everything in this topic converges on a single object: a function that takes the payload you intend to send, counts it against the model you intend to send it to, subtracts what the answer needs, and either returns a payload that fits or names what it removed. Ship that function with three properties and the failures in these seven chapters mostly stop happening. It counts the real payload rather than your strings. It reserves output before it fills input. And it removes whole exchanges in an order that was written down, reviewed and logged, so that when an answer comes back wrong there is a record of what the model was not shown.