The compaction trigger

The trigger is the predicate that decides a transcript has outgrown its budget and compaction should run. It is the least interesting part of a compaction pipeline and the part most likely to end a session outright, because every other stage decides what survives while the trigger decides whether a request is sent at all.

The version most harnesses start with is a counter: every twenty assistant turns, compact. It is one line, it needs no token counting, and it can run for months without producing a single context-length error, which is a good enough record that nobody looks at it again. What it encodes, without saying so, is that the token cost of a turn is bounded and roughly constant. Where every turn costs between 400 and 3,000 tokens, twenty turns mean somewhere between 8,000 and 60,000, and a check every twenty turns has enough margin to be safe. A single tool that can return tens of thousands of tokens makes one turn capable of consuming more window than the entire interval the check was sized for.

The two ends of that range are the assumption stated as arithmetic, and stating it is what shows when it expires. Twenty turns at the low end of 400 tokens is 8,000; twenty at the high end of 3,000 is 60,000. Against a window measured in hundreds of thousands, an interval that can cost at most 60,000 leaves room to spare at every check, so the counter is genuinely safe — not by luck, but because somebody's implicit upper bound on a turn was correct. What makes it a time-bomb rather than a shortcut is that the bound is nowhere in the code. It is not asserted, not documented, and not checked, so nothing anywhere in the system notices when a new tool violates it. The counter goes on returning the same answer with the same confidence after its premise has stopped being true, and that is the general form of the failure: an assumption that is load-bearing and unwritten cannot be invalidated, only violated.

What a rejected request looks like from inside#

Here is the arithmetic of a request that a counter cannot see coming. The session is fourteen assistant turns in, so a twenty-turn counter is six turns from firing, and the transcript stands at 158,200 tokens against a 200,000-token window.

The running-total column is the whole story and it is worth reading down. The transcript stands at 158,200, comfortably inside the window, and the assistant turn carrying the tool call adds 380 to reach 158,580 — still comfortable. The tool_result adds 45,300 in one block, and 203,880 is already past the 200,000-token limit before anything has been reserved for the answer. Adding the 8,192 tokens of max_tokens takes it to 212,072. Two things follow from where the line is crossed. The request was doomed by the tool result alone, so no completion reserve, however modest, would have saved it. And the crossing happened inside a single turn, between one API call and the next, which is the window a turn-count check cannot see into.

The next request is rejected with a 400, the status code the API returns when it refuses a request outright instead of running it, and the rejection is the last event rather than the first. Three things go wrong in sequence and only the third is visible in the logs.

The first is that the check ran in the wrong place. A turn-count check fires between assistant turns, so the window between two checks contains an unbounded number of tool calls and an unbounded volume of tool output. Even a token-based check placed at the same point would have missed this, because the transcript was at 158,200 when the interval began and the growth happened inside it.

The second is that by the time anything notices, no compaction can help. The oversized result is already in the transcript and it is the newest thing in it, so a compactor triggered at this moment summarises the 158,200 tokens of history the agent needs and leaves the 45,300-token grep dump entirely alone. Running it recovers perhaps 30,000 tokens and produces a request that fits, containing a summary of the session and a wall of grep output, which is close to the worst possible use of the window. There is a mechanical problem alongside the editorial one: the summariser call has to read what it summarises, and on a transcript this size that call is itself within a few thousand tokens of its own limit.

The third is that the failure surfaces as an API error at the end of the work, with no partial result and no recovery path unless something durable was written as the session ran.

The instinctive repair is to compact more often, and it is worth saying why the instinct is reasonable before saying why it fails. Compaction is the thing that makes a transcript smaller, the transcript got too big, and the check that would have run was six turns away — so running it sooner looks like it addresses the gap directly. The step that does not hold is the assumption that the request which failed contains material a compaction could have removed. Order the events: the compaction runs, the request is sent, the model emits a tool call, the tool returns 45,300 tokens, and those tokens are appended. Every one of those steps is after the compaction and before the next call. However often the compactor runs, it always runs before the oversized result exists, so the request that fails is byte for byte the same request under any interval.

The instinctive repair is to compact more often — every turn, or every five — and it addresses none of the three. Compacting after every turn still leaves the oversized result in the request that fails, because the result arrives after the compaction and before the next call. It costs one summariser call per turn, which on a session of this shape is about 2.5 seconds and $0.006 added to every single turn. And it repeatedly compresses recent context, which is the material the agent most needs intact: an agent that summarises the turns it is still working inside will lose exactly the details it is about to need.

A budget is measured, reserved, and checked after tool results#

The replacement has three properties the counter lacked. It measures rather than infers, it reserves room for what has not been generated yet, and it is evaluated at every point where the context can grow rather than only at turn boundaries.

CONTEXT_LIMIT = 200_000
COMPLETION_RESERVE = 8_192        # must match max_tokens on the request
TOOL_HEADROOM = 12_000            # p95 tool result size, measured
SAFETY = 4_000                    # tokenizer estimate error + block overhead

def should_compact(ctx: Context) -> bool:
    used = ctx.measured_tokens()  # see below: estimate, reconciled with usage
    ceiling = CONTEXT_LIMIT - COMPLETION_RESERVE - TOOL_HEADROOM - SAFETY
    return used > ceiling         # 175,808 on these numbers

def turn(ctx: Context) -> Message:
    if should_compact(ctx):
        ctx = compact(ctx)
    resp = client.messages.create(**ctx.as_request(max_tokens=COMPLETION_RESERVE))
    for block in resp.tool_use_blocks():
        result = run_tool(block)              # may be enormous
        ctx.append(clamp(result))             # clamp BEFORE it enters context
        if should_compact(ctx):               # check again, immediately
            ctx = compact(ctx)
    return resp

The ceiling is a subtraction, and each term in it is money set aside for something that has not been written yet. Start from the 200,000-token limit. Take out 8,192 because the model is going to generate that many tokens and they occupy the window alongside the input. Take out 12,000 because a tool result of typical size may arrive before the next check. Take out a further four thousand for safety, because the token count is an estimate and block overheads are not in the message text. What is left is 175,808, and used is compared against that rather than against 200,000. The comparison is deliberately conservative in the same direction every time, which is the property a threshold needs: it is allowed to fire when it did not have to, and it is not allowed to fail to fire when it had to.

The TOOL_HEADROOM term is what the counter had no way to express. Compaction has to be triggered early enough that the next tool result still fits, which means the threshold is not "the context is nearly full" but "the context plus a plausible next tool result plus the completion is nearly full". Setting it from the measured 95th percentile of your own tool results, rather than a round number, is the difference between a threshold that fires usefully and one that fires constantly.

That term exists because of when the check can run relative to when the context grows. A compaction is only possible between calls, so at the moment the decision is made the next tool result does not exist and cannot be measured. The only options are to reserve space for it in advance or to discover afterwards that there was none. Reserving is what TOOL_HEADROOM does, and sizing it from the 95th percentile of the harness's own history is what stops the reserve being either a guess or a round number: it means the threshold is right for nineteen results in twenty and knowingly wrong for the twentieth, which is a statement somebody can agree to. A round number is the same trade made without knowing which trade it was.

Measuring used is less straightforward than it looks, and the three available sources trade accuracy against latency. A local character-count heuristic — roughly one token per 3.6 characters of English, closer to one per 3.0 for code and JSON — is free and runs in microseconds, and on a mixed transcript it is accurate to about ±12%, which on a 175,000-token ceiling is a ±21,000-token error and far too loose to be the only mechanism. The API's token counting endpoint returns an exact count for a given request body, costs one round trip of roughly 200 milliseconds, and is the right thing to call when the estimate lands within 10% of the ceiling. The third source is free and often forgotten: every response reports usage.input_tokens for the request that produced it, so after each turn you know the exact size of the context as of that turn and can carry it forward, adding only the estimated size of what has been appended since.

The three sources are arranged the way they are because their errors and their costs point in opposite directions, and the arithmetic on the first one shows why no single source will do. An estimate accurate to about a tenth is fine when the number is small and useless when it is large, because the error scales with the number: a twelve per cent error on a ceiling of 175,000 is an uncertainty of 21,000 tokens, which is larger than the entire tool headroom being reserved. The exact count is accurate but costs a round trip of roughly 200 milliseconds on every call that uses it, which is why it is reserved for the region near the threshold where accuracy actually decides something. And the reported usage.input_tokens is exact and free but always one turn stale, since it describes the request that has already been sent. Using the exact stale figure as the base and the cheap estimate only for what has arrived since keeps the estimate applied to a small quantity, which is where its proportional error is affordable.

Several things count toward the window that a naive estimate over message text will miss. Tool definitions are part of the request and are charged on every call — a suite of twelve tools with detailed schemas runs 3,000 to 6,000 tokens and is constant overhead you can neither compact nor forget. Images and PDF blocks contribute according to their dimensions and page count rather than their text. Prompt caching, which lets the provider reuse its work on a prefix of the request that has not changed since the last call, reduces cost and latency but not occupancy: a cached 90,000-token prefix still occupies 90,000 tokens of the window, and treating cached content as free is a reliable way to build a system that is fast right up until it is rejected.

The reason the cache cannot help with occupancy is that the two limits are enforced by different things. The context limit is a property of the model: it is the number of token positions the model can attend over in one forward pass, and a cached prefix still occupies its positions, because reusing the state computed for those positions is not the same as the positions being absent. The cache is a property of the serving layer, which stores work already done so it does not have to be done again. So a hit changes what the provider recomputes and what the invoice says; it changes nothing about how many positions the request uses. The belief worth checking here is the one a fast, cheap system encourages: that because the prefix is barely costing anything, it is barely there. It is entirely there, and the rejection when it arrives will not mention the cache.

The clamp belongs at the tool boundary#

A budget trigger makes the failure visible and survivable; it does not make a 45,000-token grep result a good thing to have. No policy over the transcript can repair a tool that is allowed to inject an arbitrary fraction of the window in one call, so the clamp — a hard ceiling on the number of tokens any single tool result may contribute — belongs where the tool's own semantics are available to do something smarter than truncation.

MAX_RESULT_TOKENS = 4_000

def search_code(pattern: str, path: str = ".") -> str:
    hits = ripgrep(pattern, path)
    if len(hits) > 400:
        return (f"Pattern {pattern!r} matched {len(hits)} lines across "
                f"{len({h.file for h in hits})} files — too broad to return. "
                f"Top directories: {top_dirs(hits, 5)}. "
                f"Narrow by path or use a more specific pattern.")
    body, dropped = fit(hits, MAX_RESULT_TOKENS)
    if dropped:
        ref = write_artifact(hits)     # full result to disk, agent can read it
        body += f"\n[{dropped} further matches omitted — full list at {ref}]"
    return body

The broad-pattern branch is the one that changes agent behaviour rather than just protecting the harness. Returning 400 truncated lines teaches the agent nothing and it will often proceed as if it has seen the whole picture; returning a count, the directories where the matches cluster, and an instruction to narrow, gets a better second query and usually a better answer.

The difference between those two returns is what each one lets the model infer. A truncated list is indistinguishable from a complete list, because a list of matches looks the same whether it is all of them or the first few hundred of them; nothing in the value marks it as partial, so the model reasons as though it has the full set. The count-and-directories reply carries the one fact the truncation destroys — that the result was too broad — and it carries it in a form the model can act on, because a directory breakdown is exactly the information needed to write a narrower pattern. The general shape is that a clamp has to decide what to tell the model about what it removed, and returning less of the data while returning more about the shape of the data is almost always the better trade. The overflow branch keeps the full result addressable by writing it somewhere the agent can read on demand, which is the eviction-to-a-reference pattern from the previous chapter on state documents, applied before the payload ever enters the window rather than after.

A trigger decides when, not what#

It does not reduce context; it decides when reduction happens. Everything about which items survive, what gets pinned and how the summary is generated is unchanged by moving from a counter to a budget, and a system with a correct trigger and a summariser that drops constraints will fail in exactly the way the pinned-ledger chapter described, only on schedule.

It also cannot save you from a single tool result larger than the window. If a tool can return 250,000 tokens, there is no threshold that helps, because the result cannot be placed in the context at all — the clamp is not an optimisation in that case but the only thing standing between the tool and a permanently unusable session.

One more path is worth building even after both fixes are in, because a threshold sized from measured percentiles will eventually meet a result outside them. When a request is rejected for length, the recovery is to drop the newest oversized block rather than to fail the session: replace the offending tool_result with a short stand-in recording the tool, its arguments, its size and where the full output was written, then retry. The agent sees that its search returned too much to read and can narrow it, which is a recoverable state, whereas an exception forty minutes into a task is not. With the durable log from the previous chapter in place, even a session that does die can be resumed from its original turns.

The reserve itself has a cost worth acknowledging. Holding back 8,192 tokens for the completion and 12,000 for tool headroom on a 200,000-token window means about 10% of the context is deliberately unused, and compaction fires roughly a third more often than it strictly needs to. That is the price of never being rejected, it is usually the right trade, and it should be a number someone chose rather than an accident — tuning TOOL_HEADROOM down after measuring that your tool results are reliably under 3,000 tokens gives back real capacity.

There are now four decisions in the pipeline — what to pin, what the summary records, whether it is derived from originals, and when it fires — and each can be changed independently. Telling whether a change to any of them made the agent better is a measurement problem of its own, because the failures they cause show up in one conversation out of forty and cannot be seen in an average.