Giving up the prefix

Every worked case in this topic so far has ended with the prefix intact. That is not because the prefix always wins; it is because the cheaper repairs usually land first, and a chapter that stops there leaves the reader without the branch where they do not. A rule with a floor at three nines, a customer-visible failure behind it and an owner willing to defend it is not tradeable against a hit rate at any exchange rate the invoice can express.

Giving up the prefix therefore has to be workable as an explicit decision: what evidence justifies it, how to bound the spend that follows, and how to leave a record so the next reader finds a choice rather than a drift. The figures below come from an account agent at 120,000 turns a day, $3 per million input tokens, whose confirmation rule reached 99.8% against a floor of 99.9% after the 58-token tail restatement of chapter 5.

What has to be true before you give up the prefix#

The restatement moved that rule from 99.2% to 99.8% and then stopped, and the reason it stopped is visible in what the rule actually says. Confirmation in this product is a procedure rather than an imperative: it must name the specific object and its irreversibility, it is waived for internal admin sessions, it requires a second confirmation above $5,000, and it does not apply to change_plan inside the 24-hour reversal window. The 58-token version compressed all of that into one line, which recovered most of the compliance and introduced a new failure the eval also caught — 3.1% of read-only sessions picked up a spurious confirmation prompt, because the terse form had lost the qualifier that limits the rule to state changes. Adding the qualifiers back grew the tail block to 340 tokens and moved the rule to 99.84%, still short, with the escalation thresholds now duplicated in two places.

That is the point at which abandoning the prefix stops being a failure of nerve and starts being the remaining option. Four things had to be on the table before the team took it, and the order matters, because each one is cheaper to establish than the one after it.

The eval had to show, on paired cases at production temperature, that the layout change buys the compliance. Moving the policy behind the documents took the confirmation rule to 99.94% and left citation and price compliance unchanged within noise, which is the evidence that the position is the binding constraint rather than the wording. A layout change proposed without that measurement is the same mistake as the reordering in chapter 4, run in the opposite direction.

The rule had to be one that does not compress, which is what distinguishes it from the two rules the restatement fixed. The deterministic control had to be in place already and had to be insufficient rather than absent, because a rule with no control behind it should get a control before it gets a layout. And the floor had to be owned — a number with a name against it and a consequence attached — since a floor that nobody set is a floor that will be renegotiated the moment the invoice arrives.

Four layouts, one sheet#

The same 400 cases, the same pinned model, scored on the same afternoon, with input spend computed at 120,000 requests a day and $3 per million input tokens.

The last row is the one that turns a deliberate decision into an accident, and it is worth working out rather than accepting, because the surcharge it names sounds implausible. Moving the policy and the schemas behind the documents leaves nothing invariant in front, so the prefix is empty and every request writes a fresh entry covering the whole 14,600-token prompt. A write costs 1.25× the base rate, so the prompt is billed at a quarter more than it would cost if nobody had ever asked for caching: 14,600 tokens at 1.25× against 14,600 tokens at 1.0×, on 120,000 requests a day and $3 per million, is $6,328 against $5,256. Dividing the first by the second is the 20% surcharge, and every cent of it is buying entries that a 4% hit rate says are almost never read.

Giving up the prefix and giving up the cache are therefore separate acts. The first is a decision about where the policy sits; the second is a decision about whether a marker stays in the request. Doing the first without the second leaves the mechanism running with nothing to work on, which is not neutral because a miss still writes. Deleting the cache_control marker is the first bounding move and it takes one line.

The second bounding move is to keep whatever is genuinely immutable and does not need a late position, and the phrasing of that test is doing real work. Two conditions have to hold, not one. The nine tool schemas are 3,100 tokens of JSON that never change, which satisfies the first. They are also material the model consults when it is deciding what to call rather than a rule it is meant to follow, and no placement probe on this prompt has ever shown a schema being obeyed better at the end, which satisfies the second. So they stay in front and remain a real cacheable segment. Compare the second row with the fourth to price that: $6,328 against $4,409 is $1,919 a day, for a decision that changes nothing about behaviour and moves 3,100 tokens back across the marker.

Bounding the spend you have just agreed to#

Once the prefix is gone or shrunken, the input bill is a function of prompt length and request count, so those are the two levers left. Length is the one topic 4.13's budget discipline already gives you tools for, and the return on it changes character here. With the schemas in front and everything else behind them, 11,500 of the 14,600 tokens are now billed fresh on every call, so a token removed from the policy is a token removed at the full rate. Trimming the policy from 8,900 to 5,400 tokens takes out three and a half thousand of them, and three and a half thousand tokens on 120,000 requests a day at $3 per million is $1,260 a day.

Do the same trim under the cached layout and those tokens were being billed at roughly a tenth, so removing them is worth roughly a tenth as much — the chapter's own figure is about $110. The trim is the same trim. What changed is the rate the removed tokens were charged at, which is the whole point: content that was cheap to carry becomes expensive the moment it stops being reused, and a pruning pass that was correctly judged not worth the effort under one layout can be the largest single item on the sheet under the next one.

Request count is the more interesting lever, because the layout does not have to apply to all of it. Only sessions authenticated to the billing scope can reach a state-changing tool at all, and those are 14% of traffic.

def build_request(session, user_msg, docs):
    """Two layouts, one prompt source. The router picks by capability, not by guess."""
    can_mutate = bool(session.tools & STATE_CHANGING)

    if not can_mutate:
        # cached layout: 12k immutable prefix, bounded tail, 58-token restatement
        return assemble(
            prefix=[POLICY, SCHEMAS_READONLY], breakpoint_after="prefix",
            tail=[docs, user_msg, TAIL_RESTATEMENT],
        )

    # policy-last layout: schemas stay in front as a real 3.1k prefix
    return assemble(
        prefix=[SCHEMAS_FULL], breakpoint_after="prefix",
        tail=[docs, user_msg, POLICY_PROCEDURAL],
    )

Both of the numbers that make routing worth doing are differences against the same baseline, which is the $2,000 a day the cached layout costs everywhere. Applying the expensive layout to all traffic costs $4,409, so the floor would cost $2,409 a day. Applying it only to the 14% that can reach a mutating tool costs $2,337, so the floor costs $337 a day. Same compliance figure of 99.94% in both cases, because the sessions that cannot call a mutating tool cannot fail the rule, and a layout change made for their benefit was always buying a guarantee they did not need. Routing is cheap here for that reason rather than as a general property: it works exactly when the rule that forced the expensive layout applies to a knowable minority of requests.

It also splits the eval into two arms that both have to be scored, and it creates a class of bug where a session gains a tool mid-conversation and switches layout between turns, which the team handled by pinning the layout at session start. Neither of those is free, and both are cheaper than the difference between $2,409 and $337.

Money is not the only bill that arrives with the cache miss, and the second one follows from what a cache read is. The stored entry holds the model's computed state for those tokens, so a request that reads it does not have to run prefill over them — and prefill is the pass that has to finish before a first token comes back. A 12,000-token prefix read from cache therefore skips 12,000 tokens of prefill, worth about 310 ms of time-to-first-token on this deployment. The mutating sessions now pay that back on every turn, which took their p95 from 2,410 ms to 2,760 ms. Nobody on the team had thought to look, because the conversation had been framed as a cost decision from the first message in the thread. Latency is the degradation that hides inside "extra tokens" — three currencies were named in chapter 2 and this is a fourth that rides along with one of them — and on a surface with a latency SLO, a service level objective being the response-time number the team has committed to, it can be the binding constraint rather than the invoice.

One thing the routed layout does not buy is freedom from measuring the tail. The documents behind the policy are still ordered by retrieval score, and "Attention Sorting Combats Recency Bias in Long Context Language Models" shows that permuting retrieved documents alone — same documents, same instructions, same everything else — moves answer quality on its own. On this surface, sorting the six chunks by ascending relevance so the best one lands nearest the question was worth 1.4 points of citation compliance in a separate paired run. The ordering decision does not end at the block level.

The prompts where the argument never starts#

On the account agent, giving up the prefix took a measurement, a routing layer and a written record, because there was a large shared prefix with real money attached to it. There is a whole class of prompts where a colleague proposing "put the policy last" should get a yes in the first reply, and recognising them saves everyone a week.

A contract-review assistant at the same company is one. Each request carries the tenant's own master services agreement, between 4,000 and 30,000 tokens of it, ahead of a 2,200-token review policy. There are about 2,000 tenants, each sending one or two requests a week, so the material at the front of the prompt is unique to a request that will not repeat inside any TTL the provider offers. The prefix is already worthless; the only thing the front of that prompt is buying is a worse position for the review policy. Moving the policy behind the contract cost nothing measurable on the invoice and moved its two scored predicates by 2.1 and 0.9 points.

The general form of the case is a prompt where the invariant fraction is small or the reuse count is near one, and both are numbers you already have from the reuse measurement in chapter 2. When the invariant material is 800 tokens against 14,000 of per-request evidence, protecting it is worth around 5% of input spend, and no compliance argument has to be very strong to outweigh that. When mean reuse inside the TTL window is 1.1, there is no prefix to defend regardless of how long it is. Those two checks take an afternoon against request logs and they settle a category of design argument before it becomes a design argument.

The decision is per rule, and it has a date on it#

None of this is the general claim that compliance beats cost. On the same prompt, the citation rule and the price rule are both above their floors under the cached layout, so moving them would be paying $2,409 a day for nothing measurable, and a team that adopts "reliability wins" as a slogan will move all three because it is simpler than measuring one. The decision is per rule, and the rule that justified it here is the one with a floor at three nines and a customer-visible failure behind it.

It is also not a decision that survives its inputs. Every number in that record is a measurement with a date on it, and the two that move fastest are the model's placement behaviour and the mutating-session share. The revisit triggers exist because the cheapest version of this work in a year is re-running a 400-case paired eval, and the most expensive version is discovering from a support queue that the layout has been wrong for two quarters.

Accepting the cache miss, then, is a decision with four preconditions, three bounding moves and a record — not a concession. The preconditions are a paired eval showing position is the binding constraint, a rule that does not compress, a deterministic control already in place and insufficient, and an owned floor. The bounding moves are deleting the breakpoint that no longer earns anything, keeping the invariant material that does not need a late position, and routing the expensive layout to the traffic that needs it. The record is what makes it legible to whoever inherits it.

And it is not the whole answer even on this surface, because the routed layout still assumes that a 3,100-token prefix read 41 times inside a TTL window is a mechanism worth keeping. That assumption is worth checking rather than inheriting, and on a large fraction of the prompts an engineer touches it turns out to be false — the prefix is too short for the provider to cache at all, or the traffic is too sparse for an entry to survive between uses. The next chapter is about establishing that the mechanism engages before spending any effort optimising for it.