Tool block cost and caching

The Messages API holds no state between requests. Every turn re-sends the entire conversation — system prompt, full transcript, every tool definition — and the model re-reads all of it to produce the next block of output. That single fact is what turns a tool block from a fixed cost into a multiplied one, and it is the fact most people get wrong the first time they are asked to price it.

Take a support agent exposing 40 tools whose serialised definitions measure about 6,000 tokens, running conversations that average 12 assistant turns. The tool descriptions do not cost 6,000 tokens per conversation. They cost 72,000. The 61 tokens that the send_invoice_reminder rewrite added in chapter 4 are not 61 tokens either; on this agent they are 732 tokens per conversation. Everything below follows from multiplying by turns.

Both of those are one multiplication, and the multiplier is the number of assistant turns rather than the number of conversations or the number of users. The reason is the statelessness above: a second turn is a second complete request, and a complete request contains the tools. Twelve turns means the block is processed twelve times, so six thousand becomes twelve times six thousand and sixty-one becomes twelve times sixty-one. Nothing about the block changed between the first turn and the twelfth. What changed is that it was sent again, and it will be sent again on the thirteenth.

Measuring the block rather than guessing at it#

Before any of the arithmetic is worth doing, the input has to be real. Character counts and rules of thumb are both bad here: JSON punctuation tokenises inefficiently, and the provider adds framing around each definition that never appears in your source. The way to get the number is to ask the model's own tokeniser, by counting a trivial request twice and subtracting.

from anthropic import Anthropic

from app.tools import TOOL_DEFINITIONS

client = Anthropic()
MODEL = "claude-opus-5"
PROBE = [{"role": "user", "content": "."}]


def block_tokens(tools: list[dict]) -> int:
    """Tokens the rendered tool block adds to a request."""
    with_tools = client.messages.count_tokens(
        model=MODEL, tools=tools, messages=PROBE
    ).input_tokens
    baseline = client.messages.count_tokens(model=MODEL, messages=PROBE).input_tokens
    return with_tools - baseline


total = block_tokens(TOOL_DEFINITIONS)
print(f"whole block: {total:,} tokens across {len(TOOL_DEFINITIONS)} tools")

by_cost = sorted(TOOL_DEFINITIONS, key=lambda t: -block_tokens([t]))
for tool in by_cost[:5]:
    print(f"  {tool['name']:<28} {block_tokens([tool]):>5}")

Do not reach for tiktoken or a characters-divided-by-four estimate for this. Those are calibrated on a different tokeniser and run 15–20% low on English prose and much worse on JSON and code, which is exactly what a tool block is made of. On the support agent above, the measured block came to 6,140 tokens where the character estimate had said 4,900, and the per-tool breakdown showed something the aggregate had hidden: two tools with long enum lists and worked examples accounted for 1,780 of those tokens between them, more than the eleven smallest tools combined.

The arithmetic on a real workload#

The interesting version of the question is not one conversation but a month of them. Take an agent with 60 tools whose serialised definitions come to 4,500 tokens, running a session that averages 14 model calls, at roughly 300,000 sessions a month. Per session the tool block costs 4,500 × 14 = 63,000 input tokens. Across the month that is 18.9 billion input tokens, which at Claude Sonnet 5's list rate of $3 per million input tokens — written $3/MTok in the table below — comes to about $56,700 a month for text that never changes.

Each of those three steps changes the unit, and keeping track of the unit is most of the discipline. Four and a half thousand tokens per request becomes sixty-three thousand tokens per session when multiplied by the calls in a session. Sixty-three thousand tokens per session becomes a bit under nineteen billion tokens per month when multiplied by the sessions in a month. Then tokens per month becomes dollars per month when multiplied by a price whose own unit is dollars per million tokens — which is the step where a factor of a million goes missing if the rate is read as dollars per token. Nineteen billion divided by a million is nineteen thousand, and three dollars apiece is the fifty-six thousand seven hundred.

That figure is what you pay with caching switched off, and nobody sensible runs an agent that way. Prompt caching keeps the already-processed leading tokens of a request on the provider's side, so when the next request begins with the same tokens they are billed at a fraction of the input rate rather than processed again. It works by prefix match: the provider renders tools first, then system, then messages, so the tool block sits at the very front of the prompt and is the most naturally cacheable content in the entire request. A cache_control breakpoint — a marker in the request saying how far the cacheable prefix extends — placed on the last system block covers the tools and the system prompt together. Cache reads bill at roughly 0.1× the base input rate; cache writes bill at 1.25× with the default five-minute time-to-live (TTL), the span an unused entry survives before it is discarded, or 2× if you take the one-hour TTL. Break-even on the five-minute version is two requests, so for a fourteen-call session the economics are not close.

That break-even is one inequality and it is worth doing once, because the same inequality settles the one-hour question later in this chapter. Without caching, n requests cost n times the block. With caching they cost one write at the write premium plus n minus one reads at a tenth. Setting the second below the first and solving for n puts the crossover a little above one request, so two already pay for the write and everything after the second is close to free. Substitute the one-hour write premium, which is higher, and the crossover moves a little above two — meaning three requests. Both crossovers are small against a fourteen-call session, which is why the interesting question is never whether to cache but which time-to-live to cache with.

The second row is the number a platform lead quotes when they say caching makes this a non-issue: 4,500 tokens written once at 1.25× is 5,625 token-equivalents, and thirteen reads at 0.1× is 5,850 more, so the whole session's tool overhead prices at 11,475 tokens instead of 63,000. An 82% reduction is a real win and worth having. It is also not zero, and $10,328 a month is not a rounding error on most budgets.

The eighty-two percent is that pair of figures divided: eleven and a half thousand against sixty-three thousand is a bit under a fifth, so a bit over four fifths of the bill is gone. What survives is not evenly distributed across the session either. Almost half of it is the single write, charged at a premium over the ordinary input rate rather than at it, and the remainder is thirteen reads whose individual cost is small enough to feel free and whose total comes out slightly above the write. That distribution is worth carrying into the next paragraph, because it is exactly what makes the think-time case expensive: the term that multiplies is the one already carrying the premium.

The third row is where the "non-issue" claim comes apart. A cache entry with the default TTL survives five minutes from its last use. In a support conversation the user goes away to find an order number, reads a policy page, gets pulled into a meeting; a session that runs fourteen calls over forty minutes of wall clock will cross the TTL boundary several times, and each crossing means the next request writes the block again at 1.25× rather than reading it at 0.1×. Model that as five writes and nine reads and the monthly bill nearly triples against the best case. The awkward part is that the dashboard does not show this as a problem, because hit rate is averaged over requests and the majority of requests in any high-volume workload come from fast back-to-back turns. A 91% aggregate hit rate is entirely compatible with the slowest quarter of your sessions paying full price on half their calls.

Price that row the way the second one was priced. Five writes at the premium is a bit over twenty-eight thousand token-equivalents; nine reads at a tenth is a bit over four thousand; together they are the thirty-two thousand in the table, which is nearly three times the best case and about half the uncached figure. Two things come off that comparison. Caching is still winning, because halving the uncached bill is a real saving that nobody should give up. And the best case stands further from the realistic one, in proportional terms, than the realistic one stands from no caching at all — so a team that enabled caching, never plotted its session durations, and reported the second row has claimed a saving about two thirds larger than the one it is getting.

The five-minute TTL is not the only option, and the one-hour variant is the obvious response to think time. It costs 2× on the write instead of 1.25×, which moves break-even from two requests to three, and for the think-time session above it converts four expensive re-writes into reads at the price of a more expensive first write. On the numbers in the table that is roughly $13,000 a month against $28,958, so it is worth reaching for whenever your sessions have gaps. What it does not do is remove the failure mode, because a session that idles for ninety minutes crosses the longer boundary too, and the longer TTL is strictly worse for the short bursty sessions that make up the bulk of the traffic. Choosing between them is a decision about your session-duration distribution rather than a default, and the distribution is something most teams have never plotted.

Where the thirteen thousand comes from is that same session priced once more. One write at the one-hour premium is nine thousand token-equivalents; thirteen reads at a tenth are the same five thousand eight hundred and fifty as before; so the session lands a bit under fifteen thousand token-equivalents against thirty-two thousand on the short time-to-live. The larger premium is paid once, and the four re-writes it avoids were each being paid in full, which is the whole of the trade. It is also why the answer flips for short sessions: a two-call session on the one-hour setting pays the larger premium and then ends before it has recovered the difference.

There is a second way the cache goes cold that has nothing to do with users. Because the tool block renders at position zero, any change to any tool definition changes the prefix and invalidates every downstream cache entry. Deploying a one-word edit to one description at 14:05 drops the cache hit rate to zero for every session in flight, and the recovery is a fresh write per session. This is not an argument against editing descriptions — chapter 3's rewrite paid for itself many times over — but it does mean description changes have a cost profile closer to a schema migration than to a copy edit, and batching them into a single deploy is worth more than it looks.

Verifying that any of this is happening takes one field in the response. If cache_read_input_tokens is not growing across repeated requests with the same tool set, something is invalidating the prefix and the caching is decorative:

{
  "usage": {
    "input_tokens": 812,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 9340,
    "output_tokens": 143
  }
}

Note what input_tokens means here. It is the uncached remainder only, so an agent that has been running for an hour showing 812 input tokens has not somehow shrunk its prompt; the other 9,340 tokens were served from cache. The total prompt size is the sum of all three input fields, and a monitoring dashboard that graphs input_tokens alone will show a flat, comfortable line while the real prompt grows without bound.

The resource that cannot be discounted#

Money is the easier of the two bills. The tool block also occupies context window, and the discount does not apply there — a cached token takes exactly as much room in the window as an uncached one, on every single call. On a 1M-token context window, 4,500 tokens is 0.45% and sounds like nothing, which is true as far as raw space goes and misleading about why it matters.

What makes the tool block different from the rest of the prompt is that it is the part you cannot evict. Compaction summarises earlier turns; context editing clears stale tool results and thinking blocks. Both operate on the transcript, and neither touches tools, because the model needs every definition available on every turn in case it picks that one. The tool block is therefore the floor of your context usage — the number the window can never fall below no matter how aggressively you prune — and in a long agentic session, where the transcript has been compacted twice and the useful working state is what survived, the tool block is a larger share of what the model is actually attending to than the raw percentage suggests.

Between the two resources, the one that constrains decisions is the window rather than the money, and the ordering surprises people who came into the question through the finance conversation. Ten thousand dollars a month is a line item a platform team can absorb, defend, or halve with an afternoon of caching configuration, and it scales linearly with traffic in a way that finance already understands. The window is fixed per request, cannot be bought, and is the resource that decides whether the sixty-first tool ships. When a team says it cannot add the tool that product asked for, the binding constraint is never the marginal dollar.

The sharper version of the context cost is not space at all. Selection, as chapter 3 established, is a comparison across every candidate description, and the difficulty of that comparison scales with the number of candidates rather than with their length. Sixty descriptions in the prompt is sixty things to rank on every turn, and the model does not get better at that as the list grows.

Tool count and description length#

It is not an argument for cutting tools. How many tools an agent should expose, and where the boundaries between them fall, is the granularity question from topic 5.10, on tool granularity and surface; the token bill is one input to that decision and not a decisive one, since a tool that saves a round trip usually pays for its own definition several times over. Deleting a tool to reclaim 150 tokens per request while forcing the model into two extra turns is a loss on both resources at once.

It is also not an argument that descriptions should be as short as possible. The fourth row of the table is deliberately there: halving the block halves the caching-on bill too, and $5,164 against $10,328 is a real difference at this volume. What the row does not say is whether the halved block still routes correctly, and that is the only question that matters, because a description that saves 2,000 tokens a month and misroutes 3% of traffic has not saved anything.

Halving the block halves every term in the caching calculation, which is why that fourth row is exactly half the second rather than approximately half. The write is proportional to the block, each read is proportional to the block, and the number of writes and of reads is unchanged, so a linear quantity multiplied by fixed counts stays linear. That is worth knowing before any argument about description length, because it means the saving available from a proposed cut can be read off the size of the cut alone, with none of the caching arithmetic done again — which leaves the whole of the argument where it belongs, on what the cut does to routing.

Multiplication by turns is the whole of it. A tool block is not a fixed setup cost but a per-request one, caching turns the bill into roughly a fifth of itself without touching the window it occupies, and the window is the resource that decides what ships. Both figures are worth having before anybody argues about a description's length, because they set the price of the argument.

The arithmetic tells you what compression is worth. It cannot tell you what compression costs, and the next chapter is about measuring that side of the trade properly — including why the obvious measurement, an aggregate accuracy score on a labelled eval set, is the one most likely to tell you that a bad cut was fine.