Each turn resends the transcript, and caching needs an identical prefix

The estimate that went into the pricing model was built from a single request, the way chapter 1 built one: fixed block, retrieved context, a short user message, a short reply. It came to $0.027 per call, and since the agent was expected to take about eight turns to finish a task, the model said roughly $0.22 a session and the pricing conversation moved on. The first month of real traffic came back at $0.31 a session, 41% over, and the discrepancy was not in the token prices, the traffic volume, or the fixed block. It was in the fact that a conversation is not eight requests. It is one request sent eight times, longer each time.

The transcript is the largest input the product sends#

Take the agent session that produced the $0.31. The first request carries a 4,600-token fixed block — system instruction, fourteen tool definitions, two worked examples — plus a 400-token opening message from the user, for 5,000 input tokens. Every subsequent turn appends what the model said, what the tools returned, and whatever the user typed next, which in this workload averages 2,000 tokens per turn, dominated by tool results rather than by anything a human wrote.

Total input across the session is 96,000 tokens for a conversation whose actual content is 19,000 tokens, so 80% of what you pay for is retransmission. Output adds 1,500 tokens across the session — most turns emit only a short tool call, and only the last one writes a real answer — which at $15.00 per million is $0.0225. The session lands at $0.3105.

Eight independent single turns, each carrying the 5,000-token opening request and no history, would cost $0.12 in input and $0.0225 in output, or $0.1425. The session costs 2.2× that, and the multiplier is entirely a consequence of the transcript being resent. Turn 8 on its own costs $0.057, which is 3.8× what turn 1 cost, for a turn that added exactly as much new information as every other turn did.

Doubling the turns more than doubles the bill#

The shape is worth writing down, because it decides which product changes are affordable. With a fixed block of B tokens and an average of I new tokens per turn, the input tokens sent across n turns are:

def session_input_tokens(base: int, per_turn: int, turns: int) -> int:
    # turn k sends base + per_turn * (k - 1)
    return turns * base + per_turn * turns * (turns - 1) // 2

for n in (4, 8, 16, 32):
    tokens = session_input_tokens(base=5_000, per_turn=2_000, turns=n)
    print(n, tokens, f"${tokens * 3.00 / 1_000_000:.3f}")

# 4   32000  $0.096
# 8   96000  $0.288
# 16  320000 $0.960
# 32  1152000 $3.456

The n(n−1)/2 term is the whole story: the linear part is the fixed block, the quadratic part is the transcript. At four turns the two terms are comparable; by thirty-two turns the transcript is 97% of the input and the fixed block has stopped mattering. This is why "let the agent iterate until it is confident" is a pricing decision as much as a quality decision, and why a change that raises average session length from eight turns to twelve — a 50% increase in turns — raises input cost by 92%, from $0.288 to $0.552.

It also reframes the tool results. A tool that returns a 6,000-token file dump instead of a 600-token extract does not cost you 5,400 tokens once. It costs you 5,400 tokens on the turn it arrives and on every turn afterwards, so in a session with four turns remaining it costs 27,000 tokens, or $0.081. Trimming tool output is worth roughly turns remaining times its apparent value, which is a much stronger argument than the one usually made for it.

An average session does not cost the average#

Using "about eight turns" as the planning figure hides something that follows directly from the quadratic. Session length here is not clustered at eight: a quarter of sessions end after two turns, a third run to five, and there is a thin tail that reaches the turn cap at 32. Because cost grows with the square of turn count, the mean cost of a session is larger than the cost of a mean-length session, and the gap widens as the spread does.

Mean input cost across that distribution is $0.289, and the mean session runs 6.7 turns and emits about 1,280 output tokens, so a session costs $0.308 on average — which is why the eight-turn reference session used above happens to sit close to the fleet mean and the median session, at five turns, costs less than half of it. The 4% of sessions running past twenty turns contribute 26% of input spend. Quoting the median as the typical cost understates the bill by a factor of two, and any per-session estimate has to be an average over the length distribution rather than the cost of a representative session.

A turn cap therefore has more leverage than its hit rate suggests. Lowering the cap from 32 to 20 truncates 4% of sessions and removes 6.5% of input spend, because the sessions it touches are the quadratically expensive ones. Whether that is acceptable depends on how many long sessions were converging on an answer and how many were looping over a tool that kept failing, which the eval set can answer and the cost model cannot.

The cache that returned nothing#

Prompt caching, whose mechanics topic 15.9 covered as a lever, is the direct remedy for retransmission: mark a prefix, pay a write premium once, and read it back at about a tenth of the input rate on subsequent requests. Applied to this session with the fixed block and the growing transcript both cached, the arithmetic changes substantially. Nineteen thousand tokens are ever written fresh, at 1.25× for the write, costing $0.0713. Turns 2 through 8 read a cached prefix of 5,000 through 17,000 tokens, 77,000 tokens in total, at 0.1× for $0.0231. Input falls from $0.288 to $0.0944, and the session falls from $0.31 to $0.117 — a 62% reduction in model spend for a change that alters no behaviour.

The team that ran this shipped caching and measured a 3% saving. The cause was in the two lines that assembled the system block:

# The version that never hit
system = [
    {
        "type": "text",
        "text": f"You are an operations assistant. Current time: {now.isoformat()}.\n"
                f"Operator: {user.email}. Workspace: {workspace.name}.\n" + STYLE_RULES,
        "cache_control": {"type": "ephemeral"},
    }
]
tools = [tool_schema(name) for name in enabled_tools]  # enabled_tools is a set

Two independent defects, either of which is sufficient. The timestamp changes on every request, so the first token block differs every time and no prefix can match. The tool list is built by iterating a Python set, so its order is stable within a process and differs across processes, which produces the especially confusing symptom of decent hit rates in local testing and near-zero hit rates behind a load balancer running eight replicas. A cache prefix matches on an exact token sequence from the beginning of the request, so anything variable above the cache breakpoint — the position in the request you marked as the end of the reusable prefix — invalidates everything below it, including the 4,600 tokens that never change.

The fix is to sort what is unordered and to push what varies below the cached region:

system = [
    {"type": "text", "text": STATIC_INSTRUCTIONS + STYLE_RULES,
     "cache_control": {"type": "ephemeral"}},
    {"type": "text", "text": f"Current time: {now.isoformat()}. Operator: {user.email}."},
]
tools = [tool_schema(name) for name in sorted(enabled_tools)]

Hit rate went from 4% to 91% on the next deploy. The residual 9% is genuine: cache entries expire after five minutes of inactivity by default, and a user who walks away mid-session and comes back pays the write again.

Put cache hit rate on the cost dashboard#

Token counts alone stopped being sufficient the moment caching entered the picture, because two workloads with identical token counts can differ threefold in cost. The provider reports the split on every response, and those three fields are what the dashboard needs:

{
  "usage": {
    "input_tokens": 2000,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 17000,
    "output_tokens": 190
  }
}

Define hit rate as cache_read / (cache_read + cache_creation + input_tokens) — the share of input tokens that were served from cache — and log it per request alongside the component counts from chapter 1. Aggregate it by deploy version, by tenant, and by route, because those are the three axes along which it silently breaks. A deploy that inserts a per-user line into the system prompt shows up as a hit-rate cliff on the version axis within minutes, where on a cost-per-request chart it would be invisible under daily traffic noise for a week.

Two secondary metrics earn their place next to it. The ratio of cache-write tokens to cache-read tokens tells you whether prefixes are being reused enough to pay back the 1.25× write premium; a workload writing more than it reads is losing money on caching, which happens with short sessions, aggressive breakpoints, or traffic spread thinly across many distinct prefixes. And average turns per session, tracked as a distribution rather than a mean, is the input to the quadratic term, so it predicts cost movements a week before they appear on the invoice.

What caching does not do#

It does not remove the quadratic growth, it discounts it. Reads are cheap rather than free, so a 300,000-token session still pays a tenth of full rate on the resent prefix, and at thirty-two turns that residual is $0.35 by itself. Long-running agents eventually need the context to actually get smaller — summarisation, dropping stale tool results, restarting with a compacted state — and no cache configuration substitutes for that.

The default five-minute lifetime interacts with how humans actually use an assistant, which is in bursts separated by meetings. On this product 23% of turns arrive more than five minutes after the previous one, and each of those pays a write on the entire accumulated prefix rather than a read — at turn 6 of a session that is 15,000 tokens at 1.25×, or $0.056, against $0.0045 if the entry had survived. Providers offer longer lifetimes at a higher write premium, and whether the longer option pays comes down to the share of turns falling in the gap between the two windows, which is a number you can read straight off the inter-turn delay histogram rather than guess.

It also does not survive a workload where prefixes are not shared. Caching pays when the same prefix is reused within its lifetime, which multi-turn sessions guarantee and which a batch of one-shot classifications across thousands of distinct documents does not, unless the documents follow a common instruction block worth caching on its own. Reaching for caching in that second workload gets you the write premium and none of the reads.

The cost figures in this chapter and in chapter 1 share a limitation that matters more than either: they price successful requests. The $0.31 session is the session that ran to completion. It excludes the session where the model looped and the runtime killed it at turn 20, the one the user abandoned after two turns, the one that was retried because a tool timed out, and the embedding and storage that made retrieval possible in the first place. Model spend per successful session is a measurement of the invoice, and the cost of serving a customer is a different and larger number.