Price a proposed section before agreeing to carry it

A staff engineer opens a pull request adding a section on error-handling conventions to the shared config file: wrap external calls in a typed Result, never swallow an error to satisfy a linter, attach the correlation ID to every error surfaced to a caller — the correlation ID being the identifier that ties every log line, error and downstream call belonging to one request together, so that a failure can be traced across services — and four more of the same shape. It is well written, it reflects a real house style, and in the review someone asks what it costs. The answer that comes back is "about five hundred tokens, which is nothing", and that ends the discussion, because nobody in the thread has a better number and the assertion sounds plausible.

It is a bad answer for two reasons. The token count is a guess — the section is 640 tokens on the tokenizer this team's model actually uses — and tokens are not the cost that matters. What matters is what the section changes and what it displaces, and both of those are measurable in an afternoon by someone who is willing to run the tasks twice.

Count the tokens before arguing about them#

Estimating token counts by eye, or by a rule of thumb like four characters per token, is wrong by enough to change conclusions, and reaching for OpenAI's tiktoken is worse — it is a different tokenizer and it undercounts Claude tokens by roughly 15–20% on prose and considerably more on code. The API has an endpoint for this, and counts are model-specific, so pass the model you actually run.

from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-5"

def count(text: str) -> int:
    return client.messages.count_tokens(
        model=MODEL,
        messages=[{"role": "user", "content": text}],
    ).input_tokens

whole = open("CLAUDE.md").read()
section = open("proposed-error-handling.md").read()

print(f"file today:      {count(whole):>6,}")
print(f"proposed:        {count(section):>6,}")
print(f"file after:      {count(whole + section):>6,}")

file today:       2,810
proposed:           640
file after:       3,450

That is the input to every other calculation in this chapter, and it takes two minutes. Running the same count against git show HEAD~20:CLAUDE.md gives the growth rate, which for this team was 2,810 tokens today against 1,190 five months ago — a little over 300 tokens a month, every one of them added by someone who said it was nothing.

Three measurements, and only one of them is about tokens#

The section is worth carrying if it changes what the agent does. Establishing that means running a representative set of tasks with and without it, on the same commit, and comparing the outputs. Representative means drawn from the same session logs used for the reach estimate in chapter 1, biased toward the tasks the section claims to affect — for an error-handling section, tasks that add an external call, a new endpoint, or a retry path.

This team ran twelve tasks, three runs each per arm — one arm with the section in the file and one without, the two arms being identical in every other respect — for seventy-two sessions in total. Model outputs vary run to run, so a single pair proves nothing; three runs is enough to tell a consistent difference from a coin flip, and not enough to detect a rare one.

The result is the one this exercise usually produces, and it is worth sitting with: the section changed something a reviewer could name on two tasks out of twelve, and both of those were already failing CI for the same reason. What the section buys, on this evidence, is that the agent gets the correlation ID right before CI tells it to, saving one iteration on roughly one task in six. That is not nothing. It is also not the thing anybody claimed in the review, where the argument was that the agent "gets error handling wrong without it".

Two details of the protocol are what make that table trustworthy rather than decorative. The comparison is done blind: the seventy-two diffs are stripped of their arm label and shuffled before anyone reads them, because a reviewer who knows which arm produced a diff will find improvements in the treated one. And the criterion for "improvement" is written down before the runs, not after — for this section it was the reviewer can name which line of the proposed section the difference corresponds to. Without a pre-registered criterion the exercise degenerates into finding a story for whatever came back, and every section passes, because model output always differs somewhere and any difference can be narrated as an effect. The seven-of-twelve row exists precisely to keep that honest: differences are common and mostly mean nothing, which is why the row that matters is the one below it.

The two numbers that surprise people are the p95 total and the cold-cache latency. The p95 total shows why the token-budget framing of this argument has weakened: on a 1M-token context window, 640 tokens of standing instruction is not competing with task evidence for space in any literal sense, and a debugging session that accumulates 96,000 tokens of stack traces and file contents has not been meaningfully crowded. The cost is real but it is not scarcity of window. The latency line is where the real per-session cost shows up, and understanding it requires knowing something about how the prefix is billed.

Whether the prefix is re-billed or cached decides the arithmetic#

An always-loaded file is rendered into the prompt prefix, ahead of the conversation. On a twenty-turn session, the naive arithmetic says 640 tokens × 20 turns = 12,800 input tokens, and at $5 per million that is 6.4 cents — small, but it is the number people quote. The real number depends on whether that prefix is served from cache, and the difference is about an order of magnitude.

Being served from cache means the provider has kept the processed form of an unchanged prefix for a few minutes and re-reads that instead of processing the same tokens again on the next turn, which is what prompt caching does and what its pricing reflects. Cache reads bill at roughly 0.1× the base input rate and cache writes at 1.25× for the five-minute TTL. A cached prefix on a twenty-turn session is one write and nineteen reads: 640 × 1.25 + 640 × 19 × 0.1 = 800 + 1,216 = 2,016 token-equivalents, against 12,800 uncached. The way to find out which regime you are in is to read the usage object rather than reason about it.

resp = client.messages.create(
    model=MODEL, max_tokens=1024,
    system=[{"type": "text", "text": config_file,
             "cache_control": {"type": "ephemeral"}}],
    messages=conversation,
)
u = resp.usage
print(u.input_tokens, u.cache_creation_input_tokens, u.cache_read_input_tokens)

turn 1:   412   2810      0
turn 2:   980      0   2810
turn 3:  1544      0   2810

Turn one writes the prefix; turns two and three read it. If cache_read_input_tokens stays at zero across turns with an unchanged prefix, something above the config in the prefix is varying — a rendered timestamp, a session ID, a tool list assembled in nondeterministic order — and the whole file is being re-billed at full price on every turn. That is a real and common failure, and it is worth checking before any argument about section length, because it changes the cost of the entire file by 6×.

There is a second failure that is silent in a nastier way. Caching has a minimum prefix length, and it is model-dependent: 512 tokens on Claude Opus 5, 1,024 on Opus 4.8 and Sonnet 5, and 4,096 on Opus 4.6 and Haiku 4.5. Below the minimum, nothing caches and no error is raised — cache_creation_input_tokens simply stays at zero. A 2,810-token file caches on one model and silently does not on another, which means "what does this section cost" can have different answers on two models in the same organisation. The minimum is not monotonic across generations, so it is not safe to assume that a newer model is more forgiving.

The latency line in the table follows from the same mechanism. Cache-warm, the section adds 0.2 seconds to time-to-first-token, which nobody notices. Cold — first turn of a session, or after the five-minute TTL lapses on a session where the human went to lunch — it adds 1.2 seconds, because the tokens are actually being processed. Editing the config file mid-session, which some harnesses do when they detect a change on disk, invalidates the prefix from that point and pays the cold cost again on the next turn.

What the measurement does not settle#

Running the experiment produces a number, and a number is not a decision. Three limits are worth stating before anyone treats seventy-two sessions as a verdict.

Seventy-two sessions cannot detect a rare effect. If the section prevents a serious error on one task in a hundred, this design will see it zero or one time and call it noise. That is the honest reading of a null result: the section does not have a large, common effect, and the experiment says nothing about small or rare ones. For entries whose value is a rare catastrophic save — the billing invariant from chapter 1 is one — the A/B is the wrong instrument entirely, and reach plus consequence is the better argument.

The experiment costs something. Seventy-two sessions on this team's tasks ran to about $40 of API spend and three hours of somebody's day, most of it spent reading diffs rather than waiting. That is affordable for a contested 640-token section and absurd for a two-line one. The rule that follows is proportionate: measure the sections that people are arguing about, and let short uncontested entries in on judgement.

And measurement does not settle a disagreement about values. Two staff engineers on this team disagreed about a different section — 60 lines of API versioning conventions. One held that it belongs in the config file, because API work happens constantly and the agent gets versioning wrong without it. The other held that it is a workflow and belongs in a skill, pointing out that most sessions never touch an API. The evidence that would settle it is two-sided and cheap to specify: the reach measurement from chapter 1 on the last 200 sessions, which the first engineer predicts comes back high and the second predicts comes back low; and the firing measurement from chapter 3 on API-touching requests, which the second predicts comes back near 100% and the first predicts comes back with a visible gap. Designing that pair of measurements is the productive half of the argument, and it can be done in the review thread in five minutes.

What neither engineer has planned for is the evidence coming back in the middle — a reach that is neither high nor negligible, and a firing rate high enough to be defensible and low enough to be worrying. That case is where teams reach for a compromise that looks like the reasonable synthesis of both positions and is in fact a third mechanism with no owner. Before that argument can be had properly, though, there is a more immediate one: a file that has grown 300 tokens a month for five months, and a mandate to halve it by Friday.