Scenarios times turns times calls is the whole bill

The request to go from 40 scenarios to 160, so that five-point changes in end-to-end completion become resolvable, arrived at the same meeting as a question about why the evaluation line on the model bill had grown faster than the production line. Both are the same question, and answering either requires a model of where the money goes that is more specific than "evals cost money".

The bill has four parts and only one of them is obvious. Model spend is scenarios × turns × calls per turn, summed over three separate consumers — the agent under test, the simulated user that drives the other side of the conversation from a fixed script and goal, and the judge that scores each turn against a frozen rubric — each with its own token profile and its own price per million. Wall clock is a different currency and frequently the binding one, because a suite that takes half an hour cannot sit in a pull-request loop no matter how cheap it is. Maintenance is engineer-hours on scripts and cassettes that go stale. And there is the opportunity cost of the scenarios you did not write, which is real and unquantifiable and worth naming so nobody pretends the first three are the whole story.

The arithmetic for one run#

The whole of the model-spend half is one multiplication performed three times, once per consumer. The parameters below are the ones the travel-agent suite at Orbit runs with: a scenario is one scripted session driven end to end, a turn is one user message plus the agent's reply to it, and the token and price figures are what each consumer was measured at.

PRICES = {                       # USD per 1M tokens
    "mid":   {"in": 3.00, "out": 15.00, "cached_in": 0.30},
    "small": {"in": 0.80, "out":  4.00, "cached_in": 0.08},
}

def suite_cost(scenarios=40, turns=12,
               agent_calls_per_turn=2.4, agent_in=6_800, agent_out=320,
               sim_in=1_900, sim_out=60,
               judge_in=3_000, judge_out=150,
               agent_model="mid", sim_model="small", judge_model="mid"):
    t = scenarios * turns
    def cost(model, n_calls, tin, tout):
        p = PRICES[model]
        return n_calls * (tin * p["in"] + tout * p["out"]) / 1e6

    agent = cost(agent_model, t * agent_calls_per_turn, agent_in, agent_out)
    sim   = cost(sim_model,   t,                        sim_in,   sim_out)
    judge = cost(judge_model, t + scenarios,            judge_in, judge_out)
    return {"agent": agent, "simulator": sim, "judge": judge,
            "total": agent + sim + judge,
            "input_tokens": t * agent_calls_per_turn * agent_in
                            + t * sim_in + (t + scenarios) * judge_in}

suite_cost()
# {'agent': 29.04, 'simulator': 0.85, 'judge': 5.88,
#  'total': 35.77, 'input_tokens': 10_308_000}

Start where the function starts. t = scenarios * turns is the number of turns in the whole run, so 40 scenarios of 12 turns each is 480 turns, and all three consumers are priced off that one number. A turn, though, is a unit of conversation and not a unit of billing. Nothing is charged for a turn; things are charged for model calls, and the step from the first to the second is where suite estimates go wrong.

That step is agent_calls_per_turn, and the figure of 2.4 is the multiplier people forget when they estimate a suite from turn counts alone. A single agentic turn is not one request to the model: it is a planning call, then a tool call, and then usually a second model call to turn the tool result into a reply, which is how the average lands above two rather than at one. Multiplying gives the agent's call count, so 480 turns at 2.4 calls a turn is 1,152 model calls — exactly what t * agent_calls_per_turn computes on the line that prices the agent. An estimate built on one call per turn would understate the largest component of the bill by more than half.

The inner cost function is then the same arithmetic for all three consumers: the number of calls, times the input tokens at the input price plus the output tokens at the output price, divided by a million because the prices are quoted per million tokens. Work the agent through it. Each agent call reads 6,800 input tokens at $3 per million, which is a shade over two cents, and writes 320 output tokens at fifteen dollars per million, which is a little under half a cent more; call it something over two and a half cents a call. Spread across 1,152 calls, that is where the $29.04 agent line comes from.

The simulator is the same shape at a fraction of the price. Each of its calls reads 1,900 tokens at $0.80 per million and writes 60 at $4.00 per million, just under a fifth of a cent, and it makes exactly one call per turn — 480 of them — which is where the $0.85 simulator line comes from. The judge is the one consumer whose call count is not simply the turn count. t + scenarios is 480 per-turn scores plus one more call per scenario, five hundred and twenty in all, because the instrument reports a score for each turn and also a session-level verdict on whether the conversation as a whole met its goal. At three thousand input tokens and 150 output tokens on the mid model, a little over a cent a call, that is where the $5.88 judge line comes from.

Three numbers come out, and the ratio between them is the thing worth pulling out. The agent's $29.04 stands against $0.85 from the simulator and $5.88 from the judge, which is more than four times the other two put together, and against the $35.77 total it is 81 percent of the bill. That share rather than any of the absolute figures is what should direct effort: every optimisation aimed at the judge or the simulator is rearranging the small numbers, and anything that moves the agent's token profile moves the bill.

The last line, input_tokens, adds the same three call counts against their input-token figures and comes to 10.3 million for one run. The parameter inside it worth distrusting is agent_in, because 6,800 is a mean over a growing context rather than a constant. The first call of a session carries a system prompt and one user message; the last carries eleven turns of history behind it. That mean is therefore a property of the compaction policy rather than of the model, which is why arm B from chapter 6 — the arm run with summarisation switched off, so the context accumulates untrimmed — costs more than twice as much per session as the compacting arm: without compaction the mean context per call is around 16,000 tokens instead.

The rate limit under a run's wall clock#

Wall clock does not follow from cost in the way people expect, and the arithmetic that predicts it is a different arithmetic. A single 12-turn session takes about 102 seconds of serial time, counting the agent's calls, the simulator's replies and the judge's scoring one after another. Forty of those back to back is 68 minutes of serial work. Running them eight at a time ought to divide that by the concurrency, which gives eight and a half minutes, and that is the figure a capacity plan written in workers predicts.

The observed figure is 31 minutes, and the gap is not scheduling overhead. It is the provider's rate limit. The run pushes 10.3 million input tokens through an account allowed 400,000 tokens a minute, and dividing the first by the second gives a shade under a 26-minute floor that no amount of concurrency touches, because extra workers only queue against the same allowance. Eight and a half minutes was never on offer. The 31 minutes actually observed is that floor plus the ordinary slack of not saturating the limit in every minute of the run.

So a capacity plan for an evaluation suite is written in tokens per minute rather than in workers, and the practical consequence follows from that. Raising the concurrency divides a term that is not the binding one; raising the allowance moves the floor itself, which is why a separate, higher-limit key for evaluation traffic — so the suite is not drawing against the same bucket as production — buys more wall-clock improvement than any amount of parallelism in the runner.

What the suite costs per month#

The four rows are the entire cadence, and their monthly figures add up: $420 and $1,080 and $720 and $540 is the $2,760 in the last row. That is a closing identity rather than a tally, and what it licenses is the word complete — because nothing is running that is not in the table, the annual figure is the cost of the suite and not a sample of it, which is exactly the claim somebody will challenge in the meeting where it is defended. What it does not license is the word total. The identity closes over model spend only: wall clock appears in the table but not in the sum, the maintenance hours are a separate accounting it says nothing about, and the fourth cost has no row anywhere.

Thirty-three thousand dollars a year is a number that gets a suite cancelled, and it is also a number with a great deal of slack in it. The largest single lever is prompt caching — the provider keeping the already-processed form of an unchanging prefix and charging a fraction of the normal input price to re-read it — applied to the agent's system prompt and tool definitions, which are 2,400 identical tokens on every one of the 1,152 agent calls. That is close to three million tokens of prefix the provider would otherwise process from scratch every run. At $3 per million that block costs $8.30 per run, and at the cached read price of $0.30 per million, a tenth of the base rate, the same block costs $0.83. The difference is about seven and a half dollars, which against a $35.77 run takes 21 percent off the total.

What makes caching a lever rather than a discount is one condition: the prefix has to be byte-identical across calls and sit at the front. A tool list serialised in a different order between calls, a timestamp or a session id interpolated into the system prompt, or a per-scenario persona spliced in above the tool definitions all break the match. The failure is silent, since the calls still succeed and the bill simply does not fall, so anything that varies per scenario belongs after the fixed block rather than inside it.

The judge is next: scoring only turns 5 and later, on the grounds that the failure class this suite exists for lives in the second half of a session, removes about 40 percent of judge calls. A screening pass with the small model, escalating to the mid model only for turns it scores 2 or below, removes most of the rest at a measured cost of 3 percent disagreement with the full-price judge on a 200-turn sample. Together those take the nightly run from $36 to about $25 and the monthly figure to roughly $1,900.

Going to 160 scenarios multiplies the model spend by four, which is $7,600 a month before the trimming and around $5,200 after it. That is the real form of the resolution question from chapter 7, where a change too small for a 40-scenario suite to separate from its own run-to-run spread is unreadable however carefully it is plotted: five-point sensitivity on end-to-end completion costs about $40,000 a year in model spend, and the alternative — accepting 15-point sensitivity end-to-end while relying on the per-turn instrument for anything smaller — costs $23,000 a year and answers most of the questions people actually ask. Orbit's team went to 70 scenarios rather than 160, chosen to cover the failure shapes production complaints clustered around, and kept the per-turn measurement as the sensitive instrument. Nothing in the arithmetic picks that number. What the arithmetic does is put a price on each candidate; which candidate to buy was a decision, made on coverage grounds, and it could have been made differently.

The half of the cost that is not tokens#

The scripts rot, and there is no version of this approach where they do not. Verbatim utterances at fixed turn indices, per-turn assertion sets, recorded tool cassettes and a frozen rubric are all coupled to a product that changes weekly.

The rows are not alike in how they respond to a bigger suite, and the differences between them carry more than the levels do. Script repair at 3.8 hours is a rate times a population — about 9 scenarios in 40 breaking, at roughly 25 minutes each — and the population is the scenario count, so it scales with the suite directly. Cassette re-recording is the same shape one step removed, since a tool signature change forces a re-record wherever that tool appears and more scenarios touch more tools. Human calibration is sized by how many sessions three raters have to agree on before a rubric can be trusted, and disagreement triage by how many disagreements one person can work through in a week; neither is a function of how many scenarios exist. Scenario additions track the inflow of production complaints rather than the size of the suite.

That split is the reason 160 scenarios do not need four times the triage, and do need something closer to four times the repair work, because breakage is per-script. It is also why the maintenance figure grows with scenario count in a way the token figure does not fully capture.

Thirty-four and a half hours a quarter is about eleven hours a month, a little over an engineer-day. At a loaded cost — salary plus employer taxes, benefits and the overhead an employer carries per head — of €600 a day that is roughly €7,000 a year against $23,000 of model spend, so tokens dominate — which is the opposite of the usual intuition about test suites and is worth checking for your own numbers rather than assuming either way.

That euro figure needs its route spelled out, because the obvious route gives a different answer and a reader who takes it will think the number is wrong. Annualising the hours first and then dividing by an eight-hour working day prices more than twelve days of work, and lands noticeably higher. The €7,000 comes from the clause immediately before it: the monthly load is a little over an engineer-day, and a year is twelve of those at the stated day rate. Both routes are defensible and they are not the same number, so a stated maintenance cost should say which one it used.

The calibration line is the one teams cut first and should cut last. Without a periodic human re-anchoring, the whole apparatus is a model agreeing with a model, and chapter 3's argument — that a simulator built on the agent's own model shares the agent's blind spots, so agreement between them is not evidence of anything — applies to your own suite as forcefully as it applied to the 87-percent slide.

Only one of the four costs appears on the model bill#

Of the four costs, exactly one shows up on the model bill, and it is the one that behaves best. Model spend moves smoothly and proportionally with the thing you control: double the scenarios and it doubles, trim the judge and it falls by roughly the amount you predicted. A team watching the evaluation line on the invoice therefore feels well defended, because the quantity in front of them is real, is accurate to the cent, and responds to their decisions in the direction they expected.

The defence fails because that quantity stays smooth while the properties that actually bite move independently of it. Wall clock is governed by a tokens-per-minute allowance, so a suite can sit comfortably inside its budget and still slide past the half-hour that keeps it in a pull-request loop, and the parallelism that does not fix it never appeared on the bill in the first place. Maintenance grows per-script rather than per-token, so a suite reshaped into more scenarios of fewer turns each leaves the invoice roughly where it was while the repair hours climb. And the scenarios nobody wrote have no meter at all, so the cost of narrow coverage is invisible from every direction. Watching the bill is watching the one variable that was never going to surprise you.

The response is not to stop watching it. On these numbers the tokens genuinely do dominate, and that came out of measurement rather than from assuming it. What the shape argues for is one figure per cost: a wall-clock budget in minutes per run against an allowance in tokens per minute, a quarterly maintenance figure in hours, and a written note of which failure shapes the suite does not cover. Three of the four have to be counted on purpose, because nobody sends you a bill for them.

Starting on Monday#

The order to build this in matters, because most of the value arrives before most of the cost. Begin with eight scenarios derived from real transcripts, with fixed opening utterances, scripted constraints and interruptions, and structured goals — no judge, no rubric, nothing but the deterministic constraint checkers from chapter 4, which assert against a turn's tool calls and text rather than asking a model for an opinion. That runs for cents, catches the €610 booking — the fare taken above a ceiling the traveller had stated seven turns earlier — and is the cheapest thing in this topic.

Add the tool cassettes and a fixed clock next, because without them the run-to-run spread makes everything after it unreadable. Then add the judge and the frozen rubric, which is what makes per-turn plotting possible and is where the bill starts. Both of those are decisions about sequencing, and both are justified by where the value and the cost sit: the deterministic half catches the target failure at almost no cost, and the judge, the second-largest line in the arithmetic above, is deferred until there is something worth scoring.

One step in the sequence is not a decision. Run five identical times to establish the noise floor before believing any comparison — that is a requirement, and it is the one everything downstream rests on, because a difference between two configurations is a finding only when it exceeds what one configuration produces against itself, and until the spread is measured there is no threshold for it to exceed. Only when a plot shows a cliff at a specific turn index is the five-arm ablation worth its $540, and by then you know exactly which threshold to sweep.

Short sessions and single-shot products#

There is a clear case for not building any of this. If your product's sessions are typically two or three turns, no constraint stated at turn 2 has nine turns in which to be forgotten, and the failure class this whole topic addresses does not exist for you. If your agent never compacts, evicts or truncates because sessions comfortably fit the window, the specific mechanism from chapter 6 — summarising the earlier turns to keep the context inside the model's limit, and losing details in the summary — is absent. And if what you ship is classification, extraction or one-shot generation, the money buys far more coverage spent on single-turn cases.

The threshold worth using is whether your p90 session — the length nine sessions in ten come in under — crosses whatever context management your harness performs, and whether users state constraints that must persist across turns. Those two are requirements rather than preferences, and each fails for a mechanical reason: if the p90 session never reaches the harness's compaction step there is no mechanism to measure, and if nothing a user says has to survive into a later turn there is no adherence to check. Both true, and the suite is load-bearing; either false, and it is an expensive way to confirm things you already know.

The multiplication itself stops holding for a suite that grows by lengthening its sessions rather than by adding scenarios. Scenario count is a clean multiplier, since twice the scenarios is twice every term in the sum. Turn count is not, because agent_in is a mean over a context that grows with the turn index, so doubling the turns more than doubles the agent's input tokens unless compaction holds that mean flat. A suite that goes deeper rather than wider has to be repriced from the parameters rather than scaled from the last invoice.

Priced honestly, the suite is one multiplication and three things the multiplication cannot see. Scenarios times turns times calls per turn, across the agent, the simulator and the judge, gives a model bill the four-row table closes over exactly. The wall clock has its own arithmetic and its own binding constraint, the maintenance hours have their own accounting and their own scaling behaviour, and the coverage nobody wrote has neither. Knowing all four turns the scenario-count question into a purchase with a stated price rather than an argument about whether evaluation is worth doing.

The thing you are building is an instrument with a known uncertainty, not a score. Everything in this topic — the frozen scripts, the different-family simulator, the anchors written before the run, the noise floor, the paired comparisons, the intervals attached to every rate — exists so that when the number moves you know what moved it. The €610 booking is still going to happen occasionally, and the difference is that now it happens in a suite on Tuesday night rather than in front of a traveller at a gate.