Cost per successful task is the denominator that cannot be gamed

The figures in this chapter come from a kind of system whose cost behaves unlike anything else on the same invoice, and it is worth being explicit about what that system is before any of them are read.

An agent, in the ordinary sense the rest of this topic uses, is a loop in which a model is given a task and a set of tools, chooses one to call, reads the result, and then decides whether to call another tool or to answer. What follows from that for spending is the property that makes the subject hard: the number of model calls a task consumes is chosen at run time by the model rather than written down by an engineer. A REST endpoint that costs $0.002 to serve costs $0.002 on Monday and on Sunday. An agent handling shipping exceptions settles a simple ticket in four model calls and spends twenty-two on one where the customer's address is ambiguous, and no line of code says which of those will happen.

Agent cost and latency control is the practice of measuring what that variable work costs — money paid to the model provider, and time somebody spends waiting for an answer — and then reducing it without reducing the number of tasks the agent finishes acceptably. The second half of that sentence is what makes the subject difficult, because the measure most teams already have does not enforce it. That measure is cost per call, and the one this chapter puts in its place is cost per successful task: every dollar a task caused the system to spend, across first attempts, retries, abandoned runs, parallel branches and evaluator calls, divided by the number of tasks that came out the far end acceptably. Under that measure the iteration-cap change this chapter takes apart saved nothing at all, because the tickets it stopped resolving are missing from the denominator.

Why a per-call price cannot see an optimisation#

Write the two measures as fractions and the difference is visible before any figures are attached. Cost per call is total spend divided by the number of calls. Cost per successful task is total spend divided by the number of tasks that succeeded. The numerators are the same quantity; everything that separates the two is what sits underneath.

A per-call price describes a unit of work while the number of units is held fixed. That is the condition under which it means something, and an optimisation is exactly the kind of change that violates it. Retries add calls. Fan-out — dispatching several branches of work at the same time and keeping whichever one pays off — adds calls. A weaker model that produces a malformed tool argument adds a correction round trip that the error-recovery machinery from topic 10.6 handles quietly and charges you for. A cap subtracts calls by abandoning work. All four of those changes move the per-call number in a direction that has nothing to do with whether the system got cheaper at doing its job.

The reason the second fraction resists this is that its denominator is the thing you actually sell, and everything else in the accounting is a stage on the way there — which is why stages can be shuffled between each other without the total moving. A change can move work between stages, retry it, fork it or drop it, and none of that alters how many tasks came out acceptably unless the change altered the outcome. So the denominator only moves when something real moved, which is the property the per-call denominator lacks — calls are an implementation detail, and a measure whose denominator is an implementation detail can be improved by changing the implementation and nothing else.

The techniques the rest of this topic prices are individually cheap to implement: reuse the parts of a prompt that repeat between calls, send the undemanding steps to a smaller model, run independent steps at the same time, halt the loop once the answer is already good enough, and bound how many steps a run may take. Engineers meet the subject at one of three moments — an invoice growing faster than usage, a complaint that the agent takes a minute to answer while the dashboard reports three-second model calls, or a director who has seen the invoice and wants thirty percent off next quarter. Each of those conversations ends in a comparison between two numbers, and a comparison is only as sound as the denominator underneath it.

The denominator decides what you are allowed to hide#

Here is one week of that shipping-exception agent under five different denominators, with a single change as the only difference between the columns: the loop's iteration cap moved from twenty steps to eight.

The first six rows are the ones that get reported, and five of them improve. Every row in that table is built from the same two spend figures, and taking them apart is the fastest way to see that five improving rows and one flat row are the same fact wearing different clothes.

Start with total spend, which is not a row but is behind all of them. Cost per ticket presented is spend divided by the thousand tickets, so the first column is spending $214 per thousand and the second $186 — the figures that made the change look like a win. Divide the same spend by the number of model calls and you get the second row: fewer calls, each cheaper, because the calls the cap removed were the late ones carrying the longest context. Divide it by runs started and you get the fourth row, which improves twice as fast as spend because runs went up while spend went down.

That fourth row is the one to distrust first, and the reason is a mismatch between what the two columns count. Runs started rose from 1,270 to 1,340 per thousand tickets, which happened because truncated runs failed, the retry policy attempted them again, and the second attempt also truncated. So the two columns are dividing by populations of runs that are not the same runs and were not produced by the same process. A ratio whose denominator changes composition between the two things being compared is not a comparison; it is two different measurements printed next to each other. Any single-number view that sits above the task level will show you some version of this.

Now take the seventh row, which is spend divided by tickets the agent actually resolved. Spend per ticket presented fell by thirteen percent, and tickets resolved fell by thirteen percent. Two quantities that move by the same proportion have a ratio that does not move, which is why the row is flat to within a fifth of a cent. That is the honest reading of what happened to the agent's own efficiency: it did not become more efficient at all, it just stopped attempting the hard tickets.

State what that flat row licenses, because it is narrower than it looks. It says the cost of a ticket the agent finished is unchanged, which rules out the claim that the cap made the agent better at its job. It does not say the agent performs identically on the tickets it still handles — the same flat ratio is compatible with getting cheaper on some and dearer on others in offsetting amounts. A single average is compatible with every distribution that produces it, which is why the segmentation in the final chapter of this topic exists.

The last row is what the business paid. Fully loaded means the escalation is priced in the numerator at its real handling cost: escalations went from 90 to 208 per thousand tickets, a hundred and eighteen additional tickets finished by a person at about $6.40 of handling time each, which is the roughly $750 per thousand tickets that appeared on the outsourced desk's invoice. Add that to the model spend, divide by the smaller number of agent-resolved tickets, and the measure more than doubles, because the work did not disappear when the agent stopped doing it.

Whether $6.40 is the right figure is a question for whoever owns that contract; the point is that a number belongs there, because the alternative is an accounting in which failure is free. If your escalation path is an internal queue rather than a vendor, price it at the loaded hourly cost of the person who works it multiplied by the median handling time, and revisit the figure when either changes.

Everything the run spent belongs in the numerator#

The denominator is the part people argue about and the numerator is the part they get wrong quietly, because agent spending is spread across events that do not all look like billable work. Five categories go missing regularly. Retried runs, where the same task is attempted twice and only the successful attempt is attributed. Abandoned runs, where a user closes the tab at step six and the trace is dropped from the rollup because it has no outcome. Parallel branches, where the orchestrator fans out to four candidate approaches and the accounting follows only the branch that was selected. Cache writes, which are billed above the base input rate — 1.25× on Anthropic models for the five-minute TTL, the time to live after which an entry nothing has read is discarded — and which are easy to omit if you are summing input tokens from a field that excludes them. And evaluator or judge calls, which are model calls you made about the task and are therefore part of what the task cost.

The five have one shape in common, and seeing it is more useful than memorising the list. Each is spending that a naive rollup drops because the row it belongs to has no outcome attached, or because the outcome it does have belongs to a different row. Anything with that property will go missing, so the list is extensible by anybody who finds a new way for their system to produce work without an outcome.

Branches are the category that surprises teams most, because the discarded work is often the majority of it. When the shipping-exception agent cannot determine which of a customer's shipments the complaint refers to, it forks three lookups — by tracking number, by order line, and by delivery date window — and keeps whichever resolves. Three branches ran and three branches were billed; one produced the answer. Attributing the task's cost to the winning branch understates it by a factor close to three on exactly the tasks that were hardest.

Follow that bias one step further, because a systematic error in a measure does more damage than a random one. Under-counting forked tasks makes forking look free, so every comparison between a configuration that forks and one that does not is biased toward the forking one, and the bias is largest on the hard tickets where forking happens most. A measurement error that favours a particular design choice will select for that design choice over time, which is worse than noise: it biases every subsequent comparison in favour of whatever makes the agent fork more. The same applies to evaluator calls: a judge model that scores every completed run at 1,100 input tokens and 60 output tokens costs about $0.004 per task on a mid-tier model — providers sell a family of models at different prices and capabilities, and this is a middle rung of one — which is under 2% of the baseline figure and worth counting anyway, if only because a later chapter proposes running it more often.

That is a data-modelling requirement more than an analysis one. If your traces do not carry a stable task identifier that survives retries, branches and abandonment, no query can reconstruct the grouping afterwards. The shape that works is one row per run, with the task it belongs to, the attempt index, the parent branch if it was forked, the token counts broken out by kind, and the outcome as judged by whatever your acceptance criterion is.

CREATE TABLE agent_run (
  run_id           uuid PRIMARY KEY,
  task_id          uuid NOT NULL,           -- stable across retries and branches
  attempt          int  NOT NULL,           -- 1, 2, 3…
  parent_run_id    uuid,                    -- set when this run is a fan-out branch
  model            text NOT NULL,
  input_tokens     bigint NOT NULL,         -- uncached input
  cache_write_tokens bigint NOT NULL,
  cache_read_tokens  bigint NOT NULL,
  output_tokens    bigint NOT NULL,
  usd_cost         numeric(12,6) NOT NULL,  -- includes cache write premium
  ended_reason     text NOT NULL,           -- completed | budget_exhausted
                                            -- | tool_failure | abandoned
  outcome          text                     -- accepted | rejected | escalated | null
);

Two columns in that schema are doing the work that the rollup cannot do without. task_id is the grain the measure is defined on, and it has to be assigned when the task arrives rather than when a run starts, because a run that is a retry or a branch needs to inherit it. attempt and parent_run_id then record why a run exists, which is what lets a query count runs and tasks separately without one of them silently standing in for the other.

With that in place the measure is a two-line aggregate, and the important part is that the join is on task_id, so every run a task ever caused lands in the numerator whether or not it produced the answer that shipped.

SELECT
  SUM(r.usd_cost)                                    AS spend,
  COUNT(DISTINCT t.task_id) FILTER (
    WHERE t.final_outcome = 'accepted')              AS successes,
  SUM(r.usd_cost) / NULLIF(COUNT(DISTINCT t.task_id)
    FILTER (WHERE t.final_outcome = 'accepted'), 0)  AS usd_per_successful_task
FROM agent_run r
JOIN task t USING (task_id)
WHERE t.presented_at >= now() - interval '7 days';

The asymmetry inside that query is deliberate and it is the whole design. The numerator sums over runs, so it collects everything; the denominator counts distinct tasks with an accepted outcome, so a task with four runs contributes four costs and one success. Run that query against a week of the shipping-exception agent and the cap change shows up as a flat line. Run the per-call version and it shows up as a win. Neither query is wrong about what it computes; they answer different questions, and only one of them is the question the spend was authorised to answer.

Why truncating the tail is the tempting mistake#

Cutting the iteration cap is the reduction most likely to be illusory, and it is worth understanding why it is so persuasive. Agent run cost is heavily skewed. In the baseline week, the median successful run used 5 model calls and cost $0.09, while the 95th percentile used 17 and cost $0.71, and about 6% of runs accounted for a quarter of the spend.

Those figures explain the appeal mechanically. When a small share of runs carries a large share of the bill, a cap placed below the 95th percentile removes a disproportionate part of the spend on the day it deploys, because it is aimed directly at where the money is. The average improves immediately, with no code review, no evaluation harness, and no argument about model quality. It looks like the cleanest possible optimisation.

What makes it illusory is the direction of causation. Long runs are not wasteful loops that happen to be long; they are long because those tasks are hard — the address was ambiguous, the carrier's tracking API returned a status the policy document does not mention, the customer's order had three shipments and two of them were split. The expensive tail and the difficult tail are the same tail, which is exactly why cutting one cuts the other. Capping does not make those tasks cheaper to solve, it makes them not get solved, and the money moves to whatever picks up the failure.

A genuine efficiency change reduces spend at constant success: the same 910 tickets resolved for less. A truncation reduces spend by reducing successes, and the only way to tell them apart is a measure that carries successes in the denominator.

There is a legitimate version of the cap, and the sixth chapter of this topic is about it, so it is worth being precise about what is wrong here rather than dismissing the mechanism. The problem is not that a bound exists; a loop that can call tools indefinitely is a hazard regardless of price. The problem is presenting the bound as a cost improvement, and setting its value by looking at the spend distribution rather than at the point where a run has stopped making progress. A cap chosen to shave the 95th percentile of cost will fire on tasks that were three steps from finishing, and it will fire most often on the tasks that were worth the most attention.

The same test catches other versions of the move. Lowering the per-run token ceiling, dropping the retry allowance from three attempts to one, or routing anything the classifier finds difficult straight to the human queue all reduce cost per call and cost per run, and all of them relocate the work rather than removing it. Ask of any proposed saving: does this leave the set of completed tasks the same? If it does not, the saving is at least partly a transfer.

The measure is an average, and it is not a runtime control#

Cost per successful task cannot be enforced inside a request, because success is not known until an evaluator, a reviewer or a customer says so, and in the shipping-exception workload that verdict arrives up to 48 hours later when the ticket closes. A control has to be evaluable at the moment it fires, and this one is evaluable two days afterwards. Anything you want to enforce during a run needs a bound that is knowable during the run, which is the subject of the step budget four chapters from now. This measure is for deciding what to ship, not for policing a request in flight.

It is also an average, and averages over skewed distributions hide the cases that matter most. A workload where 2% of tasks are worth 200 times the rest can be optimised into a state where cost per successful task improves and revenue falls, because the tasks that got dropped were the valuable ones. Notice that this is the same skew that made the cap attractive, seen from the value side rather than the cost side: a long tail exists in both, and a measure that averages over tasks treats a task worth two hundred times another as one task. Segmenting by task value, or at minimum reporting the measure alongside a p95 — the value that 95% of runs come in under, the notation used throughout this topic — and a success rate broken out by task type, is what keeps that visible.

The final limitation is the one that decides whether any of this works: the number is only as trustworthy as the definition of success behind it. If "accepted" means a reviewer clicked approve, and reviewers get faster when the queue is long, the denominator drifts upward and every optimisation looks better than it is. The criterion has to be fixed, written down, and applied identically to both sides of any comparison — a rule that comes back with force in the final chapter, when the numbers in dispute are a 40% saving against a three-point drop in success.

None of this touches how long anything took. Cost per successful task will happily approve a change that makes every ticket 40 seconds slower, and time has the same gameable structure as money: a run that fails fast improves your average duration exactly as reliably as a run that succeeds quickly. The twin measure is built the same way, on the same denominator, and it comes apart into parts that can each be moved by different things.