Agent cost and latency control

Read these first

• Error recovery
• Prompt caching mechanics

What you will know

• Reduce an agent's cost measurably without losing task success

  Use a before-and-after workload with a fixed success criterion; averages can hide a small set of expensive, high-value failures.

• Use a cheaper model for some steps and justify which

  Routing errors occur when a step that looks mechanical requires hidden judgement, long-context reasoning, or reliable tool arguments.

• Parallelise independent steps

  Fan-out lowers wall-clock time but can exhaust rate limits, inflate token spend, and create races when supposedly independent steps share state.

• Terminate early when the answer is already good enough

  A measurable stopping rule must distinguish acceptable partial work from plausible-looking output that still lacks required evidence or validation.

• Report cost per successful task, not cost per call

  Include retries, failed tasks, parallel branches, cached work, and evaluator outcomes in the denominator and numerator so savings cannot be manufactured by shifting failure elsewhere.

Chapters

1. 1

   Cost per successful task is the denominator that cannot be gamed

   Per-call cost falls whenever calls get cheaper, and rises invisibly when an optimisation causes three retries, a fan-out of four branches, and a task that fails and gets redone by a human. Put retries, abandoned runs, parallel branches and evaluator outcomes into the accounting and the picture changes. It is also how you catch the classic illusion: lowering the iteration cap, which truncates the expensive runs rather than making them cheaper.

   Read chapter
2. 2

   Time to success splits into model calls, tool waits and retries

   Cost per successful task has a twin, and mean call duration is exactly as gameable as a per-call price, because a run that fails fast improves it. Decompose a task's wall clock into sequential model calls, tool waits, retries and queueing, then measure time to success with the retries and the human handoff included. That breakdown is what tells you whether parallelism is even the right lever, and it gives you the target that later bounds concurrency. It also reframes two of the levers ahead: a cache hit is faster as well as cheaper, and a smaller model is usually quicker, which is why tiering is tempting on a latency budget and not only on a bill.

   Read chapter
3. 3

   An agent loop usually invalidates its own cacheable prefix

   System prompt and tool definitions are stable, so the prefix should be cached on every iteration after the first, and then the loop prepends a fresh run summary and the shared prefix ends at token zero. Anything that mutates the front of the message list has the same effect. Learn where reuse is real in an agent loop, and how to read cached-token counts per request instead of trusting that caching is enabled.

   Read chapter
4. 4

   The step that writes tool arguments is the one not to downgrade

   Summarising a document tolerates a weaker model; producing a call with an exact record id, a valid enum and correctly nested arguments does not, and its failures arrive as retries and wrong actions rather than as worse prose. This chapter assigns models step by step, flags the steps whose mechanical appearance hides judgement or long-context reasoning, and measures the downgrade on task success rather than on eyeballed output.

   Read chapter
5. 5

   Stopping early needs a criterion, not a feeling of sufficiency

   An agent that halts when the answer looks complete will halt on fluent output that never checked the second source or ran the validation step. A usable stopping rule is checkable without the model's opinion: required fields populated, citations resolving, a test passing, a confidence signal above a threshold measured against labelled outcomes. Build it, then report what it saved and what it truncated.

   Read chapter
6. 6

   A step budget bounds the blast radius, not the bill

   An iteration cap is not only the accounting trick the first chapter warned about; it has a legitimate job, which is to bound a run that keeps calling tools, spending money and touching systems long after it stopped making progress. That makes it a safety control with a cost side effect, and it is why a budget is not the stopping rule of the previous chapter: the rule fires when the work is good enough, the budget fires when the work has gone wrong. The difference shows in what happens on exhaustion, which has to be a loud failure, an escalation or a handoff, never a partial answer returned as though it were complete.

   Read chapter
7. 7

   Fan-out cuts wall-clock time and buys rate limits and races

   Four independent calls at once finish sooner, then hit a per-minute token ceiling that throttles the whole run, and turn out to share a scratch file or a record that two branches now write. Token spend goes up too, because branches that the stopping rule would have skipped after an early answer all execute. Decide what independence actually means here, then bound concurrency deliberately against the latency target from your breakdown instead of by whatever the model emits.

   Read chapter
8. 8

   Forty percent cheaper and three points worse is not yet a decision

   Before anyone argues, find where the missing three points went: into retries that spend the saving back, into human handoffs with their own cost, or into silent wrong answers that cost far more than a token bill. Then price a failure in this workload and compare it with the saving per task. This chapter runs the before-and-after properly, on a fixed success criterion, and defends both possible answers.

   Read chapter