Pricing one accepted output

The unit that matters in a generate-then-verify loop is the accepted output, because that is the thing the product ships. Rejected candidates, retries and requests that exhaust their attempts are overhead to be amortised across the accepted ones, and a cost model built per call rather than per accepted output simply omits them.

Two numbers govern that amortisation and neither is the one usually quoted. The first is the per-attempt rejection rate, which sets how many generations and checks a request consumes on average. The second is the tail of each component — conventionally p95, the latency one request in twenty exceeds — because a check that is cheap at the median can be expensive at p95 under exactly the load conditions that also raise the rejection rate. This chapter prices a loop with both in it, and then compares the result against the alternative that a write-generating agent is usually replacing.

Pricing one accepted output#

Take an agent that provisions customer accounts and writes the rows itself. It produces a candidate INSERT set, the service opens a transaction against production Postgres, applies the writes, lets the foreign keys and unique indexes render judgement, reads the outcome and rolls the transaction back. If it committed cleanly in the trial, the writes are applied for real. The check measures 20 ms at p50 in staging against a generation step that takes about 1.9 seconds, which is the number everybody quotes. It is also 900 ms at p95 under production load, because the trial transaction takes the same locks and waits behind the same long-running reporting queries as everything else, and it rejects about 20% of first attempts.

Start with the simplest model, where each attempt fails independently with probability 0.20 and the ceiling is three attempts. Eighty per cent of requests are done in one attempt, 16% take two, 3.2% take three, and 0.8% exhaust the ceiling and are never served. The expected number of generations per request is 1 + 0.20 + 0.04 = 1.24, and the same count of checks. Per accepted output you divide by the 99.2% that succeed, giving 1.25 generations and 1.25 checks. At 2,400 input and 350 output tokens per generation and this team's contracted rate of $3.00 per million input and $15.00 per million output tokens, one generation is $0.0125, so one accepted output costs $0.0156 in model spend. The loop has added 25% to the model bill and it has not run any expensive checks to do it.

Four steps are compressed into that paragraph and each is worth performing. The shares come from multiplying the failure probability along a path: every request makes a first attempt, one in five of those fails and makes a second, and one in five of those fails and makes a third, so the shares reaching each attempt are one, a fifth, and a twenty-fifth. Multiply by the chance of failing again and 0.8 in a hundred exhaust all three.

The expected generation count is then those shares added together rather than multiplied, because a request that reaches the third attempt has also paid for the first and second — 1 + 0.20 + 0.04 = 1.24 generations for every request that arrives. Turning that into a cost per accepted output means dividing by the share of requests that produce one, which is 99.2%, and dividing by a number just under one nudges 1.24 up to 1.25. That division is the step people skip, and skipping it charges the exhausted requests' generations to nobody.

The token price is two multiplications and an addition: 2,400 input tokens at $3.00 per million is a bit over seven-thousandths of a dollar, 350 output tokens at $15.00 per million is a bit over five-thousandths, and together they make the $0.0125. Note that two-fifths of the cost of this call comes from about an eighth of its tokens, because output is priced at five times input — which is why a prompt change that adds context is much cheaper than one that lengthens the answer. Multiply $0.0125 by 1.25 accepted-output generations and the result is $0.0156. The 25% is the same 1.25 read as a percentage over one, and it is the whole cost of the loop before any check has been priced at all.

The independence assumption is generous, and measuring the real conditional rates is worth the effort because they diverge quickly. In this service the second attempt fails 35% of the time and the third fails 45% of the time, since requests that failed once are drawn from the harder end of the distribution: ambiguous customer names, tenants with unusual billing arrangements, requests that reference a parent account created seconds earlier. With those rates, 20% of requests reach a second attempt and 7% reach a third, expected generations per request rise to 1.27, and 3.15% of requests exhaust the ceiling instead of 0.8%. Per accepted output that is 1.31 generations, $0.0164 of model spend, and — the number that actually hurts — a four-fold increase in the traffic that ends up somewhere other than the happy path.

The reason the exhaustion rate moves so much further than the cost is that it is a product of three failures rather than a sum. Model spend went up by about four in a hundred, from $0.0156 to $0.0164, because it is driven by the expected attempt count and each extra attempt adds at most one generation. Exhaustion is the chance of failing all three times running, which under independence is a fifth of a fifth of a fifth, and under the measured conditional rates is a fifth of roughly a third of nearly a half. Raising each factor moves their product much harder than it moves any sum they appear in. That is the general shape to expect whenever a loop's rates are correlated: the average cost barely notices, and the thing on the end of the chain — the escalation queue, the error page, the on-call rota — takes the whole of the change.

It is also worth being clear about what "the harder end of the distribution" does to the independence assumption, because independence is not a modelling convenience here, it is a claim about the world that is false in a specific direction. A first failure is evidence about the request. Requests that fail are the ambiguous ones, and ambiguity does not go away between attempts, so the second attempt is drawn from a population that was already selected for being hard. Independence would say the second attempt fails at the same 20% as the first; the measurement says 35%. The error therefore always runs the same way — an independent model understates exhaustion, never overstates it — which makes it a bad assumption to leave in a design document even as a placeholder.

The "slow" column is generation at its p95 of 4.6 seconds and the check at its p95 of 900 ms, repeated for each attempt. It is not a p95 of the end-to-end distribution — you cannot add p95s — but it describes what a retried request looks like when it also lands on a slow database, and the point is that these two conditions are not independent in the way people assume. A request that makes three attempts runs three checks, so its probability of hitting at least one 900 ms check is 1 − 0.95³ = 14.3%, nearly triple a single-attempt request's exposure. The slow checks concentrate on the requests that were already the slowest, which is why the end-to-end p95 in this service is 7.4 seconds while the single-attempt p95 is 4.7 seconds.

That expression is short and every symbol in it is doing work. A p95 is by definition the value one call in twenty exceeds, so any one check has a nineteen-in-twenty chance of coming in under 900 ms. Three checks all coming in under it is nineteen-twentieths multiplied by itself three times, which is a bit under eighty-six in a hundred — and the quantity wanted is the opposite of that, so it is one minus the product. Computing it the direct way, as the chance of the first being slow or the second or the third, would require subtracting the overlaps; computing the complement avoids that entirely, which is why "one minus the probability that none of them happens" is the standard move for this shape of question.

The reason p95s cannot simply be added is the same argument run in reverse. For a two-attempt request to take the sum of two p95 latencies, both components would have to be in their own top twentieth at once, and that is a one-in-four-hundred event rather than a one-in-twenty one. The sum of the tails therefore describes something much rarer than the tail of the sum. The "slow" column is built by stacking generation's 4.6 seconds and the check's 900 ms — five and a half seconds of it per attempt — and it is an illustration of what a bad run looks like, not a percentile of anything. Read it as a worst-case sketch, and read the 7.4-second end-to-end p95 as the measured quantity, because that one was observed rather than assembled.

The same table answers a question that usually gets settled by intuition, which is where to put the attempt ceiling. Each additional attempt has a marginal cost paid by every request that reaches it and a marginal benefit paid out only to the ones it rescues, so the two have to be divided rather than compared. Seven per cent of requests reach a third attempt here, and 55% of those are accepted, which rescues 3.85% of traffic at a marginal spend of 0.070 × $0.0125 = $0.00088 per request. Divided across the rescues, the third attempt costs about $0.023 per output it saves, against $1.10 to send that same request to a person, so it is obviously worth having.

The shape of that calculation is the transferable part, because the two halves have different denominators and that is exactly where the intuition goes wrong. The cost is spread over every request that reaches the attempt, which here is seven in a hundred, each paying one more generation. The benefit accrues only to the ones the attempt rescues, which is fifty-five in a hundred of those seven, or just under four requests in a hundred overall. Dividing a cost carried by seven by a benefit delivered to under four multiplies the per-unit cost by about two — so the third attempt's true price is roughly twice the price of a generation, not the price of a generation. Even doubled it is two cents against a dollar ten, which is why the answer here is not close and why it is still worth writing the division down: the same arithmetic on a check that costs real money, or a fallback that costs pennies rather than dollars, comes out the other way. A fourth attempt would reach 3.15% of requests and, at the declining success rates this service exhibits, accept perhaps 40% of them, which is $0.031 per rescue — still cheap in money and no longer cheap in time, since those requests would now be running for eight seconds before anyone hears anything. The ceiling stops paying when the marginal cost per rescued output approaches the cost of the fallback, and in a loop with a generous fallback and expensive attempts that can happen at the second attempt rather than the fifth.

Where the arithmetic is genuinely uncomfortable is when the check itself is the expensive component, because then the marginal cost of an attempt is dominated by candidates that get thrown away. At 900 ms and a database connection per check, the third attempt's checks consume 0.070 × 900 ms of connection-hold time per request across the whole traffic volume, which at 40 requests per second is about 2.5 connections permanently occupied by work that will be discarded. That is invisible in a per-request cost model and extremely visible in a connection pool.

That last figure comes out of a relationship worth knowing by name, because it turns a latency into a headcount of connections. The number of things in progress at any moment equals the rate they arrive at multiplied by how long each one stays. Seven in a hundred requests reach a third attempt, and each holds a connection for 900 ms while its check runs, so the average connection-time each arriving request contributes is seven-hundredths of 900 ms, which is a little over sixty milliseconds. Multiply by forty arrivals a second and the answer is about two and a half connection-seconds consumed every second — which is to say two and a half connections busy at all times. The unit conversion is the point: seconds per request multiplied by requests per second leaves a pure count, and that count is what a pool has to be sized for. Nothing about it appears in a cost model denominated in dollars per request, and everything about it appears the first time the pool saturates.

Instrumenting so the numbers exist#

None of the above can be recovered from a single end-to-end histogram, because that histogram mixes one-attempt and three-attempt requests into one distribution and hides which component moved. Three separate instruments are needed, and the retry count has to be a label on all of them.

GEN_SECONDS   = Histogram("gen_seconds",   "generation latency",   ["attempt"])
VERIFY_SECONDS= Histogram("verify_seconds","verification latency", ["attempt", "outcome"])
E2E_SECONDS   = Histogram("e2e_seconds",   "request latency",      ["attempts_used", "served"])
GEN_TOKENS    = Counter("gen_tokens_total","tokens billed",        ["attempt", "kind"])
OUTCOMES      = Counter("outcomes_total",  "terminal outcomes",    ["outcome"])
# outcome ∈ {accepted, exhausted}; kind ∈ {input, output}

# cost per accepted output, computed from the counters rather than assumed:
#   sum(gen_tokens{kind="input"})  * price_in
# + sum(gen_tokens{kind="output"}) * price_out
# + verify_calls * verify_unit_cost
#   ────────────────────────────────────────
#   outcomes_total{outcome="accepted"}

Two habits make this worth the trouble. First, price per accepted output rather than per call, computed from the counters in the same query, so the number on the dashboard already includes the retries. Second, cut every latency panel by attempts_used; a single p95 line that drifts upward tells you nothing about whether generation got slower, the database got slower, or the rejection rate rose, and those have completely different fixes. A panel labelled "Verification latency" that mixes first-attempt and third-attempt checks into one distribution cannot separate a check that has got slower from a check whose additional invocations are landing on the requests that were already contending for locks, and both of those push the same line upward by the same amount. When the invoice team from the previous chapter finally split their panels this way, they found their end-to-end p95 regression was entirely a rejection-rate change caused by a new document layout, with both component latencies flat.

When the queue wins#

The comparison that gives all of this meaning is against the alternative you would otherwise run, which for write-generating agents is usually a human review queue: the agent proposes, a person approves, the write happens. Reviewing one proposed account provisioning takes about 90 seconds and, fully loaded, costs around $1.10.

At the measured rates the loop looks strong. Model spend is $0.0164 per accepted output, and the 3.15% of requests that exhaust their attempts have to go somewhere — almost always to that same human queue — adding 0.0315 × $1.10 = $0.0347 per request in human cost. Total is roughly $0.05 per accepted output against $1.10 for reviewing everything, so the loop is more than twenty times cheaper, and the crossover is not close.

Both terms in that total are worth assembling, because the second one is the whole reason the comparison is a comparison. The model spend is the $0.0164 already derived. The human term is the escalation rate multiplied by the price of a review: 3.15 in a hundred requests exhaust their attempts, each costing $1.10 in somebody's time, which averaged across all requests is a bit under three and a half cents. Add the two and one accepted output costs about five cents. What makes the loop attractive is not that model calls are cheap; it is that only three in a hundred requests reach the expensive path at all. The human review cost has been multiplied by a small number, and that small number is the escalation rate.

The sensitivity is entirely in the escalation rate, and it moves fast. If the per-attempt rejection rate rises from 20% to 55% — a new tenant type, a schema change, a model version that reads the prompt slightly differently — then 16.6% of requests exhaust the ceiling, human cost per request becomes $0.183, and the model spend rises to $0.028 as well. The loop now costs $0.21 per accepted output, still cheaper than $1.10 but no longer in a different league, and it has also inserted a six-second delay in front of every escalated request before the human ever sees it.

Follow that through and the disproportion is the lesson. The per-attempt rejection rate did not quite triple. Exhaustion is that rate cubed, so it went from eight in a thousand to something over sixteen in a hundred — more than twenty times — and since the human term is exhaustion multiplied by $1.10, the human term went up by the same factor. Model spend rose by well under a doubling over the same move, because expected attempts cannot exceed the ceiling however badly the generator performs. So a fourfold change in cost is almost entirely a change in how much human review the loop is buying, and the token line, which is the line anybody would have modelled, is the line that moved least.

Three conditions flip the comparison outright, and they are worth watching for explicitly. The first is a rejection rate high enough that most traffic ends in the queue anyway, at which point you are paying for model attempts on top of the full review cost. The second is a latency budget the tail cannot meet: if the product promises a response in five seconds, the 3.85% of requests taking 16.5 seconds are failures regardless of what they cost, and a queue that promises an answer in four hours may be the more honest contract. The third is severity — when an escaped false accept costs more than $1.10 in remediation, the residual error rate on accepted output, multiplied by that cost, belongs in the loop's column and can exceed everything else in it.

A verification loop is priced by three quantities and a median is none of them: the per-attempt rejection rate, the tail of each component under real load, and the cost of whatever catches the requests that exhaust the ceiling. Instrumented separately with the attempt count as a label, all three are readable off a dashboard; assumed, they produce a design review in which a 20-millisecond number stands in for a loop that costs $0.05 per accepted output and holds two and a half database connections on work it throws away.

Every number above rests on the check being the one described: the candidate write applied against production, judged by the live indexes and the transactions currently in flight. The most natural response to a 900 ms p95 under load is to move the check somewhere quieter, and that instinct is what the next chapter is about. A check is defined by what it runs against, so relocating it to reduce its cost is not a deployment detail — it changes which outputs the check accepts, and it can convert a cheap deterministic verifier into a machine that produces false accepts at exactly the moments the system is busiest.