Eval cost and cadence

An eval budget arrives as a single number in a planning document, and the useful first move is multiplication rather than negotiation. A suite of 400 frozen cases at roughly $6 a run, fired on every merge by a team merging thirty times a week, is $780 a month. Against a $200 budget that is not a trimming problem — it is a statement that the every-merge-full-set arrangement is unaffordable at this price and something structural has to change.

Which term to change is the decision, and the intuition most teams bring to it is wrong in a specific way. Case count is the visible term, sitting in a file somebody can open, so it is the one that gets cut. Cadence is written in a workflow trigger where nobody reads it as a number, and it carries the largest multiplier. This chapter decomposes both — where the money inside one run goes, and what the trigger does to the total — and then works a real budget into tiers, each with what it gives up written down beside it.

Where the money in one run goes#

Cost per run decomposes into five factors: the number of cases, the draws per case, the tokens each generation consumes, the tokens each judge call consumes, and the inflation from retries and failures. Writing it as code rather than prose makes the dominant term obvious.

from dataclasses import dataclass

@dataclass
class Rates:            # USD per 1M tokens
    gen_in: float
    gen_out: float
    judge_in: float
    judge_out: float

def cost_per_run(
    cases: int,
    draws: int,
    gen_in_tok: int,
    gen_out_tok: int,
    judge_in_tok: int,
    judge_out_tok: int,
    rates: Rates,
    retry_inflation: float = 1.08,
) -> dict[str, float]:
    per_m = 1_000_000
    gen = cases * draws * (
        gen_in_tok * rates.gen_in + gen_out_tok * rates.gen_out
    ) / per_m
    judge = cases * draws * (
        judge_in_tok * rates.judge_in + judge_out_tok * rates.judge_out
    ) / per_m
    return {
        "generation": gen * retry_inflation,
        "judge": judge * retry_inflation,
        "total": (gen + judge) * retry_inflation,
    }

# an order-support suite: 400 cases, one draw, small generator, large judge
cost_per_run(
    cases=400, draws=1,
    gen_in_tok=1_800, gen_out_tok=400,
    judge_in_tok=2_600, judge_out_tok=150,
    rates=Rates(gen_in=0.15, gen_out=0.60, judge_in=3.00, judge_out=15.00),
)
# {'generation': 0.22, 'judge': 4.61, 'total': 4.83}

The judge is 80% of the bill, which is the usual shape and the first place to look for savings that cost no signal. Judge input is large because the prompt carries the rubric, the user question, the retrieved context and the full answer. Trimming the retrieved context out of the judge prompt where the rubric does not reference it took this suite from 2,600 to 1,700 judge input tokens and $4.83 to $3.75 a run, with a flip rate between repeated runs that did not move.

The 80% is the two judge rows of the share column added together, 65 and 15, and it is worth asking why a step that only reads text costs twenty times what producing the text costs. Two multipliers stack. The judge's prompt is structurally larger than the generator's, because the generator reads the question and its retrieved context while the judge reads all of that plus the answer the generator wrote plus the rubric it is grading against. And the judge usually runs on a stronger model, priced an order of magnitude above the generator per token, because deciding whether an answer is correct is a harder reading task than producing a plausible one. A larger prompt at a higher rate multiplies, which is how the step nobody thinks of as the workload ends up being four fifths of it.

The trim is that multiplication run backwards, and following it shows why the judge prompt is where to look first. Dropping the retrieved context takes nine hundred tokens off every judge call. Nine hundred tokens at the judge's input rate is a little under three tenths of a cent for one case, and 400 cases turn that into a little over a dollar a run — which is the whole distance from $4.83 to $3.75. The saving is large because the case count multiplies it, and it costs nothing measurable because the rubric never referred to the context that left: the discordance rate, meaning the fraction of cases whose pass or fail verdict flips between two runs of the same unchanged commit, stayed where it was.

A provider's batch API, which accepts a 24-hour turnaround for a 50% discount, applies cleanly to a weekly run and not at all to a merge-triggered one, since a 24-hour result is a nightly by another name.

The multiplier on the outside#

Per-run savings are bounded by the run's content. Cadence is unbounded, which is why it dominates. The same 400-case suite at $4.83 costs anywhere from $21 to $2,946 a month depending only on the trigger:

Every row is the same multiplication with only one term changed. The runs-per-month column is a count of events rather than an estimate, and each entry has an obvious source. Nightly is 30 because that is how many nights a month has. Weekly is 4.3, the same 4.33 weeks a month that turned thirty merges into 130. Per release is the release rate multiplied by those weeks, and every merge to main is the merge rate multiplied by them. Every pull request lands on the same 130 as every merge, because a pull request that gets merged is a merge. Every push multiplies that figure again by the number of pushes a pull request receives before it merges, which the arithmetic two chapters ago measured at a little under five. Multiplying any of those counts by $4.83 gives the spend column: 130 runs is $628, and 4.3 runs is $21.

What a slower trigger buys is that spend column. What it costs is written in the last one, and the last column understates it, because a detection time is also a window and the window has merges in it. The same 130 merges a month split across 30 nightly runs puts a bit over four merges inside each window; split across 4.3 weekly runs it puts thirty of them there. A red merge-triggered run names a commit. A red nightly run names a night, and somebody has to work out which of four changes did it. A red weekly run names a week, which is thirty commits with no ordering information attached, and at that point the eval has stopped saying what broke and is only saying that something did.

The second half of the cost is exposure, and it compounds the same way. A regression caught at merge reaches nobody. One caught weekly has been live for up to seven days, and the support contacts a live regression produces arrive at a rate rather than as a one-off penalty. Lengthening the detection window does not add a constant to the damage; it multiplies that rate by however long the window is. Neither half of that cost appears in the spend column, which is what makes a slower trigger look free on the day it is chosen: the money it saves lands immediately and in one place, and the bisecting and the customer contacts it buys land later and on other people.

Halving the case set from 400 to 200 moves every row down by half and costs resolution — the smallest drop the gate can tell apart from its own noise goes from 4.4 points to 6.3, which is above a 5-point regression a team might have decided they would refuse to ship. Changing the trigger from every-merge to merges-that-touch-the-relevant-paths moves the same row from $628 to $251 and costs nothing at all in resolution. What it costs is coverage of a specific failure mode: a regression introduced by a change to code the path filter did not consider relevant. That is a real cost, and it is an enumerable one, which makes it a much better thing to trade than statistical power.

Money is not the only budget#

A second resource carries the same multiplier and fails first in practice. A 25-minute run at 130 runs a month is 54 hours of wall clock; serialised through the single concurrency group that keeps each merge's number attributable, that is 12.5 hours a week, which fits inside a week only because merges are spread out. They are not spread out. A team of eight merges in bursts, and six merges landing between 14:00 and 16:00 puts the sixth run's result at 16:30 for a change made at 15:10, by which time three more merges have landed on top of it and the paired comparison has drifted from the thing it was measuring.

Each of those figures is worth one line, because each is easy to round away in a planning document. The 54 hours is 130 runs of 25 minutes, a little over three thousand minutes, converted at sixty minutes to the hour. The 12.5 hours a week is that divided by the same 4.33 weeks. The 16:30 is the same division done at a granularity no monthly average can see: six runs of 25 minutes is two and a half hours of queue, so a merge landing at 15:10 behind five others waits out every one of them before its own run begins. A bill is fungible across a month and a queue is not, which is the whole of the difference between the two budgets.

Provider rate limits bite at the same point. A 400-case run at three concurrent requests finishes in 25 minutes; at a tier-2 limit of 500 requests per minute the arithmetic allows far more concurrency than that, but the judge model usually sits on a tighter limit, and two eval runs overlapping start returning 429s that the retry ladder absorbs into a longer run rather than a failure. A run that took 25 minutes on a quiet morning takes 47 on a busy afternoon, and nothing in the score reports that it happened.

Counting the requests is what shows which limit is actually binding. A case is not one request but two — a generation call and then a judge call on what it produced — so a 400-case run issues eight hundred requests, and eight hundred requests spread over 25 minutes is around thirty a minute. That is far below 500 a minute, which is why the tier number in the provider's documentation is the wrong one to plan against. The limit that binds is whichever model in the run has the tightest one, and that is normally the judge, because the strong models teams pick for judging are the ones providers ration hardest.

The token cost of a confirmation rerun is also not its full cost. Each rerun occupies the same queue and the same rate limit, so a noisy gate consumes wall clock at twice the rate it consumes money. Budgeting cadence therefore means budgeting two quantities that scale together, and the one that produces complaints is time.

Where caching helps and where it lies#

Caching sits awkwardly across those tiers and is worth placing deliberately rather than switching on globally. Keying generations on a hash of the rendered prompt plus the model id and temperature makes a rerun of an unchanged commit nearly free, which is exactly what a confirmation run wants, and it is why re-running to confirm a red result costs the judge portion only.

It is the wrong default for a merge detector. A change that alters the prompt produces a cache miss on every case, which is correct; a change that alters retrieval ranking produces different chunks and therefore also a miss. So the cache saves nothing on the merges the detector exists for and saves plenty on the ones it does not care about. A release gate should run uncached, because the point of that tier is to measure what will actually be served, and a cached generation from three days ago is not that.

The path filter deserves its own scrutiny, given how those tend to go wrong. A filter added to save money is the same mechanism that silently removes a third of merges from measurement, and the difference between the two is entirely whether something else covers the gap. Here the list is short, versioned next to the workflow, and reviewed whenever a new module joins the request path, and the release gate exists precisely so that a merge the filter missed is still measured before it can reach anyone. A filter with nothing behind it is the paths-ignore failure again, wearing a budget justification.

The line that does not move#

If the budget were halved to $100, the tier to keep is the release gate at full resolution. It is the only check positioned between a regression and a customer, and the only one whose number is accurate enough to be worth blocking on. Dropping to $100 means deleting the merge detector and the weekly run, accepting that regressions get named at release time rather than at merge time, and accepting that drift goes unmonitored — both of which are losses of speed and awareness rather than losses of protection.

The version that must not happen under budget pressure is keeping all three tiers by shrinking every set until the total fits. That preserves the shape of the process and destroys every number in it: three jobs running, three scores posted, none of them able to resolve anything the team cares about, and everyone reassured that quality is checked in three places. What a shrink to fit takes is the readability of every number the three jobs post: a score is readable only when somebody can say what a movement in it would mean, and once every set has been shrunk to make a total work, nobody can say that about any of the three. Cutting a tier is visible and can be argued about in a planning meeting. Cutting resolution is invisible and gets discovered during an outage.

The ordering that falls out of all this is worth carrying into any budget conversation. Change the trigger first, because it is the term with a hundredfold range and no effect on what a run can see. Trim the judge prompt second, because the judge is most of the bill and rubric-irrelevant context is usually sitting in it. Batch anything that can tolerate a day's delay. Reduce draws per case before reducing cases, since draws buy noise reduction that a larger set also buys. And treat case count as the last lever, because it is the only one whose cost does not appear anywhere in the run's output — the job still runs, still posts a number, still refuses promotion below the threshold, and quietly cannot see the thing it was built for.

What no budget buys at any size is a different kind of visibility. Every tier above reports an aggregate over a sample, and there is a class of defect that an aggregate over a sample is structurally unable to show, however many cases are bought.