Costing a judge harness

Evaluation spend is an inference bill with the same structure as any other: calls, tokens per call, and price per token, multiplied by how often the thing runs. The difference is that nobody watches it. Production traffic has a dashboard and a capacity plan; a judge harness has a CI configuration written by someone who was thinking about unit tests, and the multiplier hiding in that configuration is usually pushes rather than merges.

What follows turns a judge harness into a costed line item and then works through four levers that change the shape of the bill. None of them measures less: the reductions come from removing duplicated work, cached work, and inference spent on questions a substring test could answer.

The arithmetic nobody wrote down#

Five criteria, each its own call, over 400 golden examples, is 2,000 judge calls per suite run. Each call carries a system preamble and one criterion at about 300 tokens, the retrieved context at about 1,200, and the answer under test at about 400, so roughly 1,900 input tokens. Reasoning before the verdict produces about 190 output tokens. At $3.00 per million input and $15.00 per million output:

CALLS_PER_RUN = 400 * 5
IN_TOK, OUT_TOK = 1_900, 190
IN_PRICE, OUT_PRICE = 3.00 / 1e6, 15.00 / 1e6

run_cost = CALLS_PER_RUN * (IN_TOK * IN_PRICE + OUT_TOK * OUT_PRICE)
print(f"${run_cost:,.2f} per suite run")          # $17.10 per suite run

PUSHES_PER_PR, PRS_PER_WEEK = 7, 25   # CI runs on every push, not every merge
weekly = run_cost * PUSHES_PER_PR * PRS_PER_WEEK
print(f"${weekly:,.0f}/week  ${weekly * 52:,.0f}/year")  # $2,993/week  $155,610/year

Every step of that is worth doing on paper once, because the surprise is in the multiplication rather than in any single figure. One judge call carries about 1,900 input tokens, and at $3.00 per million input tokens the input side of a call is 1,900 divided by a million, times three dollars — a bit over half a cent. The output side is 190 tokens at $15.00 per million, which is a bit under three tenths of a cent. Adding them puts a single call a hair under nine tenths of a cent. That number is unalarming and it is the whole problem: nothing about a sub-penny unit cost invites anyone to check what it is multiplied by.

What it is multiplied by is three things in sequence. Five criteria as separate calls times 400 examples is 2,000 calls, so one suite run is 2,000 times nine tenths of a cent, which is the $17.10. Seven pushes per pull request times 25 pull requests a week is a hundred and seventy-five suite runs a week, so the week is that many times $17.10, which is the $2,993. Fifty-two of those weeks is the $155,610. No step there is larger than a factor of a few hundred, and three of them compose into five orders of magnitude.

Seventeen dollars a run is why nobody costed it. The suite running on every push rather than every merge is what turns seventeen dollars into $156,000 a year. Set that against production: an assistant serving about 40,000 answers a month at roughly $0.004 each is $160 a month, which is a little under two thousand dollars a year, so evaluation is costing roughly eighty times what the product it evaluates costs to run.

That ratio is not automatically wrong. An early-stage system that is being changed constantly and whose failures are expensive can rationally spend more on measurement than on serving. What is wrong is spending it without knowing, since a cost nobody has attributed is a cost nobody can trade against anything.

What the money is actually buying per call#

Before cutting anything, it is worth seeing where the tokens go, because the biggest line is not the one people reach for.

Two of those rows deserve a second look. The context is 1,200 of the 1,900 input tokens, which is where the 63% comes from, and the 190 output tokens are only a tenth of the token count and a third of the money — because output is billed at five times the input rate, so 190 tokens of reasoning cost what five times as many prompt tokens would. Sorting by token count and sorting by cost give different answers, and only the second one is the invoice.

The context is repeated on all five calls for the same example, which is 4,800 tokens of duplication per example — four extra copies of 1,200 — or 1.9 million per suite run across 400 examples. Provider prompt caching, where the provider keeps the processed form of a prompt prefix it has already handled and bills later calls that reuse it at a fraction of the usual rate, addresses part of this when the repeated span is a shared prefix, at a typical discount of 90% on cached input tokens.

The word "prefix" is the constraint that decides whether any of this is available, and it follows from what the provider is actually storing. A model processes a prompt left to right, and the internal state it builds at each position depends on every token before that position and none after it. So the work done on positions one through n can be kept and reused only by a later request whose first n tokens are byte-identical; the first token that differs invalidates that position and everything downstream of it. A prompt with the criterion text at the top and the context underneath therefore shares nothing at all between the five calls, even though 1,200 of the 1,900 tokens are the same text, because the difference occurs at position one. Move the criterion below the context and the five calls share a long identical opening.

Reordering the prompt to put the volatile part last and enabling caching took this suite's per-run cost from $17.10 to $8.40, which is the single cheapest change available and involves no reduction in what is measured. The size of that drop follows from the split above: the shared prefix is most of the input, four of every five calls read it at a tenth of the price, and the output tokens are untouched — so the input side of the bill nearly disappears and the output side, which was already a third of it, becomes most of what is left.

Four levers, and what each one gives up#

The first lever is to run deterministic checks before spending anything on inference. A format check is a regular expression. A citation-span check, once the criterion requires a verbatim span, is a substring test against both the answer and the context. Neither should ever have been a judge call, and removing them takes 2,000 calls to 1,200. More significantly, an example that fails a deterministic check is already a failure, so judging its remaining criteria produces detail nobody will read before the obvious defect is fixed. Short-circuiting those examples removed a further 14% of calls on this suite. What you give up is the judge's rationale on the short-circuited examples, which occasionally matters when a format failure and a grounding failure have a common cause.

The second lever is caching by content. Most pull requests do not change the assistant's outputs at all — a change to the billing service does not alter what a support assistant answers — and a verdict is a pure function of the example, the answer, the criterion text and the judge model version. Key on a hash of those four and the entire suite becomes free for any push that leaves generation unchanged:

import hashlib, json

def verdict_key(example_id: str, answer: str, criterion: str) -> str:
    payload = json.dumps({
        "example": example_id, "answer": answer,
        "criterion": criterion, "judge": JUDGE_MODEL_VERSION,
        "schema": SCHEMA_VERSION,
    }, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

On this repository, 61% of pushes changed nothing that fed the assistant, so the cache hit rate over a week was 58% and the weekly bill fell accordingly. Follow that through the call count and the table below stops being a list of assertions: the deterministic-checks tier was running about 180,000 calls a week, and with 58% of them served from cache the fraction still to be paid for is a little over two fifths, which is the 76,000 in the next row.

The reason a hit is safe here — the reason serving a stored verdict is not cheating — is that the four things in the key are the only inputs the judge receives. A verdict is a function of the example, the answer, the criterion text and the judge model version, and if all four are unchanged then re-running the call is asking a question that has already been answered by the same instrument. The failure mode to guard against follows directly from that argument: leaving JUDGE_MODEL_VERSION or SCHEMA_VERSION out of the hash means the key no longer covers every input, so a judge upgrade silently serves months-old verdicts. That is worse than no cache, because the dashboard keeps reporting and nothing about the numbers looks stale.

The third lever is sampling. Running a stratified 80-example subset on each push and the full 400 nightly cuts per-push cost by a factor of five, and it costs detection power in a way that is worth quantifying rather than hand-waving. At a pass rate near 85%, the standard error of a proportion is 4.0 points at n=80 and 1.8 points at n=400, so comparing two runs, the smallest regression a subset run can distinguish from noise is around 11 points, against roughly 5 points for the full suite.

Both of those figures come out of one calculation applied twice, and the second half of it is the part usually skipped. The standard error of a proportion is the pass rate times the fail rate, divided by the sample size, square-rooted: at 85% passing that is 85 hundredths times 15 hundredths, a bit over a tenth, and dividing by 80 and taking the root gives the 4.0 points. The same numerator over 400 gives the 1.8. But a gate compares two runs, and each of them carries that error, so the quantity that has to be cleared is the error on a difference rather than on a rate. Two independent errors combine by adding their squares and taking the root, which for two errors of the same size means multiplying by the square root of two — so a difference between two 80-example runs wobbles by a shade under six points, and the roughly two standard errors conventionally required before anyone calls it a finding is the 11. Run the same two steps on 1.8 and the full suite's threshold comes out near 5.

A regression that breaks 8% of answers will therefore usually pass the per-push gate and will be caught by the nightly run, which means the team finds out the next morning rather than in review. Stratification is what keeps the subset from being worse than that: sample proportionally across question categories and over-sample the examples that have failed in the last 30 days, so the rare-but-known failure modes are always represented.

The fourth lever is a cheaper judge for the per-push tier, calibrated against the expensive one on a fixed slice. A small model at roughly a twelfth of the price came out at kappa 0.71 against human labels where the large judge scored 0.88, which is not good enough for the number on the dashboard and is often good enough for a smoke test whose job is to catch a 20-point collapse. The discipline this requires is re-calibration: the cheap judge's agreement with the expensive one has to be re-measured whenever either model version changes, and a team that calibrates once and then treats the cheap judge as equivalent has built a gate whose behaviour drifts silently.

The last row is a 98% reduction against the starting point, and the honest description of it is that the per-push check has become a smoke test while the measurement anyone quotes now happens nightly.

Wall-clock time follows the same arithmetic and usually forces the decision before money does. Two thousand calls at a p50 of 1.6 seconds is 53 minutes if the harness is sequential, since two thousand calls at a second and six tenths each comes to a little under an hour of waiting, which no engineer will tolerate on a pull request. Dividing that across 32 concurrent requests brings it to about 100 seconds, subject to whatever requests-per-minute ceiling the account has. The rate limit is the real constraint on most teams, and it is the reason the per-push tier ends up small regardless of budget. Deciding the split between push and nightly is therefore a question with a concrete form: what is the largest subset that finishes inside the time developers will wait, and does that subset have enough examples to resolve a regression you would actually block a merge for? If the answer to the second half is no, the per-push tier is a smoke test and should be described as one rather than dressed up as a gate.

Cutting the suite is not the same saving#

None of these levers is an argument for evaluating less. The suite that costs $63 a week measures the same five criteria over the same 400 examples every night. A team that responds to the invoice by cutting the suite from 400 examples to 120 has bought a similar saving by making every number noisier, and the standard error arithmetic above says what that costs — at n=120 the resolution is roughly half what it was, permanently, on the number that decides releases.

The distinction between the two kinds of saving is what makes the earlier levers defensible and this one not. Prompt caching, deterministic checks and the content cache all remove work that was producing a result the harness already had or could get another way, so the verdicts they replace are the verdicts they would have computed. Cutting the golden set removes examples, and an example that is not in the suite has no verdict at all. The first three change the cost of the measurement; the last changes the measurement. The subset-per-push lever sits between them, which is why it is the one with an explicit trade written next to it: the examples are still measured, just not on the tier where a developer is waiting.

Retries deserve a line in the model too. A judge call that returns malformed JSON, hits a content filter or times out gets retried, and a 3% failure rate with one retry adds 3% to the bill quietly. The version that is not quiet is a transient provider degradation during which timeouts climb to 30% and a naive retry loop with no backoff, meaning it fires again immediately rather than waiting a growing interval between attempts, triples the call count for an hour. Capping retries per run and emitting the retry count as a metric turns that from an invoice surprise into an alert.

Judge spend also needs to appear somewhere a human looks at. Tracking dollars per suite run, dollars per merged pull request, and judge tokens per week on the same dashboard as the pass rate is what keeps the next 40% of growth attributable. Each of the things that produce such growth — retries after parse failures, a criterion someone added, a context window that grew when retrieval started returning more passages — is a normal engineering decision, and each of them multiplies a per-call cost by a call count nobody is watching.

An evaluation harness, then, is a system with a unit cost and a call rate, and it is the only such system in most codebases that nobody has ever multiplied out. What the multiplication produces is not primarily a saving — it is the ability to say what a measurement costs, which is the precondition for trading detection sensitivity against money deliberately instead of discovering the trade in an invoice review three months later.

The harness described across these chapters is now specified: binary criteria, one per call, reasoning first, a primary judge outside the generator's family, a calibration slice, and a budget. The harder situation is the one you are more likely to walk into, which is a harness that violates every one of these rules simultaneously and is gating releases while it does so.