Tiering the eval suite

Every eval suite has two jobs and one schedule, which is the problem. One job is to tell a developer, within the span of their attention, that the change they just pushed has not broken something obvious. The other is to establish, before anything reaches production, that quality on the full case set is where it should be. Those have different latency requirements, different accuracy requirements, and therefore different budgets — and a suite that runs whole on every push is paying the second job's price for the first job's question.

The fast job belongs to a category that already has a name in ordinary testing, the smoke test, borrowed from the hardware practice of powering a new board up and watching whether it emits smoke: it establishes only that the thing is not obviously broken, which is a much weaker claim than the full suite makes and a much cheaper one to check. This chapter is about splitting an eval suite along that line — what goes in each tier, how the fast subset is chosen and validated, and how loose its threshold has to be given what the previous chapters established about noise.

The bill and the wait, in numbers#

Take a suite of 800 cases running end to end through a real pipeline — retrieval, generation, tool calls, and an LLM judge on every case — at twenty-two minutes of wall clock and about $14 in API spend per run. What a suite costs a team is not that per-run figure but that figure multiplied by how often the run is triggered, and a per-push gate is triggered by pushes rather than by pull requests. Fifteen pull requests a day at 2.7 pushes each is forty and a half pushes, which is the 40 runs a day quoted in the table below. Forty runs at fourteen dollars apiece is five hundred and sixty dollars, so $560 a day; and because pushes happen on working days, a month of about twenty-two of those at that rate is the roughly $12,300 that eventually becomes a line item somebody queries.

The wall clock matters more, and it lands on a different party than the bill does. A developer who pushes a fix and waits for the gate before pushing the next one has 22 minutes injected into each turn of their loop, and three turns is more than an hour of a change's life spent waiting on a check that will almost always pass. The wait does not merely delay the next push, either. It also costs the developer the reload of everything they had in their head about the change.

Neither number causes avoidance on its own, and the step from a cost to a behaviour is where the chapter's argument actually sits. The money lands on a team budget and the hour lands on one person, while the benefit of having run the check — a regression caught before release — is spread across everybody who ships that week. The arithmetic an individual does is therefore not the arithmetic the team would do: for the developer, an hour of their own day buys a small share of a diffuse good. Avoiding the check is locally rational under that split, and it is rational for a conscientious person rather than only for a careless one. So any mechanism that permits avoidance will eventually be found. Nobody has to set out to defeat the gate for that to happen: the search for a faster loop runs continuously across a whole team, and it terminates wherever the cheapest route happens to be.

That is why the incentive rather than the seam is the thing to fix. Closing a discovered seam by policy does work in the narrow sense — nobody uses that particular route again — but it has changed which routes exist without changing what makes a route attractive, so the search resumes and terminates somewhere else. The relocations available are worse for the codebase than what the policy banned. People will iterate on a branch without opening a pull request at all, which takes the change out of review as well as out of the gate. Or they batch a week of work into one enormous change, which fires the gate exactly once, against a diff too large for anyone to read usefully and too large to bisect when it goes red. The property a gate needs, and the one this gate has lost, is that running it must be cheaper than avoiding it — and that is a property of the ratio between two costs, which a policy cannot move and a cheaper check can.

This is also where the re-run problem from the previous chapter becomes expensive rather than merely dishonest. On a 22-minute, $14 suite, a single re-run costs the team $14 and costs the developer most of an hour of context, so the pressure is not toward re-running until green but toward not triggering the check at all. Flakiness and cost compound: a check that is both unreliable and slow teaches people that the correct response to a red eval is to find a way for it not to have run. By the time that norm exists, tightening the threshold — the move already argued against on statistical grounds — is actively counterproductive, because every additional failure now carries a 22-minute penalty and pushes more work out of sight.

Split the suite by what each run is for#

Daily spend goes from $560 to about $200, and the number that matters more is that the developer's loop now closes in a little over two minutes instead of twenty-two. The full suite still runs before anything is deployed, because merging to main is upstream of release, so nothing has been given up on the release path. What has changed is where in the process the expensive measurement sits.

Both of those numbers come straight off the table, and both are worth doing rather than believing. The day's bill is each tier's cost per run multiplied by the number of times that tier runs, summed over the four rows. The assertions tier is ninety cents on each of forty pushes, which is thirty-six dollars. The smoke tier is fifty-five cents on the same forty pushes, which is twenty-two dollars. The full suite is fourteen dollars on each of eight merges, which is a hundred and twelve dollars. The nightly is thirty-one dollars, once. Thirty-six and twenty-two make fifty-eight; adding a hundred and twelve takes the running total to a hundred and seventy; adding the nightly's thirty-one lands the day at a little over two hundred dollars.

The developer's two minutes is the other two rows read the same way. Only the top two tiers fire on a push, so what a developer waits for is forty seconds of assertions and ninety-five seconds of smoke, which is a little over two minutes together — and what the team pays for that push is ninety cents plus fifty-five cents, the $1.45 the worked case below quotes. The full suite's twenty-two minutes and fourteen dollars have not become cheaper by any amount. They have moved off the path where somebody is sitting still and waiting for them.

That arithmetic is written out because the chapter's own rows have already caught an error in it. An earlier draft of this chapter put the new daily spend at $175, and the four products above do not add to that under any reading of the table. Where a stated figure and the arithmetic that should produce it disagree, the arithmetic is the thing to trust, because it is the part a reader can check: four costs, four counts, four products, one sum. The figure is $200, and it is shown as working rather than swapped in quietly because a reader who can reproduce a total can also notice the next time it drifts.

The assertions tier deserves the top row it has, and the gap between its row and the full suite's says why. Both put all 800 inputs through the same pipeline; the only difference is that the full suite then sends every response to an LLM judge to be scored. Ninety cents against fourteen dollars for the same work plus judging means the judging is roughly fifteen of every sixteen dollars the full suite spends, so removing it is not a marginal saving but very nearly the whole bill. What survives that removal is still a real execution of the product: the request is made, retrieval runs, the tool is called, and a response comes back to be inspected. Running 800 inputs through the pipeline without a judge is cheap because the judge is most of the cost, and the properties it checks — the response parses against the schema, every cited document ID exists, the tool name is real, no email address or card number appears in the output, the response is non-empty — are the ones whose failures are unambiguous and whose fixes are obvious. Each can be decided by looking at the response itself, without anyone forming an opinion about how good the answer was, which is why no judge is needed for them. The case for these on variance grounds was made in the chapter on re-runs; on cost grounds they are better still, because they catch the class of break that a smoke test of 60 judged cases might miss by sampling.

Choosing the sixty, and proving they were the right sixty#

The tempting way to build the smoke tier is to sample 60 cases at random on each run. It is one line of code, it feels statistically respectable, and a uniform sample genuinely is the right instrument for estimating an average. It is the wrong instrument here, and the reason has nothing to do with statistics.

Follow what a fresh draw does to the property the first chapter of this topic established. That chapter bound an eval score to a commit by fixing every input to the measurement — the case list, the model version, the temperature, the prompts — so that the resulting number was a fact about the commit rather than about the afternoon it was measured on. A subset drawn afresh on each run is an input that changes while the commit does not. Two runs on the same SHA therefore score different cases and can disagree without anything about the code having changed, and a developer looking at a red check cannot tell whether the code broke something or the draw happened to include a case the code was always weak on. A check that cannot separate those two readings has stopped identifying anything about the code, which is the entire job of a gate. So the subset has to be fixed, committed to the repository, and changed by pull request like any other configuration — and the point of putting it under review is that changing the case list changes what the gate means, which is precisely the kind of change that should not be able to happen quietly.

Selection has three inputs. Coverage comes first, and it is what makes the sixty stratified rather than sampled, in the sense that the subset is filled to a quota per category instead of drawn uniformly: every route, intent, and tool the product exposes should appear at least once, because the failures a smoke test is for are the ones that take out a whole path. Discriminative history comes second, meaning cases that have actually changed verdict when something real broke, which you can read off the stored per-case results in the artefacts from every previous run. Stability comes third: the cases that flip verdict on an unchanged commit are excluded outright, since a 60-case suite cannot afford a case that flips on its own.

The first of those is the one that gets argued about, so it is worth seeing the mechanism that decides it. A uniform draw allocates slots in proportion to how common each route is in the full 800, which is the correct behaviour when the goal is an unbiased estimate of overall quality. It is the wrong behaviour when the goal is to notice that a route has stopped working. A route holding a small fraction of the suite stands a good chance of contributing no case at all to any particular draw of sixty, and the failure that route would have exposed is exactly the kind the tier exists for: not a two-point slide in an average but a path that returns nothing, or returns the wrong shape, for everyone who takes it. Filling a quota per category is what guarantees each path has a witness in the subset, whatever its share of the traffic. The trade is real and worth saying out loud: a stratified sixty is a worse estimate of the suite's overall score than a random sixty would be, because it over-represents rare routes deliberately. The tier is not an estimator, it is a detector, and what a detector needs is coverage rather than proportionality.

The part teams skip is validating the result, and it is the part that turns the subset from an opinion into a measurement. Everything in the three inputs above is judgement: somebody decided which distinctions count as categories, and somebody decided that a case with a history of catching things has earned a slot. A subset chosen that way is a hypothesis — that these sixty are the cases which would have caught what actually went wrong — and a hypothesis about the past can be tested against the past. You have a history of real regressions — reverted commits, incident-linked fixes, pull requests whose description says the word "regression" — and each of them comes with the commit that introduced it and the commit immediately before. Running the subset against both and asking whether its verdict changed is what converts the hypothesis into a number.

# evals/validate_subset.py — would these 60 cases have caught the last year's regressions?
import subprocess, json
from evals.run import run_suite

SUBSET = json.load(open("evals/smoke_subset.json"))["case_ids"]   # 60 fixed IDs

# commits known to have introduced a regression, and their parent
KNOWN_REGRESSIONS = [
    ("2f9a1c4", "8b3e07d", "retrieval top-k dropped to 1"),
    ("c81d5e2", "44a9f10", "system prompt lost the citation instruction"),
    # ... 10 more, from incident tickets and reverts
]

def caught(bad: str, good: str) -> bool:
    before = run_suite(commit=good, case_ids=SUBSET)
    after  = run_suite(commit=bad,  case_ids=SUBSET)
    regressed = [c for c in SUBSET if before["per_case"][c] and not after["per_case"][c]]
    return len(regressed) >= 3          # the smoke tier's blocking rule

hits = [(b, note, caught(b, g)) for b, g, note in KNOWN_REGRESSIONS]
for sha, note, ok in hits:
    print(f"{'CAUGHT ' if ok else 'MISSED '} {sha}  {note}")
print(f"\n{sum(ok for _, _, ok in hits)}/{len(hits)} caught by the 60-case subset")

That run came back 9 out of 12, and the three misses are more informative than the nine catches, because they are not all the same kind of miss. Two of them were small quality drifts of 2 to 3 points that no 60-case suite could see. A drift that size sits below what sixty cases can resolve — the paragraphs below put a figure on why — so those two are the smoke tier working as designed, and the correct response to them is nothing at all. Making the tier catch them would mean either growing it back toward the full suite or tightening a threshold that the tier's own noise would then trip on its own, and each of those undoes the reason the tier exists.

The third miss was a different animal. A regression in a document-summarisation path went undetected because the subset contained no case that exercised that path at all, so there was no measurement available to fall below any threshold. That is a coverage gap rather than a sensitivity limit, and it has a cheap fix the other two do not: swap a case in, which is what was done. The distinction matters because the two kinds of miss are indistinguishable in the validation output — both of them simply print MISSED — and only one of them is a defect in the subset. Re-run the validation whenever the subset changes and it stops being a judgement call. One condition on reading 9 out of 12 honestly: it scores the subset against regressions somebody recorded as regressions, so a class of breakage nobody ever noticed cannot appear in that denominator, and the true catch rate is an upper bound rather than an estimate.

Because the smoke tier is small, its threshold has to be loose, and the relationship between case count and spread says how loose. That relationship is worth deriving rather than importing, because the figure it produces decides where the gate sits. A suite's score is an average over its cases, and each case contributes a verdict that can flip between runs on identical code — on the suite these chapters have been measuring, about six cases in every hundred do. The spread of an average shrinks as the square root of how many things are being averaged, which is the ordinary behaviour of an average and the reason a larger suite is a quieter one. Read backwards, it says how much noisier a smaller suite is. Cutting the case count from the hundred and twenty of the previous chapters down to sixty is halving it, and halving the count of an average widens its spread by the square root of two, which is a bit over two-fifths more. A standard deviation of two point two points on a hundred and twenty cases, widened by that factor, is about three point one points on sixty.

Two conditions make that step legitimate and both are worth carrying forward. The square-root relationship assumes the case flips are independent of one another, and they are not exactly independent: a broken retrieval step flips a whole family of related cases together, which makes the real spread wider than the formula says. It also assumes the per-case flip rate on the sixty matches the rate on the full suite, and a subset chosen for stability should if anything flip less. It is a good enough approximation to place a threshold with and a poor one to argue small differences over. At 60 cases the standard deviation of the score is around 3.1 points, so an absolute gate placed for a 1% false-block rate — which the previous chapters put at about two and a third standard deviations below the mean — sits about 7 points under it, since three point one points taken two and a third times is a little over seven. That is the honest limit of the tier: it will not see a 3-point regression, and it is not supposed to. It blocks on an absolute floor well below the full suite's, on any deterministic assertion failing, and on three or more paired case regressions against the same 60 cases run on the base commit — a count of named cases rather than a delta, which is both tighter and easier to act on.

The reason a per-push check can be this loose is that it is not the last line. The full suite runs on merge to main before anything is released, so a 3-point drift the smoke tier missed is caught within the hour by a run that has 800 cases and a properly tight threshold. What the smoke tier buys in exchange for that looseness is that developers find the obvious breakage in 95 seconds instead of at the end of a 22-minute wait they were incentivised to skip.

The pipeline mechanics that make the saving real#

Three configuration details do most of the work, and all three are ordinary CI practice that eval workflows tend not to inherit.

name: eval-smoke
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]   # drafts too
    paths:
      - "src/**"
      - "prompts/**"
      - "evals/**"
      - "!**/*.md"

concurrency:
  group: eval-smoke-${{ github.event.pull_request.number }}
  cancel-in-progress: true      # a new push kills the run for the previous one

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 6
    strategy:
      matrix:
        shard: [0, 1, 2, 3]     # 60 cases across 4 shards, ~25 s each
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements-eval.txt
      - run: |
          python -m evals.run \
            --subset evals/smoke_subset.json \
            --shard ${{ matrix.shard }} --shards 4 \
            --baseline-ref ${{ github.event.pull_request.base.sha }} \
            --max-regressed-cases 3 \
            --fail-under 0.78

Including draft pull requests in the trigger is deliberate, and it is only defensible because the check now costs 95 seconds and half a dollar. The draft is where the iteration was happening precisely because the gate did not reach it, so widening the trigger while the check was still expensive would have removed the escape route without removing the reason anybody wanted one. The paths filter means a documentation change does not spend anything, which is the same principle turned on a different kind of run: a change that cannot alter the answer should not have to pay to be told so.

Concurrency cancellation matters more than it looks, and what it protects is correctness rather than money. A new push supersedes the one before it, which means the code the earlier run is measuring no longer exists anywhere on the branch. Without cancellation that earlier run carries on regardless and finishes at its own pace, so a developer pushing three times in five minutes pays for three overlapping runs whose completion order has nothing to do with the order the pushes went out in. They read whichever finishes last, which is to say they may read a verdict about code they replaced two pushes ago — green on a commit that was never green, or red on a bug that is already fixed. The wasted spend is the visible cost and the smaller one. With cancel-in-progress the earlier runs are killed the moment they are superseded, so the only run left able to report is the one measuring the code that is actually there.

Sharding splits the fixed case list across four runners so that each scores a quarter of it and the results are combined at the end, and it turns 95 seconds of serial work into 25 seconds of parallel work at the same cost. The reason it is free is a property of what these runs spend their time doing. A run of the suite is almost entirely waiting on API calls: the request goes to a model provider and the process sits idle until a response comes back, using no processor time of its own. Waiting is not work, so four processes can wait at once for no more effort than one process waiting alone, and four runners each handling a quarter of the list finish in roughly a quarter of the time. What gets billed is the number of API calls and the tokens in them, and that is unchanged: the same sixty requests are issued, by four processes instead of one. The CI machine time comes out about level too, since four runners for a quarter as long is about the same total. The only quantity that moves is wall clock, which is the one quantity the developer experiences.

Where tiering differs from deleting cases#

Tiering is not the same thing as cutting the suite down, and it is worth being clear about that when it is proposed, because the two look identical in a cost summary and differ completely in what they give up. Eight hundred cases is coverage accumulated one incident at a time, and the cheapest way to make the bill disappear is to throw most of it away. A team that responds to the bill by deleting 740 cases has bought the same cost saving and given up the coverage, and will discover which 740 mattered over the following two quarters. Under tiering, every one of the 800 cases still runs, with the same threshold and the same rigour, on every merge to main and every night. What moved is when the expensive measurement happens relative to a developer's attention, not how much of it happens.

The property that decides between the two is the one the first chapter of this topic argued for: a quality claim is only checkable if the score is bound to the exact commit and inputs that produced it, and stored where somebody can go back and look. The distinction shows up in the artefact record: under tiering, every commit that reaches main has a full 800-case result bound to it, which is what the provenance argument requires and what the release process can point at. The per-push tiers are extra measurements taken earlier, not substitutes for that one. Under deletion the binding survives and the thing it binds does not. Every commit still has a result attached, and that result is now a 60-case measurement, so no commit anywhere in the history carries an 800-case number behind it and there is nothing to point at when somebody asks what the coverage actually was.

The sentence a reader is most likely to write after all this is that the smoke tier is a weaker version of the full suite, and it is tempting for a decent reason: the two run the same harness over cases drawn from the same list, and one of them has fewer cases and a looser threshold. What is true instead is that they answer different questions. The full suite asks whether quality across the whole case set is where it should be, and its threshold is placed to make that answer trustworthy. The smoke tier asks whether anything is obviously broken, and its threshold is placed so that noise almost never answers yes. A weaker measurement of the first question would simply be a bad one; the tier is a sound measurement of the second. Its looseness carries a condition, and the condition is load-bearing: the tier is only allowed to be this insensitive because the full suite still stands between main and release. Put the tier on the release path and every criticism of it becomes fair at once.

There is a cost interaction with the previous chapter worth making explicit. Gating on the mean of three runs, which was the honest answer to the re-run habit, means paying for the suite three times over: fourteen dollars taken three times is forty-two, so it costs $42 and an hour on the full suite and is unaffordable per pull request. On the smoke tier the same three runs are fifty-five cents taken three times, so it costs $1.65 and about three minutes, which is affordable, and it is not needed there because the smoke tier's threshold is already loose enough to absorb the variance that averaging was invented to remove. Repeated runs belong on the nightly tier, where a low-variance number feeds the threshold ratchet and the trend line, and where nobody is waiting.

The rule the whole arrangement follows is that a gate people route around has a quality of zero regardless of how good its measurement is, so the per-push check is designed around the developer's attention rather than around the suite's completeness. Sixty stratified cases validated against real regressions, plus assertions that cost forty seconds, catch the obvious breakage in the time it takes to read a diff; the full 800 keep their tight threshold and their claim on the release path, one merge later.

Every tier in that table scores one thing, which is answer quality. The next chapter is about a change that improves that number while making the product materially worse, and passes every gate described so far.