Tests vs evals

Read these first

• Why traditional testing fails
• Testing code that calls an LLM

What you will know

• Draw the line between a test and an eval in your own codebase

  A flaky model judgment or human-rated outcome cannot provide the repeatability expected from a commit-level assertion.

• Wire deterministic tests into every commit and sampled evals into a separate gate

  Keeping the paths separate preserves fast feedback while still making quality failures release-visible.

• Reproduce a slow or costly eval gate and patch the pipeline so its failures remain visible without blocking fast commits

  A skipped job is operationally equivalent to no protection unless ownership, alerts, and a later gate remain intact.

• Measure one eval run's latency and spend, then set a dataset and model budget for its gate

  API cost scales with examples, retries, and token volume, so an unbounded sample quietly becomes a deployment tax.

• Classify prompt rendering, retry behavior, schema validation, and answer quality as tests or evals, with a reason for each classification

  Infrastructure invariants are repeatable assertions, while semantic quality depends on a stochastic model and a scoring method.

Chapters

1. 1

   Tests, evals and repeatability

   The dividing line is not cost and not whether a model gets called: it is whether the same commit yields the same verdict twice, in minutes. This chapter defines both halves, shows why the two popular proxies mis-sort a human-rated check into the test suite and push a recorded-response validator out of it, works the arithmetic on temperature 0, and shows how a check can be moved across the line by changing how it is measured.

   Read chapter
2. 2

   Sorting a feature's checks

   Four checks from one order-support assistant get classified with the reason written down: prompt rendering against a fixed retrieval fixture, retry and backoff behaviour, schema validation, and whether the answers are good. Three are infrastructure invariants with one right result; schema validation turns out to be two checks under one name; and answer quality carries variance from both the generator and the grader, which is what puts it on the far side of the line.

   Read chapter
3. 3

   Triggers and detection latency

   Having sorted the checks, place them. This chapter shows why the time between a bad merge and anybody knowing is a property of the trigger rather than of the check, lays out the two paths — a free deterministic suite on every push, and a frozen-sample eval on merge to main whose number is stored by commit sha and consulted by the release job — and covers what has to be frozen for two runs to be comparable at all.

   Read chapter
4. 4

   Bypassed gates and stale greens

   An eval that costs thirty-eight minutes and nine dollars on every push gets marked non-blocking, and six weeks later nobody reads it. This chapter states precisely what protection remains, works the arithmetic that made the bypass inevitable, audits six weeks of run history before anyone uses it as a baseline — skipped runs, masked crashes, dropped error cases, an instrument that drifted — and wires a release check that can tell a fresh number from a stale green.

   Read chapter
5. 5

   Sizing an eval set

   How many cases an eval needs is the answer to a question, and the question is how small a regression the gate must be able to see. Starting from the effect you would refuse to ship and the pass rate you actually sit at, the arithmetic hands back a case count — and shows why pairing on a frozen set buys the resolution of 800 unpaired cases for 314, and why cutting 400 cases to 100 leaves a job that still runs, still costs money and can no longer resolve what it was built for.

   Read chapter
6. 6

   Eval cost and cadence

   Cost per run is cases times draws times generation and judge tokens, inflated by retries — and the judge is usually 80% of it. That figure is then multiplied by how often the gate fires, which is the term with the largest range and the one written in a workflow trigger where nobody reads it as a number. This chapter decomposes both, prices the same suite from $21 to $2,946 a month on trigger alone, and shows why shrinking the case set to fit a budget is the one cut with no visible symptom.

   Read chapter
7. 7

   Rare defects and sampled aggregates

   The argument for deleting a deterministic suite once an eval covers the path end to end is right about the happy path and wrong about everything else. A defect hitting 1 request in 200 moves a 400-case score by half a point, which is a ninth of the noise floor; and when the score does move, what arrives is a number rather than a stack trace, so every hypothesis costs a full run. This chapter prices both, and gives the rule that keeps a deterministic suite from growing without bound.

   Read chapter