Evaluation in CI

Read these first

• Eval tooling landscape and selection criteria

What you will know

• Gate a pull request on an eval threshold

  A green result must be reproducible for the evaluated commit rather than dependent on a developer’s local rerun.

• Choose a threshold that fails on real regressions and not on noise

  Flaky judges, small samples, and model variance can turn a useful policy into merge roulette.

• Keep the CI eval cheap enough that people don't skip it

  A slow or expensive check becomes an ignored check, regardless of how valuable its signal is.

• Add a cost and latency budget to the same gate

  A quality improvement that doubles p95 latency or serving spend can still be a production regression.

• Explain what happens when the gate fails and the change is still right

  An override without an owner, rationale, and expiry turns an exception into a permanent policy bypass.

Chapters

1. 1

   Binding an eval score to a commit

   A number pasted into a pull request is an assertion about an unspecified computation: nothing in it shows which code, which cases, which model and which grader produced it. Running the suite in the pipeline against a named commit and emitting a run manifest is what makes a score interpretable — and a green check still supports a much narrower sentence than a reviewer hears.

   Read chapter
2. 2

   Placing an eval gate threshold

   The spread of the score on an unchanged commit is the noise floor, and it decides both numbers at once: how often a correct change is blocked, and the smallest regression the gate can reliably see. Measured first, the choice between an absolute threshold and a relative one stops being a matter of taste — and the relative form needs a wider band, not a tighter one.

   Read chapter
3. 3

   Re-runs and gate precision

   An eval score is a distribution, so re-running a failed check is another draw rather than a repaired measurement, and the reported number becomes the maximum over attempts. Measure the empirical flip rate before arguing about it, then reduce the variance — because tightening the threshold against a re-run habit increases the number of times people have to fight the gate.

   Read chapter
4. 4

   Tiering the eval suite

   A gate has to be cheaper to run than to avoid, and an 800-case suite at 22 minutes and $14 a push is not. Splitting it into deterministic assertions and a fixed 60-case smoke tier per push, with the full run on merge to main and a low-variance nightly, closes the developer's loop in two minutes — and the fast tier has to be validated against real past regressions before it can gate.

   Read chapter
5. 5

   Cost and latency budgets in CI

   An eval run already makes every API call the product makes, so the token counts and stage timings are sitting in response objects being thrown away. Reporting all three axes on the same run turns a change that buys quality with latency into a decision somebody makes, rather than a side effect a green quality gate green-lights.

   Read chapter
6. 6

   Eval gate overrides

   Sometimes a blocked change is right and the eval set is stale, and merging is the correct outcome. What separates an exception from a permanent hole in the gate is entirely in what gets recorded — an owner, a reason, an expiry and a scope, in a reviewed file with a sweeper enforcing the date. And a log of correct overrides is a verdict on the case set, not on the gate.

   Read chapter