Eval tooling landscape and selection criteria

Read these first

• Deterministic and code-based checks

What you will know

• Run an eval through at least one existing tool

  A real run exposes schema constraints, flaky judge behavior, and the gap between a demo and repeatable CI.

• Compare at least three eval tools and a hand-built harness against the same dataset, scoring integration effort, reproducibility, latency, cost, and failure visibility

  A polished interface can conceal weak versioning or opaque graders, so include a deliberately failing case and a CI run.

• Write a versioned eval schema that keeps prompts, model settings, raw outputs, labels, and failure categories independent of any tool

  Without raw artifacts and stable taxonomy IDs, later tooling cannot reproduce regressions or compare results.

• Build a minimal harness by hand and measure what it gives up

  Record missing capabilities such as parallelism, caching, review workflows, retries, or statistical analysis rather than treating fewer lines as equivalent.

• Migrate an eval between tools without losing the labels

  Export and import must retain stable case IDs, annotations, taxonomy versions, and disagreement metadata, or trend lines silently reset.

Chapters

1. 1

   Repeatability in an eval tool trial

   The score an eval tool shows in a demo measures your assistant; the score the same tool produces in CI measures whether the tool can hold a measurement still. Ten repeats of one unchanged suite separate the two, and three controls — a seed on the grader call, sampling settings, and a repeat-and-aggregate mode — decide whether a candidate can block a merge at all.

   Read chapter
2. 2

   Eval artifact ownership

   Nothing about choosing an eval tool is hard to reverse; where the artifacts ended up while you were using it is. Six things — prompts, model settings, raw unscored outputs, labels carrying their annotator and taxonomy version, grader definitions, and a stable case ID joining all of them — are what let you reproduce a six-month-old run case by case, and only one of them has no recovery path.

   Read chapter
3. 3

   Build versus buy for an eval harness

   Two hundred lines produce the same score on the same cases as the vendor tool you trialled, which is the one axis on which hand-written code always looks good. The honest list of what is absent — concurrency with rate-limit backoff, caching, errors separated from failures, a review queue, variance handling — comes to eight to ten engineer-weeks, and three conditions still make keeping the harness the right answer.

   Read chapter
4. 4

   Running an eval tool bake-off

   A feature matrix assembled from vendor documentation scores marketing, because every vendor claims every property. What separates candidates is one frozen case set, six deliberately broken cases, and a stated rubric for what each tool tells you when it breaks — plus a rule about which of the five criteria may be filled in from reading and which have to be observed.

   Read chapter
5. 5

   Migrating between eval tools

   Row counts and aggregate scores are invariant under a relabelling of identity, which makes them the two checks guaranteed to pass through the exact failure that destroys a migration: an importer that mints its own case IDs. What to verify instead, in what order, and what a text-hash recovery gets back when the identifiers are already gone.

   Read chapter
6. 6

   Exit planning for an eval vendor

   A shutdown notice gives you thirty-one days, and what you keep is whatever already lives outside the vendor. An export contains the fields they chose to model — typically cases and scores, typically not raw outputs, annotator identity, taxonomy versions or grader definitions — and raw outputs are the one row whose absence disables the recovery of every other.

   Read chapter