Instrumenting an eval run

A nightly eval that posts one number to a channel is emitting a metric in the sense monitoring used the word a decade ago: pick the questions in advance, emit a counter or a gauge for each, roll them up. That model works until production asks a question nobody pre-registered — why are requests slow for this one customer on this one endpoint after the deploy — at which point the aggregates are useless, because the dimensions that would separate the slow requests from the fast ones were summed away before the data was stored. The replacement was the wide event: one row per unit of work, carrying every dimension you have, queried after the fact — including for the questions nobody was in a position to anticipate at the moment the row was written.

An eval run is the same shape of problem. A score of 0.831 is a gauge. The row per case is the wide event, and the dimensions that matter for context engineering are known in advance even though the questions are not: how many tokens each block contributed, what retrieval returned and in what order, how long the call took, whether the answer truncated, which slice the case belongs to, and what configuration produced all of it. None of these require new infrastructure. They are values the harness already holds in memory a few milliseconds before it discards them.

The row#

type CaseResult = {
  // identity — what makes the join possible
  run_id: string;            // "run_20250812_0300_A"
  arm: string;               // "with-style-guide"
  case_id: string;           // "sup-0142"
  slice: string;             // "refunds/out-of-window"
  repeat: number;            // 0..n-1 when repeats are on

  // quality
  score: number;             // 0.0 or 1.0, or 1..5 for a graded judge
  judge_votes: number[];     // [1, 1, 0] when the judge is run 3x
  truncated: boolean;        // output hit max_tokens

  // cost, per assembled block — this is the part people leave out
  tokens: {
    system: number;          // 900
    style_guide: number;     // 1500
    few_shot: number;        // 2400
    passages: number;        // 9800
    conversation: number;    // 880
    output: number;          // 218
  };

  // provenance
  retrieval: {
    index_snapshot: string;  // "idx-2025-08-01"
    k: number;               // 5
    chunk_ids: string[];     // ordered, rank 0 first
    scores: number[];        // parallel to chunk_ids
    gold_present: boolean;   // gold chunk id ∈ chunk_ids
    gold_rank: number | null;
  };
  block_order: string[];     // ["system","style_guide","passages","few_shot","conversation"]

  // operations
  latency_ms: number;        // 3410
  retries: number;           // 0
  model: string;             // "claude-sonnet-4-5-20250929"
  judge_model: string;
  judge_rubric_version: string; // "support-helpfulness-v4"
};

Every field on that type answers a question somebody eventually asks in a regression thread, and the mapping is worth being explicit about because it is what justifies the fields to whoever has to write them.

The retrieval fields carry one word that needs unpacking, since it recurs through the rest of this topic: the gold chunk is the passage a human labelled as the one containing the answer for that case, so gold_present records whether retrieval returned it at all and gold_rank records where in the ordering it landed, and the pair separates a retrieval failure from a generation failure without a re-run. Topic 3.14, on component and end-to-end evaluation, established that the two answer different questions and that attribution needs the trace; this row is where that trace becomes queryable across a whole run rather than readable one case at a time. Storing the ordered chunk IDs and their scores means you can diff two runs' retrieval sets directly — set difference per case, count of cases where rank 0 changed — instead of inferring retrieval behaviour from answer quality.

A regression, as a query#

With rows, an argument about what changed becomes four queries and about ten minutes. The first asks which configuration fields differ between two runs and what the aggregate columns did, which kills every hypothesis that names a field the rows show to be identical.

SELECT run_id,
       ROUND(AVG(tokens_passages))                     AS avg_passage_tokens,
       ROUND(AVG(tokens_system + tokens_style_guide
                 + tokens_few_shot + tokens_conversation)) AS avg_other_tokens,
       ROUND(AVG(CASE WHEN gold_present THEN 1.0 ELSE 0.0 END), 3) AS gold_hit_rate,
       ROUND(AVG(latency_ms))                          AS avg_latency_ms,
       ROUND(AVG(score), 3)                            AS score
FROM   case_results
WHERE  run_id IN ('run_20250811_0300_A', 'run_20250812_0300_A')
GROUP  BY run_id;

run_id                  avg_passage_tokens  avg_other_tokens  gold_hit_rate  avg_latency_ms  score
run_20250811_0300_A                  9 812             5 682          0.913           3 388  0.871
run_20250812_0300_A                 12 604             5 682          0.847           4 021  0.831

The passages block grew by 2,792 tokens per case while every other block stayed the same to the token, the gold passage stopped being retrieved on 6.6% more cases, and latency rose 633ms. That is a retrieval-side change, and index_snapshot names which one. None of it required a hypothesis about prompts; it required the run to have written down what it sent.

Each of those three figures is one subtraction between two rows of that output, and doing them by hand is what the rows are for. Passage tokens go from nine thousand eight hundred and twelve on the first night to twelve thousand six hundred and four on the second, and the difference is the per-case growth quoted. The gold hit rate falls from a little over nine tenths to a little under eight and a half tenths, and the difference between two proportions of the same five hundred cases is a percentage of cases, not a percentage of the rate — which is what makes it comparable with the score column standing next to it. Mean latency goes from three thousand three hundred and eighty-eight milliseconds to four thousand and twenty-one, and the difference is the rise quoted. The column that did not move is doing as much work as the three that did: avg_other_tokens is identical to the token across both runs, which eliminates every hypothesis about the prompt in one glance rather than in one thread.

Why cost and quality have to be the same row#

The obvious objection to the token fields is that cost already lives somewhere: the provider's usage dashboard, bucketed by hour and API key. That data cannot answer any of the questions this topic asks, because the aggregation it arrives in has already destroyed the join. Cost per hour tells you the eval run was expensive. It cannot tell you that the 2,792 extra tokens landed on the cases whose gold passage disappeared, which is the sentence that identifies the bug.

The same argument applies to every decision in the rest of this topic. "Is 1.5 points worth 4,000 tokens and 600ms" is a question about a distribution, not two scalars: the 4,000 tokens may be concentrated on 12% of cases, the 600ms may be entirely in the p95 tail — p95 being the latency that 95% of requests come in under — and the 1.5 points may come from the same 12%. When quality, tokens and latency sit in one row per case, that is a GROUP BY away. When they live in three systems keyed by different things, it is a research project, and the research project does not get done — the trade gets argued from intuition instead, which is how a team ends up shipping a reranker because it felt more principled.

One field earns its place for a subtler reason. judge_votes records the three individual judge calls rather than only their majority, which lets you separate cases the judge is confident about from cases it is flipping a coin on. A case that scores [1, 1, 0] in one run and [1, 0, 0] in the next has moved, but it was never stable, and a regression made mostly of such cases is a rubric problem rather than a model problem. Counting unanimous against split judgements across a run gives a cheap health check on the judge itself: on the support set, roughly 6% of cases come back split, and when that fraction jumps after a rubric edit, the edit made the rubric vaguer rather than sharper.

Rows produce candidates, arms produce conclusions#

A well-instrumented run is still an uncontrolled run if more than one thing changed, and the Tuesday example is not a counter-example: the query narrowed five candidates to one because four configuration fields were recorded and happened to be identical, not because the rows established causation. The chunker change is a candidate with strong circumstantial evidence, and confirming it means doing what the first chapter of this topic described — pin everything, change the chunk size back, re-run. Rows turn arguments into candidates; arms turn candidates into conclusions.

Nor is this the same thing as production tracing. Everything here describes an offline run over a frozen set, where you control the inputs and can afford to write two kilobytes per case. Production tracing has different constraints — sampling, the handling of personally identifiable information, latency budgets that do not tolerate synchronous writes — and different questions, since production has no gold labels to compare against. The two are complements, and a team that instruments only production discovers regressions from customer complaints rather than from the nightly run.

The storage argument against all this does not survive contact with the numbers. A CaseResult row serialises to roughly 2 kB with fifty chunk IDs; 500 cases times three repeats times two arms is 3,000 rows, or about 6 MB per ablation. A nightly run plus a handful of ablations is comfortably under 500 MB a year, which is a rounding error next to the ninety engineer-minutes one unattributable regression costs.

There is also a limit the schema itself imposes. The row records what was assembled and what came back; it does not record what the model attended to. When a style-guide block hurts a case, this data cannot distinguish "the model read the block and followed it badly" from "the block pushed the relevant passage 1,500 tokens further down and the model stopped using it". The second explanation is a positional effect rather than a content effect, and separating the two needs an experiment that varies position deliberately — which in turn raises a question no amount of instrumentation answers, namely whether the sheer volume of context is degrading the model independently of what the context says.

A run that cannot be attributed cannot be acted on, and attribution is a property of what the harness wrote down rather than of how clever the analysis is afterwards. One row per case per arm, carrying quality, cost, provenance and configuration together, is what turns a ninety-minute thread into four queries — and it is the same set of columns that priced the trade in the previous chapter and that the rest of this topic keeps reaching for.

Before any of it can be used on a prompt change, though, there is a simpler confusion to clear up: the eval you are watching may be structurally incapable of seeing the block you removed.