Eval artifact ownership

An eval artifact is anything a run leaves behind that somebody might want to read again: the case that was run, the prompt it rendered into, the string the model produced, the verdict a grader gave that string, and the human label somebody later attached to it. Every eval tool stores some of these, because it has to in order to draw a chart. The question that decides how much a tool choice costs you is not which of them it stores but which of them it is the only copy of.

The relationship worth having in mind is the one a BI dashboard — business intelligence, the charting layer a company reads its numbers in — has to the warehouse underneath it. Nobody agonises over choosing between dashboard vendors, because the tables are in Postgres and the dashboard is a view over them; swapping the dashboard costs a sprint of rebuilding charts. The moment the dashboard is also the place the numbers are computed and stored, the decision changes character entirely. Eval platforms sit on exactly that boundary, and most of them will happily be both if you let them.

That comparison is an analogy and it is worth saying where it stops holding, because the place it breaks is the place the money is. A dashboard reads data that exists whether or not the dashboard does; the warehouse was filled by something else entirely. An eval platform is not only a view. Its review queue is where human labels come into existence, and its grader is what computes verdicts, so left to itself it becomes the origin of two of the six artifacts rather than a reader of them. That is the difference between a decision you can reverse in a sprint and one you cannot reverse at all, and it is why the arrangement at the end of this chapter is specifically about making the platform read what you wrote rather than write what only it holds.

The question that decides what to store#

One question settles which artifacts matter, and it is not abstract. Support escalates a ticket in November about a case the eval history says passed in June. Between June and November the prompt was edited eleven times, the retrieval index was rebuilt twice, the model was moved to a newer snapshot, and the grader rubric gained a clause about escalation language. Can you reproduce June's run for that one case and compare it, side by side, with today's?

Answering yes requires six things, and missing any one of them turns the investigation into speculation:

The third row is the one teams skip, and it is the one that cannot be recovered afterwards. Scores are cheap to store and everyone stores them; raw outputs are bulkier and feel redundant next to a verdict that already summarises them. They are not redundant, because a verdict is a lossy function of the output under a specific grader, and the grader changes.

"Lossy function" is the whole argument in two words, and it is worth unpacking them. A grader takes a paragraph of model output and returns a pass or a fail. Enormously many different paragraphs return a pass, so knowing that a case passed narrows the set of things the model might have said to a set with a great many members in it. The output cannot be reconstructed from the verdict, in the same way that the numbers that went into a sum cannot be recovered from the sum. That would not matter if the verdict were the only question anyone ever asks. It is not: the question that arrives in November is "what would the September rubric say about the June answer", and that question takes the output as an input. Storing only the verdict answers one question forever and every other question never.

When a rubric gains an escalation clause in September, the honest thing to ask is how the June outputs would score under the new rubric — which is a two-minute job if the outputs are on disk, and impossible otherwise. The fallback of regenerating June's outputs does not work either: a snapshot retired by the provider in the meantime means there is no amount of money that buys those strings back.

A schema that outlives the tool#

None of this needs a data platform. It needs a directory and a discipline about identifiers. The shape below is deliberately boring — newline-delimited JSON, one file per run, cases and taxonomy in git, raw outputs in object storage — and it survives every tool decision made on top of it.

evals/
  cases/returns.v3.jsonl          # in git, reviewed by PR
  taxonomy/failures.v4.yaml       # in git, category ids are permanent
  graders/groundedness.v2.md      # the judge prompt, in git
  graders/refund_policy.v1.py     # deterministic checks, in git
runs/                             # object storage, write-once
  2026-06-14T02:11Z_a3f19c/
    manifest.json
    outputs.jsonl
    scores.jsonl
labels/
  2026-06-18_batch7.jsonl         # object storage, append-only

A case record carries the identifier and nothing that changes when a run happens:

{"case_id": "ret-0031", "suite": "returns", "created": "2026-02-03",
 "question": "I bought these through the marketplace, can I return them here?",
 "history": [], "locale": "en-GB", "policy_version": "2026-01",
 "reference_answer": "Accept the return directly; do not redirect to the marketplace.",
 "supersedes": null}

A run manifest records everything about the execution that the outputs alone do not tell you:

{"run_id": "a3f19c", "started": "2026-06-14T02:11:07Z", "suite": "returns",
 "case_set_sha": "9f1c02e4", "prompt_template": "returns_answer.v11",
 "prompt_template_sha": "b7d5a1", "model": "gpt-4o-2024-08-06",
 "sampling": {"temperature": 0.2, "top_p": 1, "max_tokens": 600, "seed": 7},
 "retrieval_index": "policies-2026-06-02", "graders": ["groundedness.v2", "refund_policy.v1"],
 "tool": "vendor-b@2.4.1", "git_sha": "4c8e19b"}

And an output record is the raw string before any grader has looked at it, kept apart from the score so that re-grading is a pure function over stored data:

def record_output(run_id: str, case: EvalCase, completion: Completion) -> dict:
    return {
        "run_id": run_id,
        "case_id": case.id,                       # never the vendor's id
        "rendered_prompt": completion.request.prompt,
        "raw_output": completion.text,            # verbatim, unparsed, untrimmed
        "finish_reason": completion.finish_reason,
        "usage": {"input": completion.usage.input_tokens,
                  "output": completion.usage.output_tokens},
        "latency_ms": completion.latency_ms,
        "provider_request_id": completion.request_id,
    }

The storage cost is the reason people expect an argument here and there is not one. Two thousand four hundred cases with a 1,900-token prompt and a 400-token answer come to roughly 22 MB of JSON per run, or about 8 GB for a year of nightly runs, which is a rounding error against the cost of generating them. Compressed and lifecycled to cold storage after ninety days it is a few dollars a month.

Those two figures are worth arriving at rather than accepting, since the objection they answer is always raised as a worry about scale. A case carries 1,900 tokens of prompt and 400 of answer, which is two thousand three hundred tokens between them; two thousand four hundred cases at that size is a little over five and a half million tokens, and written out as JSON text that is the 22 MB in the sentence above. A year of nightly runs is 22 MB three hundred and sixty-five times, which is the 8 GB. Then set that beside what produced it: those same tokens have to be sent to a model and generated by it every night, and inference is billed per token while object storage is billed per byte-month at rates several orders of magnitude apart. You are already paying the expensive part. The cheap part is the one that decides whether next year's investigation is possible.

Identifiers are the part that has to be decided once#

case_id is the only field in the whole schema that cannot be fixed later, so it gets rules. It is minted by you, not by any tool, and every tool receives it as an opaque attribute. It is never derived from the case text, because normalising whitespace or redacting a customer name would then silently produce a different identifier for the same case. It is never reused, and it is never renumbered when the case set is reorganised. When a case changes materially — the question is rewritten, the reference answer is corrected because policy changed — you mint a new ID and set supersedes to the old one, which keeps the old results attached to the version of the case they actually judged while preserving the lineage.

Each of those rules blocks a specific way the identity can shift underneath the data, and the derivation rule is the one that looks harmless. Suppose the ID were a hash of the question text. A redaction pass replaces a customer's name with a placeholder, or a copy-editing pass collapses a double space, and the text is now a different string, so the hash is a different hash, so the case is a different case as far as every join is concerned. Nothing errored and nothing was deleted; the history simply stops at the edit. A minted identifier has no such coupling, because it was never computed from anything that can change.

The supersedes rule blocks the opposite failure. Editing a case in place while keeping its ID leaves last quarter's verdicts attached to a question that was never asked in that form, so a chart of that case over time is comparing judgements of two different things and reporting the difference as a trend. Minting a new ID separates them, and the supersedes pointer is what stops the separation from also losing the connection.

Failure categories need the same treatment for the same reason. They are your failure taxonomy: the agreed list of named ways an answer can be wrong — redirects the customer, invents a policy clause, answers in the wrong language — that an annotator picks from instead of writing free text, so that failures can be counted and trended by kind. A taxonomy entry is an ID and a version, and the human-readable name is a display string that may change freely:

version: 4
categories:
  - id: fc-003
    name: "Redirects customer to the wrong channel"
    since: 2
  - id: fc-014
    name: "Cites a policy clause that is not in the retrieved context"
    since: 3
    split_from: fc-009      # fc-009 covered all citation errors until v3

Renaming fc-003 costs nothing, because every label points at the ID. Splitting fc-009 into two categories is a real event, and the split_from field is what stops a chart of category frequencies from showing a cliff in March that has nothing to do with the assistant's behaviour. A label record therefore has to carry the taxonomy version it was assigned under, alongside who assigned it:

{"case_id": "ret-0031", "run_id": "a3f19c", "label": "fail",
 "categories": ["fc-003"], "taxonomy_version": 3,
 "annotator": "s.okonkwo", "labelled_at": "2026-06-18T09:41Z",
 "confidence": "high", "note": "policy is explicit for marketplace orders"}

Two annotators labelling the same case produce two records rather than one merged verdict, because the disagreement is data. Knowing that a case was labelled fail by one reviewer and pass by another is what tells you the case is ambiguous, and a schema that stores only the resolved majority throws that away permanently.

That is the same lossiness as the raw-outputs row, arriving at a different table. A resolved verdict is a function of the individual judgements, and it is a function that maps many inputs to one output: two reviewers agreeing on fail and two reviewers splitting with the tie broken toward fail both resolve to fail. Read the resolved column afterwards and those two situations are indistinguishable, even though the first is a case worth gating on and the second is a case whose rubric needs a clause. Storing both records costs one extra row and keeps the distinction available; storing the resolution alone discards it at the moment it is created, which is the only moment at which it was ever recoverable.

Buying a view over data you own#

This is not an argument for self-hosting, and it is not an argument against buying. A platform that gives your annotators a good review queue, computes agreement statistics, and draws trend charts is doing work you would rather not do, and none of that requires it to be the system of record. The arrangement that works is that your harness writes the artifacts above, then pushes a projection — a copy carrying only the fields that tool needs to do its job — into whichever tool you are using, with your case_id carried as an external key, which is the field a platform keeps verbatim so you can join its records back to yours. The tool becomes a fast, pleasant view over data you own.

The push in that direction is a small amount of code and it is worth writing deliberately, because the direction of the arrow is the whole point. Your harness runs the suite, writes outputs.jsonl and scores.jsonl to object storage, and then uploads a projection to the platform's API with case_id in whatever external-key field it offers — metadata.case_id, external_id, slug, the name varies and the requirement does not. Anything the platform computes on top, such as agreement statistics or a category breakdown, is pulled back down on a schedule and written into labels/ in your own format. If a run exists in the platform and not in object storage, that is a bug in the pipeline rather than a shortcut, and it is worth an assertion in CI, because the exceptions are how the system of record drifts back inside the vendor one convenience at a time.

This is also a different concern from request tracing, which is the neighbouring thing it gets confused with. Traces from production are high-cardinality, sampled, retained for thirty days, and organised around latency and errors; eval artifacts are low-cardinality, complete, retained for years, and organised around a fixed case set. The two overlap in that a production trace is often where a new eval case comes from, and they have opposite storage economics, which is why folding evals into your observability pipeline usually ends with the retention policy quietly deleting the thing you most wanted eighteen months later. It is not the same thing as versioning your scores either, which is the version of this advice that gets implemented and does not help: a score history in a Postgres table survives a tool migration and answers exactly one question — what the number was — while every question that starts with why needs the raw output next to it.

The limit worth being honest about is that none of this makes a run truly reproducible, because you do not control the model weights behind an API. A snapshot id pins behaviour more tightly than a floating alias and providers still retire snapshots, typically with six to twelve months of notice. What the schema buys is the ability to re-grade old outputs with a new grader and to compare case by case across time, which covers most real investigations.

Re-generating a June response in November is not on the table at any price, and that asymmetry is the whole argument compressed into one sentence: the artifacts outlive every tool that produces them, and exactly one of them has no recovery path. Keep the six outside, keyed by an identifier you minted, and the eighteen-month question has a boring answer — you rebuild the view and keep the data.

With the artifacts outside, the tool genuinely is replaceable infrastructure, and that makes a particular argument very tempting: if the valuable part is a directory of JSON and the tool is a view, why pay for the view at all when the harness is a couple of hundred lines. That argument deserves a real answer rather than a reflex, and the answer is a list of specific capabilities with prices on them.