Exit planning for an eval vendor

Exit planning is the part of a tool decision that gets deferred because it has no deadline until it has a very short one. For an eval platform it comes down to a single question asked of each artifact a run leaves behind: if this vendor stopped serving requests tonight, where is the copy you would work from tomorrow. That question has an answer today, and writing it down takes twenty minutes and a whiteboard.

Do it before you need it, because the notice period is not when the work happens. A shutdown email gives thirty-one days, which is enough to pull data and stand up a replacement runner and not enough to recreate an artifact that was never stored. Go through the six artifacts an eval run leaves and write down, for each, where the authoritative copy lives:

Three of six intact is a week of work. Row three empty is a quarter, and the rest of this chapter is about why that particular row carries so much more weight than the others.

An export is a statement about the vendor's data model#

The reflex answer to a shutdown email is that there is an export endpoint, so the data comes out and goes somewhere else. That is true and it is also the wrong frame, because an export contains the fields the vendor chose to model, and those were chosen to serve their product rather than your reconstruction. Here is a line from a real-shaped export, with a returns case in it:

{"id": "case_88f3a201", "dataset": "returns-prod", "input": "I bought these through the marketplace, can I return them here?",
 "expected": "Accept the return directly; do not redirect to the marketplace.",
 "metadata": {"case_id": "ret-0031", "locale": "en-GB"},
 "scores": [{"name": "groundedness", "value": 1.0, "run": "r_5512", "at": "2026-06-14T02:19Z"},
            {"name": "policy_match", "value": 0.0, "run": "r_5512", "at": "2026-06-14T02:19Z"}],
 "annotations": [{"label": "fail", "category": "Redirects customer to the wrong channel",
                  "by": "usr_41", "at": "2026-06-18T09:41Z"}]}

Read it against the table above. The case text is there, the scores are there, the timestamps are there. The model's actual answer in June is not there, because the platform stores run outputs in a separate object it does not include in the case export and offers only through a paginated runs API that rate-limits at 100 requests per minute — which for 2,400 cases across eighteen months of nightly runs is roughly 540,000 records, or about ninety hours of continuous pulling if you start on day one and nothing fails. The annotation carries a category as a display string with no ID and no taxonomy version, so a category that was split in November and a category that was renamed in January are indistinguishable in the export. usr_41 is a vendor user ID whose mapping to a person lives in their admin console, which is also going away. The grader called groundedness produced a 1.0, and the prompt that defines what groundedness meant on 14 June is in their rubric editor.

That ninety-hour figure is one division, and it is worth doing because it is the number that decides whether the notice period is enough. 540,000 records pulled at 100 requests a minute is five thousand four hundred minutes, and five thousand four hundred minutes is ninety hours. Ninety hours inside thirty-one days sounds comfortable until you notice what it assumes: that the pull runs unattended for the best part of four days, that it resumes from where it stopped rather than restarting, and that nothing else in the company needs that rate limit for the duration. The number is not the problem. The number's preconditions are.

Why raw outputs decide the timeline#

Every other missing artifact has a recovery path with a price on it. Labels can be re-collected: 2,400 cases at two annotators and ninety seconds each is about 120 hours of reviewer time, which at a fully loaded £40 an hour is roughly £4,800 and three calendar weeks with a small team, and the result is arguably better than what was lost, since it is labelled against the current taxonomy. Grader definitions can be rewritten from the rubric text people remember plus the scores you did export, in a few days. Annotator identity can be partially reconstructed from a Slack search and the admin console if you screenshot it before the shutdown. Run configuration can often be inferred from your own git history.

The first of those prices comes out of two multiplications, and it is worth doing them because a number with a currency symbol on it turns an argument about principle into a line item somebody can approve. 2,400 cases judged twice is four thousand eight hundred judgements. Four thousand eight hundred at a minute and a half apiece is 120 hours, and 120 hours at £40 is the £4,800. Expensive, schedulable, and entirely possible — which is the point of contrast with the row below.

Raw outputs have no recovery path at all, and this is not a matter of cost. Regenerating June's answers requires the model that produced them, and a snapshot like gpt-4o-2024-08-06 retired in the meantime means there is no endpoint that will produce those strings for any amount of money. Every one of the recovery paths above also runs through the outputs: re-labelling 2,400 cases means annotators reading model answers, and if the answers are gone the annotators are labelling nothing. A rewritten grader can only be validated by running it over stored outputs and checking that it reproduces the old verdicts, which is impossible without them. Labels without outputs are annotations attached to text that no longer exists, which makes them a record of a judgement nobody can audit.

That is the asymmetry that turns a week into a quarter: outputs are the only artifact whose absence disables the recovery of the others.

It is worth laying the dependencies out in one direction, because the table above presents six rows as though each stood alone. Re-collecting labels requires annotators to read something, and what they read is the model's answer, so labels depend on outputs. Validating a rewritten grader requires running it over answers whose old verdicts you know, so graders depend on outputs. Reconstructing what a run configuration must have been often means checking a stored answer against a suspected prompt, so even manifests lean on them. Nothing in that list runs the other way: outputs do not depend on labels, or on graders, or on annotator identity. So the six artifacts are not six independent risks of equal weight. Five of them sit downstream of one, and the one they sit downstream of is the only one with no way of being produced a second time.

The reason it cannot be produced a second time is worth separating from the reason the others can. Everything else on that list is information a person once wrote down and a person can write down again, at a cost in hours. An output is not information anybody wrote down. It is what a particular model, at a particular version, produced from a particular prompt on a particular night, and the only process that produces it is that model. When a provider retires the snapshot, the process no longer exists, so the artifact has moved from expensive to impossible — and that transition happens on the provider's schedule, silently, whether or not anyone is thinking about exits.

The five conditions behind a one-week exit#

The harness writes cases, manifests, raw outputs and scores to your own storage on every run, and the vendor receives a projection. Grader definitions — judge prompts and deterministic checks alike — live in git and are pushed to the vendor, never authored in their editor. Labels are pulled back down on a schedule into your own append-only store, carrying annotator, taxonomy version and every individual annotation rather than a resolved majority. CI names an interface rather than a vendor:

from typing import Protocol

class EvalBackend(Protocol):
    def submit_run(self, manifest: RunManifest, outputs: list[dict]) -> str: ...
    def fetch_labels(self, since: str) -> list[dict]: ...
    def run_url(self, run_id: str) -> str: ...

class VendorBBackend:
    def submit_run(self, manifest, outputs) -> str:
        payload = [{"external_id": o["case_id"], "input": o["rendered_prompt"],
                    "output": o["raw_output"]} for o in outputs]
        return self._post("/v2/experiments", {"meta": manifest.as_dict(), "rows": payload})["id"]
    ...

class LocalBackend:
    """Writes the same records to disk. The fallback that makes the deadline survivable."""
    def submit_run(self, manifest, outputs) -> str: ...

The interface is thirty lines and its value is not abstraction for its own sake; it is that on the day of the email, the change required to keep CI running is a config value, and the eval suite continues to produce numbers while the migration happens at a sane pace.

The mechanism is worth being explicit about, because "add an interface" is advice people have heard often enough to stop hearing. Without one, the vendor's SDK is called from wherever it was convenient — the CI script, the nightly job, a notebook, a reporting task — so the set of places that have to change when the vendor goes away is the set of places somebody once imported it, and nobody knows what that set is. With one, every one of those callers names EvalBackend instead, and the only code that mentions the vendor is the class that implements the interface. Replacing the vendor is then a change to which class gets constructed. The interface does not make the migration easier; it makes the size of the migration knowable in advance, which is the property that matters when the notice period is fixed.

LocalBackend is worth having permanently, because it is also what makes the fifth condition possible.

The fifth condition is a restore drill that has actually been performed. Once a quarter, export everything, restore it into an empty target, and run the verification from the chapter on migrating between tools against it:

#!/usr/bin/env bash
set -euo pipefail
STAMP=$(date -u +%Y%m%dT%H%M)
evalctl export --all --out "drills/${STAMP}/export.jsonl"
evalctl restore --backend local --in "drills/${STAMP}/export.jsonl" --out "drills/${STAMP}/restored/"
python -m evals.verify \
  --src "drills/${STAMP}/export.jsonl" \
  --dst "drills/${STAMP}/restored/cases.jsonl" \
  --fail-on-any
python -m evals.regrade \
  --outputs "drills/${STAMP}/restored/outputs.jsonl" \
  --grader graders/groundedness.v2.md \
  --compare-to "runs/latest/scores.jsonl"

The last step is the one that catches the expensive surprise, because re-grading restored outputs and comparing against the recorded scores is the only check that proves the outputs were actually in the export and are actually usable.

Every earlier step in that script passes without them. export writes a file whether or not the file contains outputs. restore reads that file successfully, because a missing field is not a parse error. The verification compares case IDs, case text and label counts, none of which involve an output. So four green steps in a row establish nothing about the one row that decides the timeline. The re-grade is different in kind rather than in thoroughness: it is a computation that takes an output as an input, so it cannot complete when the outputs are absent, and it cannot silently complete on the wrong ones because its result is compared against scores recorded at the time. The general rule is that a backup is only verified by something that consumes it, and every check that merely inspects it will agree that it looks fine.

Teams that run this drill discover the gaps in a quarter with no deadline attached. Teams that assume the export works discover them with three weeks left, which is how a week of work becomes a quarter.

Freezing instead of migrating#

Thirty-one days is not enough for a faithful migration of eighteen months of history, and the honest response is often to stop trying. Freeze instead: pull whatever the export gives you, store it as an immutable archive, and put a small read-only query layer over it — DuckDB, an analytical database that runs inside your process and queries files on disk without a server, answers most historical questions over NDJSON — newline-delimited JSON, one record per line — in a couple of days of work, against roughly three weeks for a faithful migration into a new platform's schema.

Accepting the freeze requires three things to be true. Historical questions have to be answerable from the archive, which means someone actually tries a few — when did ret-0031 last pass, what did category fc-014 look like in Q3 — before the vendor goes dark. The boundary has to be documented where the charts are, so that a step change in April is read as a change of instrument. And no compliance or contractual obligation may require case-level continuity across the boundary, which is worth checking rather than assuming, since a model-risk policy in a regulated industry sometimes does require exactly that.

A frozen back-reference set is what makes the two eras comparable at all: a couple of hundred cases with their raw outputs, kept in your own storage, so any future grader can be run over them and calibrated against any past grader. Two hundred cases is about 4 MB, and it is the cheapest insurance in this entire topic.

Where the ground is still moving#

Portability is slowly improving from the outside. The OpenTelemetry GenAI semantic conventions — an open standard that fixes what the fields of a recorded operation are called, so that any tool can read another's records — have given the industry a shared vocabulary for model calls, expressed as spans, meaning one timed unit of work carrying model id, token usage, prompt and completion, and a growing number of platforms ingest traces in that format, which means the run-level half of this problem is becoming a solved standards question rather than a per-vendor negotiation. What the conventions do not cover is the eval-specific half: labels, annotator identity, taxonomy versions, grader definitions, case sets. Those remain modelled differently by every tool, which is why the schema you own still has to exist and why the drill still has to run.

A vendor exit, then, is not an event you respond to; it is a report on decisions already taken. The notice period buys time to pull an export and repoint a runner, and it cannot buy an artifact that was never written outside the vendor — which in practice means the raw unscored outputs, because they are the only one with no recovery path and the one that every other recovery depends on. Three of six artifacts held outside makes a shutdown a week. Row three empty makes it a quarter, and no amount of urgency in April changes that.

The thing to do with all of this is not to read another comparison. Take whichever eval you already run, spend an afternoon writing your case set and taxonomy into files with permanent IDs, point your runner at both your own storage and one tool, and then — this is the part that converts the topic into a capability — delete the tool's data in a scratch project and rebuild the view from your files. Whatever you cannot rebuild is the list of what you do not own yet, and that list is short, specific, and much cheaper to close on a quiet Thursday than on the thirty-first day.