Migrating between eval tools

Moving an eval suite from one tool to another looks like a data-transfer problem and is really a question about identity. What comes across easily is content: the text of each case, its reference answer, the scores it has been given. What comes across only if somebody insists on it is the claim that a record in the new system and a record in the old one are the same case — and every question worth asking of eval history is a join on exactly that claim.

The vocabulary that makes the failure legible was settled by data warehousing decades ago and is worth borrowing wholesale, because it makes the fix obvious. A natural key is derived from the data — the customer's email, the question text — and it changes when the data is corrected. A surrogate key is minted once, means nothing, and never changes, which is what the chapter on artifact ownership required of case_id. The wrinkle warehouses call a slowly changing dimension is the same wrinkle as a case whose reference answer gets corrected: keep the old row, mint a new key, record the lineage, which is what a supersedes field is for. An eval case set is a dimension table — the stable list of the things you measure, one row per case — with a fact table of run results, one row per case per run, hanging off it, and every mistake available in that modelling is available here.

Counting rows and comparing a mean survive a relabelling#

Take 2,400 labelled cases exported from tool A and imported into tool B, and run the two checks that get run: 2,400 rows in, 2,400 cases out, and a full re-run in B scoring 0.863 against A's last recorded 0.863. Both are invariant under a relabelling of identity, which is a formal way of saying they are the two properties guaranteed to survive the failure most likely to occur. Shuffle the identifiers on a set of 2,400 records and the count is still 2,400 and the mean is still 0.863. Reverse them, hash them, replace them with UUIDs, and the two green ticks stay green. That is not bad luck; those checks were chosen because they are easy, and easy checks are usually the ones with the most invariances.

"Invariant" is doing precise work there and it is worth taking apart, because the same test applies to any verification anybody proposes. A check is a function: it takes the data and returns a value. Ask what the value depends on. Counting rows depends on how many records there are and on nothing else — no arithmetic in a count ever consults an identifier, so replacing every identifier with a different one leaves the count exactly where it was. The mean depends on the scores and on how many there are; identifiers do not enter that calculation either. So both functions return the same value on the data with identity intact and on the data with identity destroyed, and a function that returns the same value in both situations cannot tell you which one you are in. This is the same argument as the one about a score with two sources, and it has the same consequence: no amount of care in reading the output recovers information the output does not depend on.

Turn the test round and it also tells you what to check instead. A verification can only detect a failure of identity if its value changes when identity changes, so the useful checks are exactly the ones that consult the identifiers: whether the set of IDs on one side equals the set on the other, whether a per-case join succeeds, whether the number of labels attached to each named case matches. Every check in the script below has that property, which is why it is longer than two lines and why two lines were never going to be enough.

What survives the boundary is the top-level number, so the chart on the team dashboard is continuous and looks healthy. What does not survive is every question worth asking: which cases regressed, whether this case has ever passed, whether the November rubric change moved these particular forty cases, whether the cases labelled by one annotator behave differently from the rest.

What to verify before anyone decommissions the old tool#

The verification is a script, it runs against both exports, and it fails loudly. Writing it takes an afternoon and it is the entire difference between a migration and a data loss event.

import hashlib, json
from collections import Counter

def load(path):
    return [json.loads(line) for line in open(path)]

def norm(text: str) -> str:
    return " ".join(text.split()).strip().lower()

def verify(src_path: str, dst_path: str) -> list[str]:
    src, dst = load(src_path), load(dst_path)
    problems = []

    src_ids = {r["case_id"] for r in src}
    dst_ids = {r.get("case_id") or r.get("metadata", {}).get("case_id") for r in dst}
    if src_ids != dst_ids:
        problems.append(f"case_id mismatch: missing={len(src_ids - dst_ids)} "
                        f"invented={len(dst_ids - src_ids)}")

    # the id must be the join key in the destination, not a metadata string
    if any(r.get("case_id") is None for r in dst):
        problems.append("case_id survived only inside metadata; joins will not work")

    src_text = {r["case_id"]: hashlib.sha256(norm(r["question"]).encode()).hexdigest()
                for r in src}
    dst_text = {r["case_id"]: hashlib.sha256(norm(r["question"]).encode()).hexdigest()
                for r in dst if r.get("case_id")}
    drifted = [k for k, v in src_text.items() if dst_text.get(k) != v]
    if drifted:
        problems.append(f"case text changed for {len(drifted)} cases, e.g. {drifted[:3]}")

    src_labels = Counter(l["case_id"] for r in src for l in r["labels"])
    dst_labels = Counter(l["case_id"] for r in dst for l in r.get("labels", []))
    if src_labels != dst_labels:
        diff = [k for k in src_labels if src_labels[k] != dst_labels.get(k, 0)]
        problems.append(f"label count changed for {len(diff)} cases — "
                        f"disagreements were probably collapsed")

    for field in ("annotator", "taxonomy_version", "labelled_at"):
        missing = sum(1 for r in dst for l in r.get("labels", []) if field not in l)
        if missing:
            problems.append(f"{missing} labels arrived without {field}")

    return problems

Four of those checks catch the failures that actually happen. Identifier equality catches the minting. The case_id is None check catches the subtler version, where the string is present in a metadata blob and the destination will neither index nor join on it — the data is there and the join is still impossible. The label-count comparison catches collapsed disagreements: where A stored two labels for the 87 cases whose annotators disagreed and B's importer accepts one label per case, 2,487 labels become 2,400 and the disagreement metadata has evaporated with no error. The arithmetic there is the whole detection: 2,400 cases, of which 87 carry a second label, is 2,487 label records; an importer that keeps one per case keeps 2,400 of them, and the 87 that vanish are precisely the second opinions. Counting labels catches it and counting cases does not, because the case count was 2,400 before and after. The per-field checks catch annotator identity and taxonomy version, which importers drop routinely because they are not part of the destination's model of a label.

There is a fifth check that only becomes possible once the join works, and it is the one that retires the matching aggregate as evidence for good. Take A's last recorded per-case verdicts and B's first re-run, join on case_id, and count the cases where they disagree. On one returns case set the two systems agreed on 2,190 of 2,400 cases: 105 cases that A passed were failed by B, and 105 that A failed were passed by B, which cancels perfectly in the mean and leaves 8.75% of the suite being read differently by the two graders. That is not a defect in either tool — it is two judge implementations interpreting the same rubric — and it is precisely the size of effect that a per-case comparison finds and an aggregate hides. A team that has this number can calibrate; a team that has 0.863 twice believes the two systems are interchangeable.

Both halves of that sentence are arithmetic worth doing. The disagreements are 105 in one direction and 105 in the other, which is two hundred and ten cases between them; two hundred and ten out of 2,400 is 8.75%, and the 2,190 agreements plus those disagreements account for the whole suite. The cancellation is even simpler: a mean of pass counts goes up by one case for every fail that became a pass and down by one for every pass that became a fail, so 105 of each leaves the count where it started. That is the same mechanism a flat composite uses to hide a regression against an improvement, arriving here at the level of individual verdicts rather than of stages. Two systems that disagree about one case in eleven can report identical scores, and identical scores are the evidence people take as proof that the migration preserved everything.

Add one non-automated step. Pick thirty cases at random, open them in both systems side by side, and read them. Automated checks compare the fields you thought of; a human reading a case notices that the Arabic has been mangled, that a reference answer has lost its second paragraph, or that a category shows as fc-003 in one place and Redirects customer to the wrong channel in the other with no version attached.

The order in which the steps happen matters as much as the checks themselves, because a decommissioned system cannot be re-exported. The sequence that holds up is: export from A and store the export as an immutable artifact in your own object storage, import into B, run the verification script, put A into read-only mode rather than shutting it down, run both systems on the same suite for the overlap period, and only then let anyone cancel the contract. Read-only is the cheap insurance in that list — most vendors will keep an account alive on a downgraded plan for a few hundred a month, and the two months it buys are the window in which every problem in this chapter is still fixable. Cancelling on the day the import succeeds converts a recoverable mistake into a permanent one for a saving of a few hundred pounds.

What makes that ordering matter is that one step in it is a one-way door and the others are not. Import can be redone: delete the destination's records and import again with a different field mapping. Verification can be rerun as often as you like. Decommissioning cannot be undone, because after it there is no system to re-export from, and every repair in this chapter begins with a second export carrying a field the first one dropped. So the sequence is not a matter of thoroughness. It is a matter of putting the irreversible step last, which is the same discipline that applies to any procedure with one step you cannot take back.

Recovering a join after the fact#

When the identifiers are already gone, the fallback is to join on a hash of the normalised case text, which is the natural key you were told not to use. On that same 2,400-case set it recovered 2,187 cases, or 91%. The remaining 213 split into two piles: 63 cases whose text is duplicated — the same question asked by different customers, which is common in support data and means the hash matches two or three rows — and 150 cases whose text was edited at some point after the migration, or whose PII redaction pass — the step that replaces personally identifiable information, names, emails and order numbers, with placeholders before the text is stored — ran with different rules on the two sides, so the hashes simply differ. Those figures account for each other exactly: 2,400 cases less the 2,187 that matched leaves 213, and 63 duplicates plus 150 edits is that 213. Nothing is unexplained, which is what makes the residue worth reasoning about rather than writing off.

Doing the recovery properly means treating it as a data-quality exercise rather than a one-line merge. Normalise aggressively on both sides — collapse whitespace, lowercase, strip the redaction placeholders that differ between systems — then hash, then bucket by hash and inspect every bucket with more than one member on either side by hand, resolving what you can from the labels and timestamps and marking the rest unmatched. Write the resulting mapping into a checked-in file, id_recovery.v1.jsonl, with a confidence field per row, so that every later analysis can filter on it and nobody has to redo the judgement calls. The mapping is itself an artifact worth versioning, since it encodes decisions nobody will remember in six months.

A 91% join is worse than it sounds for trend work, because the 9% is not random. Duplicated questions are the most frequent customer intents, and edited cases are the ones somebody cared enough to correct, so the missing slice is biased toward the parts of the data that matter most. It is a reasonable rescue for a specific investigation into a specific case and it is not a basis for restating a year of history.

Following each of those two piles back to its cause is what shows the bias is structural rather than unlucky. A hash collides when two cases carry the same text, and two cases carry the same text when many customers asked the same thing — which is the definition of a high-volume intent. A hash misses when the text changed, and text changes when somebody corrected a reference answer or re-redacted a transcript, which happens to the cases people were paying attention to. So the recovery fails on the busiest questions and on the most-examined questions, and those are the two categories anybody investigating a regression is most likely to be holding. A 91% join that dropped a random 9% would support a trend line with a wider error bar on it. This one supports nothing, because the missing part is selected by exactly the property being studied.

What no export brings across#

Tool A's export contained case text, labels, scores, and run timestamps. Against the six artifacts an eval run leaves, that is four, and the two absentees are the ones that cannot be rebuilt: the raw unscored outputs, which A stored and did not export, and the grader definitions, which lived in A's rubric editor as versioned records that the export flattened into a rubric name. Annotator identity came through as A's internal user IDs, usr_41, usr_88, with no mapping to people, and the taxonomy arrived as display strings rather than category IDs with a version.

That loss has a consequence for reporting, and the honest handling is to stop pretending the line is continuous. Three things make the boundary survivable:

The overlap is the part teams skip and the part that does most of the work. Two tools scoring the same 206 cases in the same week gives a measured offset rather than an assumption of equivalence, and offsets of one to three points are normal, because rubric interpretation differs between judge implementations even when the rubric text is identical.

None of this is an argument against migrating. Tools improve, contracts end, and staying on something that flips eleven cases in five identical runs to preserve a clean trend line is choosing the wrong invariant. The argument is narrower: the two checks that get run are the two that cannot see the failure that occurs, so the verification is the work and the export is not. A migration where somebody wrote that script is a Tuesday. A migration where somebody compared two aggregates is a Tuesday plus a permanent hole in the record, discovered by whoever next asks whether a particular case has ever passed.

This whole chapter assumed the easy case, which is a cooperative migration with a working export endpoint, both systems running at once, and as much time as the verification needs. The version that decides whether any of this was prepared properly is the one where the vendor sends an email announcing that the service shuts down in thirty days.