Pseudonymising an eval case

A captured eval case contains whatever the customer wrote and whatever the system retrieved on their behalf: names, email addresses, street addresses, order ids, sometimes a photograph. It then sits in a repository that runs in CI, gets cloned onto laptops, and is readable by everyone who touches the assistant, which is a materially wider audience than the production data store it came from. Something has to be done about that before the set can exist at all, and the obvious something is a scrubber.

This chapter is about why the obvious version destroys the set and what replaces it. Pseudonymisation substitutes each real identifier with a fake one of the same shape, standing in the same relationships to everything else, applied consistently everywhere that identifier appears rather than blanked out where it is found. The chapter compares it with the three alternatives that get proposed in the same meeting, sets out the three stages of a substitution pass and the two design decisions inside it that get argued over, is precise about what the design buys legally — because teams routinely overclaim here — and covers the mechanical verification that has to run on every row. It ends on the residue: the cases no surrogate can save, and the measurable ways a surrogate world perturbs the model.

Substitution, not deletion#

Take the multi-account case from the previous chapter. Julien Moreau becomes Adrien Fabre; Juliette Moreau becomes Adrienne Fabre, chosen so that the near-collision survives; j.moreau@example.net becomes a.fabre@example.net; ORD-77120 becomes ORD-41908, which passes the same format check the real one did; 14 Rue des Lilas, 69003 becomes 26 Rue des Peupliers, 69003, keeping the postcode's city because retrieval and the shipping tool both key on region. Every one of those substitutions is applied to the conversation, to the retrieved documents and to the tool fixtures in the same pass.

Four approaches get proposed whenever this comes up, and they differ on properties that matter unevenly:

Underneath those rows sit three properties, and naming them makes the table something you can predict rather than something to memorise. The first is that distinct real entities must map to distinct surrogates. Redaction is the only method in the table that fails it, and it fails it completely: every name goes to one symbol, so two account documents that differed only by a name come out as the same bytes and the relationship that made the case a case cannot be recovered from what survives. The second is that a surrogate must satisfy the same format rules the real value did, because the system under test validates its own inputs and will reject a value that does not look like the thing it is standing in for. The third is that wherever the similarity between two values was load-bearing, the surrogates have to be similar in the same way. Every row of the table follows from which of the three a method keeps, and what pseudonymisation is for is keeping all three at once.

Keyed hashing is the answer that sounds rigorous and is the one to watch for, because it does preserve joins and it is genuinely irreversible without the key. It fails on the two rows in the middle. An HMAC digest — the output of a hash function combined with a secret key, so that only the key holder can reproduce it — where an order id belongs is a 64-character hex string, so the assistant's own validator rejects it before any lookup happens and the case fails for a reason that has nothing to do with the behaviour under test. And 9f2c... and 4b81... are not similar names, so a multi-account case loses the exact property it was built to exercise. Hashing is designed to destroy similarity; that is its purpose everywhere else, and here similarity is the payload.

That last point is worth dwelling on, because keyed hashing satisfies the first of the three properties perfectly, and the first property is what most people have in mind when they say a case must survive sanitisation. Distinct inputs give distinct digests, so every join inside the case still joins and nothing collapses the way it does under redaction. What a digest cannot do is be shaped like the thing it replaced, and it cannot be nearly-equal to another digest, because a hash function is built precisely so that two inputs one character apart produce outputs with no visible relationship between them. The property that makes hashing the right way to store a password is the same property that makes it the wrong way to sanitise an eval case, and it is the same property in both places rather than a coincidence of two different ones.

Doing the substitution so it holds together#

Sanitisation belongs at capture time, in the pipeline that writes a row into the eval store, so that no unsanitised row is ever written down. Doing it as a later batch job over an existing store means the raw data has already been in the repository, in a backup, and possibly in someone's notebook.

The pass has three stages. Detection finds the entities in the trace, using pattern matching for structured identifiers and a named-entity model — one trained to pick out people, places and organisations in running text — for the free-text names that appear in sentences like "this is Julien again, third time asking". Allocation assigns a surrogate to each distinct real entity within the case, preserving declared relations between them. Rewriting applies the mapping to every string in the row — messages, retrieved document text, tool arguments, tool responses, and the ground-truth criteria, which is the field teams most often forget and which is why a sanitised case sometimes gets scored against a criterion naming an account id that no longer exists.

from dataclasses import dataclass

@dataclass(frozen=True)
class Entity:
    kind: str            # "person" | "email" | "order_id" | "address" | "postcode"
    value: str
    relations: tuple = ()  # e.g. ("same_surname_as:e2", "same_address_as:e2")

def allocate(entities, rng):
    """One surrogate per real entity, consistent within the case,
    with relations between entities preserved by construction."""
    mapping, households = {}, {}
    for e in sorted(entities, key=lambda e: (e.kind, e.value)):
        if e.kind == "person":
            surname = _pick_surname(e, households, rng)      # shared across a household
            given = _pick_given_name(e, households, rng)     # near-collision if the real ones collide
            mapping[e.value] = f"{given} {surname}"
        elif e.kind == "email":
            person = mapping[e.relations_owner]              # derive from the allocated name
            given, surname = person.lower().split()
            mapping[e.value] = f"{given[0]}.{surname}@{e.value.split('@')[1]}"
        elif e.kind == "order_id":
            mapping[e.value] = _valid_order_id(rng)          # matches ^ORD-\d{5}$ and the checksum
        elif e.kind == "postcode":
            mapping[e.value] = e.value                       # region is load-bearing; keep it
        else:
            mapping[e.value] = _surrogate_address(e, households, rng)
    return mapping

Two design decisions inside that function are worth arguing over. Surrogates are allocated per case rather than globally, so the same real customer appearing in two cases receives two different fake identities; a global mapping would let anyone with the set link a person's behaviour across cases, which is precisely the linkability you were trying to remove. What re-identification needs is not a name but the equality relation — the knowledge that the person in one case is the person in another — and a global mapping preserves that relation exactly, since one real customer would always receive one fake identity. Anyone holding the set could then group cases by pseudonym and assemble a behavioural profile without ever learning whose it was, and a profile is most of the way to a name. A per-case mapping breaks the relation at the case boundary, and breaking it is the whole point. The cost is that a scenario spanning several conversations by one customer has to be captured as one case, not three. And the postcode is deliberately not substituted, because the shipping tool's response depends on the delivery region and swapping Lyon for Lille would change the correct answer. Each field is a judgement about whether its value or only its identity is load-bearing, and that judgement has to be recorded in the field policy rather than left to whoever wrote the regex.

The mapping itself goes into a separate store — a table in an encrypted, access-controlled database that the eval repository has no credentials for — and it exists so that you can go back to the original trace when a case behaves strangely, and so that fixtures can be re-derived when a source document changes. It is worth being precise about what this buys you legally: under the GDPR, the European Union's General Data Protection Regulation, pseudonymised data is still personal data as long as a mapping exists somewhere, so holding the key means the eval store remains in scope. What the design achieves is that the eval store on its own cannot re-identify anyone, which is a real and meaningful reduction in exposure and is not the same thing as being out of scope. Teams that want the stronger claim delete the mapping after capture, and accept that they can never trace a case back to its origin.

The claim to make in a review is correspondingly narrow, and it is worth rehearsing before somebody else words it for you: the eval store cannot re-identify anyone on its own, the mapping lives in a system with its own access control, and the store stays in scope for as long as that mapping exists anywhere. The step to make somebody demonstrate is the middle one. Who holds the key, and what stops the credentials sitting in the eval repository's CI configuration from reaching the mapping store? That is the step the whole design rests on, and it is the step an architecture diagram tends to assert with an arrow.

Verification has to be mechanical, because detection is the weak link and a missed entity is a leak. Two checks run on every sanitised row before it is committed: a leak check asserting that no string from the case's detected-entity list survives anywhere in the serialised row, and a format check asserting that every surrogate satisfies the same validator the production system applies to that field.

def assert_clean(raw_entities, sanitised_row_json):
    for e in raw_entities:
        assert e.value not in sanitised_row_json, f"leak: {e.kind}"
    for oid in ORDER_ID_RE.findall(sanitised_row_json):
        assert luhn_ok(oid), f"surrogate order id fails production validator: {oid}"

Detection is where this design fails when it fails, because the leak check can only assert the absence of entities that detection found. The logic is visible in the two lines of the check itself: it iterates over raw_entities and asserts each one is missing from the row, so its domain is whatever detection produced and its guarantee is conditional on that domain being complete. A green leak check is proof that no detected entity survived, which is a materially weaker statement than the one people read off it, and the distance between the two statements is exactly the detector's miss rate. The pipeline cannot be made to verify its own recall either, because a mention it never saw is a mention it cannot go looking for.

Structured identifiers are easy — an email, an order id and a postcode all have patterns, and pattern matching finds them at essentially perfect recall. Names in free text are the hard case, and a good named-entity model runs at around 0.94 recall on conversational text — recall being the share of the mentions that are really there which the model finds, so 94 in every 100 — which over 147 traces containing perhaps 600 name mentions means roughly 36 mentions survive. That count is one multiplication: finding ninety-four in every hundred means missing six in every hundred, and six in every hundred of six hundred mentions is thirty-six. Recall rather than precision is the figure to quote here, because the two errors are not comparable. A false positive substitutes a surrogate for something that was never personal data, which costs a little realism and is caught by review. A false negative leaves a real customer's name in a repository that CI can read. One of those is a defect and the other is an incident, and a detector tuned to look good on a combined score is being tuned on the wrong axis.

Most of the surviving mentions are second references: "Jules said the same thing last week", a signature block, a name inside a quoted email. The mitigation is layered rather than clever. Run detection over the raw trace and independently over each fixture, seed the entity list from the account record joined to the conversation so that the customer's known name and email are searched for literally rather than discovered, and have a human review the first thirty sanitised rows in full before trusting the pipeline on the rest.

Not everything in a trace should be substituted, and over-sanitising costs the same kind of damage as redaction. Policy documents, product names, SKUs, carrier names and public help-centre articles are not personal data, and replacing them breaks retrieval matching and changes the correct answer. The field policy should be an allowlist of what gets substituted rather than a denylist of what is spared, so that a new field arriving in the trace format defaults to being left alone and reviewed, rather than defaulting to being mangled by a pattern that half-matches it.

The residue a good design still leaves#

The 147 cases came out the far side as 138. Nine were dropped because no surrogate could preserve what made them interesting: a customer disputing a charge by quoting a bank statement whose merchant descriptor encoded the real merchant id, two cases turning on a genuinely unusual name that the assistant had mishandled because it was unusual, a conversation about a specific news event affecting one customer's delivery, and a handful where the trace contained a photograph of a damaged item. Those nine are gone from the set and the failure modes they represented are not, which means the honest record includes a list of dropped cases with the reason for each, so nobody later concludes those behaviours are covered.

Surrogates also perturb the model, sometimes measurably. Names drawn from a small fixed list start to read as fictional after a while, and there are documented cases of assistants becoming more cautious with inputs that feel like test data. Tokenisation shifts too: a name that is two tokens in the original and five in the surrogate changes the position of everything after it in a long context, which is enough to move a borderline case. Neither effect is large and neither is superstition, and neither is a reason to distrust substitution. They are the reason the acceptance test in the scenario above is not optional: a case whose verdict is stable when the names change was never resting on the names, and a case whose verdict moves is telling you something you needed to know before it went into a gate. In the rebuild described above, six of the 138 sanitised cases produced a different verdict than the original production trace had, and unpicking them found four where the surrogate name collided with a well-known fictional character and one where a rewritten address landed on a street the retrieval index had a stale document for.

The human review of the first thirty rows found four missed mentions in ninety minutes, all of them nicknames, which is the recall figure above showing up in practice rather than in a benchmark. That review is not optional and it does not scale; what makes it affordable is that it runs once per policy change rather than once per case.

The direction this is heading is worth knowing about even though it is not yet the default. Several teams now generate the surrogate world with a model rather than a substitution table, asking it to rewrite the conversation and its documents into a consistent fictional scenario that preserves the structural properties a policy file names — the near-collision, the missing identifier, the ordering of retrieval scores. It reads more naturally than table-driven substitution and it drifts, quietly changing a detail the ground truth depended on, so every rewritten case needs a human to confirm the criteria still apply. That review takes about three minutes a case, which is affordable for a set of this size and is the reason the technique is spreading.

What separates a sanitised set from a set you can defend is that the second one was tested. Substitution preserves the joins, the formats and the near-collisions that redaction destroys, and it does so at the cost of a mapping store, a field policy that says which values are load-bearing, and a leak check on every row. None of that tells you the cases still work. Only replaying them against their own recorded outcomes does, and the nine cases that had to be dropped and the six that changed verdict are the honest output of running that comparison rather than evidence the design failed.

A sanitised, replayable set of 138 cases is now a real artefact, and it still has no verdicts on most of it. Two engineers have been labelling in parallel to get through them in a day, and the moment their labels are compared, the set stops being a data problem and becomes a disagreement.