Masking that preserves identity keeps a trajectory followable

The deadline is four weeks and the instruction is one sentence: customer personal data stops being written to the trace store. It is not an unreasonable instruction. The store holds ninety days of spans, forty-one people have dashboard logins because everyone who might be on call needs one, and there is a Slack integration that posts a link and a preview snippet into a channel with contractors in it.

The objection came from support, with a case rather than a principle. A customer had reported that the agent kept telling them their account had no open shipments when it plainly did. The run that explained it went like this: turn one, the agent called lookup_account and got back account ACCT-4471902. Turn two, it called lookup_shipments and passed 4471902 — correct. Turn three, after the user mentioned a different order, it called lookup_shipments again and passed 4471920, two digits transposed, which is a valid-looking account number belonging to nobody. The tool returned an empty list, successfully, and the agent reported no shipments. Finding that took ninety seconds and consisted entirely of reading one number down a column of nine spans and noticing where it changed. Under a policy that removes the number, the same investigation is nine spans that each say a lookup happened and a support engineer with nothing to compare.

Both requirements are real and they are not actually in conflict, because they are about different properties of the value. Legal cares that the value is readable; support cares that it is the same value.

Stability and type are the properties worth preserving#

The technique is deterministic pseudonymisation with a keyed hash: every occurrence of a given entity is replaced by a placeholder derived from that entity, so the same input always produces the same output, everywhere, in every span, across every trace. ACCT-4471902 becomes ACCT-8FQ2K1MB in the tool input, in the rendered prompt, in the completion, and in the retrieval filter, every time it appears, this week and next month. 4471920 becomes something else entirely, because it is a different value — which is exactly the observation the investigation above turned on.

Two properties do the work, and it is worth separating them because implementations usually get one and miss the other. Stability is that the mapping is a function: same input, same placeholder, no randomness, no per-span salt. Type preservation is that the placeholder announces what kind of thing it replaced and keeps a recognisable shape — ACCT- prefixes an account, ORD- an order, <addr:…> a postal address. Stability alone gives you a soup of interchangeable blobs that you can compare but not interpret. Type alone gives you readable labels you cannot follow.

This is also the honest reframing of the keyed-hash proposal that always appears alongside it. A keyed hash is the primitive here; what separates "hash everything" from field-aware masking is not the hash function but what you feed it. Hash the whole tool.output string and you get one opaque token for a JSON document, useless because that document is never byte-identical twice. Extract the entities first, hash each one under its type, and put the results back where they came from, and you have preserved both the structure of the document and the identity of every value in it.

The step nobody budgets for is normalisation. The agent wrote 4471902 where the tool returned ACCT-4471902, and a masker that hashes the literal strings gives those two occurrences different placeholders — failing precisely in the case support brought to the meeting. Each entity type needs a canonical form computed before hashing: strip the prefix and leading zeros from account numbers, lowercase and trim emails, collapse whitespace and casing in addresses. Over-normalise and you collide instead: reduce everything to digits and order ORD-4471902 masks to the same token as account 4471902. Putting the type into the hash input alongside the normalised value prevents that, and costs nothing.

import base64, hashlib, hmac, re

KEY = load_masking_key()          # from KMS; never in config, never in the image

NORMALISE = {
    "account": lambda v: re.sub(r"\D", "", v).lstrip("0"),
    "order":   lambda v: re.sub(r"[^0-9A-Z]", "", v.upper()),
    "email":   lambda v: v.strip().lower(),
    "address": lambda v: re.sub(r"\s+", " ", v.strip().lower()),
    "phone":   lambda v: re.sub(r"\D", "", v)[-9:],
}
FORMAT = {
    "account": "ACCT-{t}",
    "order":   "ORD-{t}",
    "email":   "u_{t}@masked.invalid",
    "address": "<addr:{t}>",
    "phone":   "<tel:{t}>",
}


def mask(kind: str, value: str) -> str:
    canonical = f"{kind}:{NORMALISE[kind](value)}".encode()
    digest = hmac.new(KEY, canonical, hashlib.sha256).digest()
    token = base64.b32encode(digest).decode("ascii")[:8]
    return FORMAT[kind].format(t=token)

Eight base32 characters is 40 bits, so a collision between two distinct values of the same type becomes likely somewhere around a million distinct values — comfortable for account numbers, worth widening to twelve characters for anything at the scale of email addresses across every tenant.

What the alternatives cost#

Of the two proposals on the table, dropping content capture is the worse one, and the reason is asymmetry of recovery. Keyed hashing on a structured identifier field preserves linkage: you cannot read ACCT-4471902, but you can see that spans three, five and nine carry the same token and span seven carries a different one, which is 80% of the investigation that opened this chapter. Dropping the field preserves nothing, and there is no query, no join and no clever aggregation that recovers it afterwards — the data was never written.

Where the two become equivalent is on prompts and completions, which are prose containing entities rather than single entities. There, hashing the field gives you one token per unique document, every document is unique, and you have paid full storage to keep a column of distinct meaningless strings. That is the specific case where "we hashed it" and "we deleted it" are the same decision with different bills.

Where it runs, and why the collector is the wrong place#

The masking pass belongs in a span processor inside your process, ahead of the batch processor that queues spans for export. A span processor is the hook every OpenTelemetry SDK exposes for code that runs as each span starts and ends, before the span is handed to anything else, which makes it the last point at which a value can be rewritten while it is still only in your memory. Two reasons put it there rather than in the collector. The exporter's queue is already outside the boundary in any sense that matters — it is memory the operator can dump, and with persistent queueing enabled it is a file on disk — so a raw payload that reaches it has left the place where the policy is enforceable. And the processor is the only component that knows your schemas: it can be told that lookup_account returns an object whose account_id, billing_address and contact_email fields are entities of known types, which turns most of the work into a JSON-path walk rather than a guess.

That schema-driven pass is fast and exact, and on this workload it covers about 87% of entity occurrences at roughly 0.2 ms per span. The remainder lives in prose — the user's own message, a free-text note field on an order, an OCR'd document body — and needs an entity recogniser (a model that reads free text and marks the parts of it that are names, addresses, phone numbers and the like), which is probabilistic and runs at a few milliseconds per kilobyte. On 9 kB prompts that is around 55 ms, which is why the pass belongs on the export path, in the batch processor's worker thread, rather than inline where it would land on the user's latency.

The collector still gets a pass, with a different job. A regex scan there for account-shaped digit runs, email addresses and card-like numbers is a tripwire: it should find nothing, and when it does find something it raises an alert naming the span and the attribute so somebody fixes the schema mapping. It is a detector of escapes, not the policy — a distinction worth writing into the runbook, because a tripwire that quietly redacts instead of alerting will hide the gap it exists to reveal.

A regex scrubber fails in both directions at once#

Running the regex pass as the policy is the standard proposal and it is worth being specific about why it does not hold, because the failure has two independent halves that get blurred together.

The detection half misses. A pattern for account numbers finds account numbers; it does not find the address a customer typed into a free-text note, a surname, a passport number in a format nobody wrote a rule for, or the same account number written with spaces because that is how it appears on the invoice. Free text inside tool results is where most of the leakage actually lives, and it is the category patterns are worst at. Meanwhile the same patterns fire on things that are not personal data: a rule for 13-to-19-digit runs eats shipment tracking numbers, and a rule for eight-digit account numbers corrupts order ids and version strings that happen to share the shape. You get a store that still contains personal data and no longer contains reliable identifiers, which is the worst available combination.

The substitution half is separable and is the one this chapter is about. Replacing every match with a fixed [REDACTED] destroys the relation between occurrences, so one customer becomes an identical blob in every span and a different customer becomes the same identical blob — at which point the trajectory is no longer followable and the transposed digit that explained the whole incident is invisible. Note that these two halves fail independently: a regex detector paired with mask() would at least keep linkage on the entities it does catch. Detection and substitution are separate design decisions, and most implementations get both wrong for the same reason, which is that they were written as a single re.sub.

Rotation, erasure, and the ninety days already in the store#

Rotating the masking key changes every placeholder, so a value masked before the rotation and after it looks like two different values. The linkage property is only as long-lived as the key. In practice that means rotating rarely, holding the key in a KMS — a key management service, which stores the key and performs operations with it so that no copy ever sits in a config file or a container image — with an access list shorter than the dashboard's, and treating a key compromise as a re-identification incident that is handled by deleting the affected window rather than by re-masking it. Aligning the rotation period with retention keeps the damage bounded: with ninety-day retention and a ninety-day rotation, at most one retention window ever spans two keys.

Determinism pays for itself once more in a place people do not expect. When a subject-erasure request arrives, you have the person's real values in the source system, so you can compute their placeholders and delete exactly the spans that contain them. Under a random-salt scheme, or under fixed-token redaction, that request is unanswerable except by deleting a time range.

The traces already in the store are the part of this project that has to be scheduled first, because masking new spans does nothing about old ones and the deadline covers both. Spans are immutable in every backend worth using, so the choices are to delete by time range — usually possible at index or partition granularity — or to export, mask and re-ingest into a fresh index and then delete the original. The second costs days of engineering, doubles storage during the migration, and still requires the deletion at the end. Delete the window, keep the derived metrics, which contain counts and durations and no customer data and are therefore untouched, and shorten retention so that the next policy change is a thirty-day problem rather than a ninety-day one.

What this buys, and what it does not#

Stable pseudonyms are pseudonymisation, and pseudonymised data is still personal data. The linkage that makes a trajectory followable is a persistent identifier for a person, which is precisely the property that keeps the store in scope for access requests, erasure requests and the retention policy. What changes is exposure: forty-one people can no longer read a customer's address, and — because the hash is keyed rather than bare — none of them can recover a low-entropy field like a postcode or a date of birth by enumerating candidates, which is the failure that makes unkeyed hashing a false comfort. Presenting this to a security team as "reduced and access-controlled" is defensible; presenting it as "anonymous" is a claim that will not survive the first person who asks what happens if someone has both the store and the key.

There is also a category of debugging that does not come back. Questions about what the customer said — whether the phrasing contained an injection attempt, whether the complaint mentions a second shipment, why the model concluded the package was damaged — need the words, and the words are entities-and-prose in a ratio nobody controls. The short-retention full-fidelity tier is where those questions get answered, and it is the reason to keep one even after masking works.

What the six chapters here add up to is a specific set of things being true of a request at the moment somebody asks about it: one span per model call carrying the resolved model, the parameters, the rendered prompt and the completion; a trajectory whose parent-child edges survive whatever the executor does; a sampling policy that decides after the interesting part rather than before; semantic outcomes recorded as attributes rather than forced into statuses; and entities that read the same everywhere so a person can be followed through a run without being exposed by it. The next thing to do with all of that is to stop reading traces one at a time — to turn these spans into the latency, cost and quality series that tell you how often the 45-second case fires, which tenants see the empty-retrieval case, and whether last week's fix moved anything at all.