A detector that was never invoked is not a detection failure

Two incident reports land on the same team in the same week, and both of them say that customer personal data reached somewhere it should not have.

The first concerns a Dutch mobile number. A support agent had pasted a transcript into the summarisation endpoint, the transcript contained the string 06-24 11 87 03, and that string arrived intact in a prompt sent to an external model provider. The redactor was called. It ran its phone recogniser, which was built around E.164 — the international format that writes every number with a leading + and a country code — and North American formats, saw a nine-digit run broken up by spaces and a hyphen with no country code, scored it below threshold, and passed it through. The transcript went out with the number in it.

The second concerns a customer's full name, and the trace of what happened is stranger. The name never reached the provider at all — the redactor removed it, exactly as designed, in the request handler. The name reached the observability vendor, because the tracing middleware records http.request.body as an attribute on the server span — a span being one timed unit of work in a distributed trace, shipped off to a monitoring backend so engineers can see where a request spent its time — and middleware runs before the handler. By the time the redactor was invoked, a copy of the raw request body had already been serialised into a span, batched, and shipped to a third-party backend with ninety-day retention and a much broader internal access list than the production database has.

Both reports are about the same category of data, and it is worth naming that category plainly before the argument goes any further. Personally identifiable information — PII, in every document you will ever be handed on the subject — is any value that identifies a particular living person, or that can be combined with other values to do so: a name, an email address, a phone number, a postal address, a national identity number, a payment card number, a date of birth. PII detection is the job of finding those values inside text that carries no markup saying where they are, using a component usually called a detector or a recogniser, which scores candidate spans of text against patterns and models. Redaction is what then happens to what the detector found: the value is deleted, or replaced by a placeholder such as [EMAIL], before the text goes anywhere else.

You meet the problem the moment a feature takes free text that a customer or an agent wrote and passes it to a hosted model API, a log, or a third-party tool, which is to say in nearly every AI feature that touches customer data. It has to be an engineering control rather than a policy document because those destinations are not yours: a prompt sent to a provider becomes subject to that provider's retention rules, a request body captured into a trace lands in an observability vendor's storage, and a transcript ingested into a retrieval corpus is copied into an index, a cache, and every backup taken since. None of those sends can be recalled once made, so the only place the problem can be solved is before the send.

The scale of the two incidents is also different in a way that is not obvious from the reports. The phone number went out once, in one prompt, in one request. The tracing middleware had been capturing bodies on every request to every route for eighteen months, which at roughly 6,000 summarisation requests a day and a ninety-day retention window means something on the order of 540,000 raw transcripts sitting in the observability backend at any moment, plus whatever the vendor's own archival tier holds. Nobody had counted that until the incident, because the middleware was working exactly as its author intended.

These two reports describe different classes of failure, and the distinction matters more than anything else in this topic, because the remediation for one does nothing whatsoever for the other. The first is a coverage failure: the detector was invoked on the payload, examined it, and did not recognise an entity that was there. The second is a topology failure: the detector was never invoked on that path, so its quality is irrelevant. A detector with perfect recall — recall being the share of the personal data actually present that the detector finds, a metric chapter 3 takes apart properly — on every entity type in every language would have changed nothing about the second incident, because a component that is not called cannot be accurate.

The shape most teams' redaction actually has#

The design that produced the second incident is not careless. It is the design that most reviews approve, because it puts the control immediately adjacent to the risk you were asked to think about.

@app.post("/v1/summarise")
async def summarise(req: SummariseRequest) -> SummariseResponse:
    clean = redactor.redact(req.transcript)
    completion = await provider.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content": PROMPT.format(text=clean)}],
    )
    return SummariseResponse(summary=completion.content[0].text)

Read as a line of code, that is correct: nothing unredacted goes to provider. Read as a statement about the process, it is much weaker, because it constrains exactly one of the ways a payload leaves the process and says nothing about the others. The request body is in memory from the moment the framework parses it, and every piece of infrastructure that touches a request in flight is free to copy it somewhere.

@app.middleware("http")
async def trace_requests(request: Request, call_next):
    body = await request.body()
    with tracer.start_as_current_span("http.server") as span:
        span.set_attribute("http.request.body", body.decode()[:4096])
        return await call_next(request)

The middleware is the one that caused the incident, and it was added eighteen months earlier by a different team debugging a malformed-payload problem. It is four lines, it is generic, it applies to every route, and nobody who reviewed the redaction design read it, because it lives in a different package and has nothing to do with the model integration.

Once you start looking for the other exits rather than the one you designed for, the list is longer than the diagram anyone drew.

One edge in that diagram is redacted. The rest carry the original, and each of them has its own retention, its own access model, and its own export path. The error tracker captures local variables in the stack frame when an exception is raised, which on a 502 from the provider means the full prompt and often the full transcript. The retry queue holds the untouched request so a worker can try again, and after five failures the message lands in a dead-letter topic that no one has ever set a retention policy on. The semantic cache stores the prompt as part of its key so that similar requests hit, which means the cache is a durable store of prompts held in a system chosen for speed rather than for confidentiality.

Placement is a property of boundaries, not of call sites#

The correction is to stop treating redaction as something a handler does and start treating it as something a boundary enforces. A boundary here is any point where bytes leave the process — an HTTP client, a log sink, a span exporter, a queue producer, a cache writer. There are usually between five and a dozen of them in a service, they are enumerable in an afternoon, and each one can be wrapped once.

class RedactingTransport(httpx.AsyncBaseTransport):
    """Every outbound body passes through here, whatever called it."""

    def __init__(self, inner: httpx.AsyncBaseTransport, redactor: Redactor):
        self._inner, self._redactor = inner, redactor

    async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
        if request.headers.get("content-type", "").startswith("application/json"):
            request = _with_body(request, self._redactor.redact_json(request.content))
        return await self._inner.handle_async_request(request)

The same shape applies to the other sinks: a log processor that runs the redactor over the formatted record before it is written, a span processor that drops or rewrites attributes on export, a queue producer that redacts before serialising. What makes this different from the handler version is not that the code is better but that the guarantee is stated over a set that is closed. You can list every transport the process constructs. You cannot list every call site a future engineer will add.

Wrapping a sink has a cost that the handler version avoids, and it is worth naming rather than discovering. Every log line and every span now passes through the detector, so a service that redacted 6,000 payloads a day starts redacting several million strings a day, most of which contain nothing. That is why the sink wrappers usually run a cheap pre-filter — a compiled regex set over the obvious high-signal patterns — and reserve the expensive model-based recognisers for payloads bound for the provider. The asymmetry is defensible because the sinks mostly carry short, machine-generated strings, and the free-text transcript is the case where recall actually has to be high.

That closure is why the topology failure gets fixed first when both incidents are open at once. Both need remediation of the disclosure itself, and the phone number that reached the provider is genuinely gone in a way we will come back to. As prevention budget, though, the topology defect is bounded work with a bounded result — you enumerate the sinks, wrap them, and every entity type is covered on every path at once. Extending the phone recogniser to Dutch formats is unbounded work with a narrow result, because it buys you one format, and the next transcript will contain a Belgian one, an internal employee number, or a national identifier written with dots instead of dashes. Fixing coverage first also leaves the span exporter shipping raw bodies while everyone congratulates themselves on the improved recogniser.

Structured fields are the easy half, and a different problem#

Everything so far has been about free text, where you have a string and no idea what is inside it. A large share of the personal data in a typical pipeline is not like that at all. It sits in typed columns — customers.email, customers.date_of_birth, orders.shipping_address — where the schema told you exactly what the field contains before a single row was read.

For those fields there is no detection problem, and treating them as one is a category error that costs both accuracy and money. You do not need a recogniser to find an email address in a column called email. You need the column not to be in the payload. The two mechanisms are projection, where the field is never selected into the object that gets serialised, and tokenisation, where it is swapped for a stable surrogate at the serialisation layer.

-- The summariser reads this view, not the tickets table.
create view ticket_for_model as
select
    t.id,
    t.created_at,
    t.product_area,
    ct.token          as customer_ref,   -- stable, meaningless outside our estate
    t.body            as body            -- free text: still needs a detector
from tickets t
join customer_tokens ct on ct.customer_id = t.customer_id;

The customer_ref token is worth dwelling on because it does something a mask cannot. It is stable per customer, so two tickets from the same person still look like the same person to a model reasoning about history, and it is meaningless outside the system holding the mapping table, so it carries no information to the provider. There is no threshold, no false negative rate, and no language coverage question, because nothing was recognised — the value was known in advance.

The residual risk in the structured half is different in kind. It is schema drift: a notes column that was meant for internal categories and now holds pasted email threads, a metadata JSONB blob that accumulated a contact_phone key last quarter, a new column added by a team that did not know the view existed. Those are caught by schema review and column-level classification rather than by a detector, and they fail silently because the view keeps working.

What this does not give you#

Redaction at the boundary is a control on what leaves the process, and it is easy to let it stand in for three things it is not. It is not access control: the raw transcript is still in your database, still readable by anyone with production credentials, and still exported by the nightly analytics job. It is not encryption, which protects data from parties without a key while leaving it fully present. And redaction is not anonymisation in the legal sense, because a record with names removed can very often still be linked back to a person through the fields that remain — which is a problem serious enough that it gets its own treatment shortly.

There is also an honest limit on the boundary approach itself. Wrapping sinks stops payloads from leaving unredacted; it does not stop a payload from being written to a location inside your own estate that you later forget about, and it does nothing about the copies made by systems you do not control, such as a managed queue's internal audit log or a load balancer's request sampling. Enumerating the sinks is a design activity, and design activities go stale. Chapter 5 is about getting evidence rather than a diagram.

The immediate open question is narrower. Wrapping every sink means running a detector over far more traffic than the handler ever did, and running it hard enough to catch a Dutch phone number means running it hard enough to flag a great deal that is not personal at all. Deciding how hard to run it requires knowing which entities you would actually be sorry to lose, and those are not equally bad. A work email address in a prompt is not the same event as a national identifier, and the next chapter builds the ranking you will need before any measurement is worth taking.