Removing PII from an index means finding every copy of the chunk

Ticket DOC-77213 was created on 14 April and ingested into the retrieval corpus the same night. It contains a customer's full name, their home address in full including the postcode, and a card number written by an agent as 4539 88XX XXXX 1187 while confirming which card the charge had gone to. The redaction step in the ingest pipeline was not applied to this corpus, because the corpus was added by a different team six weeks after the redaction work was signed off and nobody wired it in. Someone noticed on 11 August, four months later, while debugging an unrelated retrieval bug and reading raw chunk text.

The first question anyone asks is how many times it went out. The answer available at that moment is that the chunk has been retrieved into prompts an unknown number of times, and finding out is itself a piece of work that must happen before some of the cleanup, because part of the cleanup destroys the ability to answer.

Order the steps by what each one forecloses#

The ordering principle is not urgency. It is that some steps make later steps impossible, so anything that destroys information comes after everything that reads it, and anything that stops new exposure comes before everything that takes hours.

Containment first, and it is not deletion. Deleting nine chunks from a vector store, checking replicas, and forcing a compaction is a multi-hour operation, and during those hours the chunk is still retrievable and still going out in prompts. What stops the exposure in eight minutes is a denylist applied at query time: a filter on the retrieval call that drops the affected chunk ids from every result set before they reach a prompt. It is crude, it is reversible, and it takes effect on the next query.

DENYLIST = load_incident_denylist()      # chunk ids, hot-reloaded, incident-scoped

def retrieve(query: str, k: int = 8) -> list[Chunk]:
    hits = index.search(embed(query), k=k + len(DENYLIST))
    return [h for h in hits if h.id not in DENYLIST][:k]

Evidence second, while the records still exist. The retrieval logs for the four-month window give the count that the incident report needs: chunk c4f1a… appears in 212 retrievals, of which 141 produced a prompt actually sent to the external provider and 71 were served from the response cache. Snapshot those log ranges into an incident bucket before any retention job expires them, and snapshot the chunk record itself, because you will need to prove later what the text contained and you are about to delete it.

Scope third, and this is where the count grows. The document produced six chunks, of which two carry the address and one carries the partial card number. Searching the corpus for the address string finds it in three further documents — a duplicate ticket raised by the same customer, an internal wiki page about a delivery escalation that was ingested last year, and a QA fixture someone built from a real ticket. Nine chunks in total, from four documents, in two corpora.

Only after those three steps does deletion start, and the surface it has to cover is the diagram rather than the vector store.

Three rows in that table are worth dwelling on because they are the ones teams skip. The Postgres row carries the point-in-time recovery log from the previous chapter, so the fourteen days of residual there is the same fourteen days; the warehouse row's time travel is the equivalent facility in a columnar store, which keeps superseded partitions queryable for a week after a rewrite.

Vector stores usually tombstone. A delete marks the vector as removed from search results and leaves the encoded vector and its payload in the immutable segment on disk until a merge rewrites it, which happens on the store's own schedule and can be hours or days out. The chunk is no longer retrievable, which satisfies containment, and the text is still on a disk and still in whatever snapshot that disk is included in. Forcing a compaction is a specific operation with a specific cost — on a 214,000-chunk index it ran for six hours and roughly doubled read latency while it worked — and it needs to be requested explicitly and confirmed.

Evaluation sets are the copy that gets forgotten, and they are the reason lineage matters here rather than only for reproducibility. The chunk was sampled into two versioned eval sets, and the lineage records for those sets are what tell you so without reading their contents. Re-cutting them from the cleaned corpus changes the sets, which means comparisons across the version boundary are no longer apples to apples, and the honest response is to record the discontinuity alongside the new version rather than to pretend the old numbers still line up.

Backups are the row where you stop being able to act. There are 122 nightly snapshots and 2 quarterly archives containing the chunk, none of them rewritable without a restore-scrub-resnapshot cycle that most organisations will not authorise for one record and that carries its own risk. The standard answer is to accept the residual, document it, put a scrub step into the restore runbook so that a restore does not silently reintroduce the data, and let the 35-day nightly retention expire. The quarterly archive with a seven-year retention is a genuine, dated, unrecallable exposure, and it goes in the register with that description.

{
  "incident": "INC-2244",
  "subject_ref": "cust_88214",
  "unrecallable": [
    {"system": "quarterly archive", "copies": 2, "expires": "2033-04-30",
     "compensating_control": "restore runbook scrub step, access log alerting"},
    {"system": "external provider", "prompts": 141, "window": "2026-04-14/2026-08-11",
     "note": "deletion filed 2026-08-12; abuse-monitoring copies excluded per DPA s.7.3"}
  ]
}

That register is the deliverable. Everything else on the list ends in a confirmation; these two rows end in a description, and a remediation that produces no such register almost certainly did not look hard enough.

Whether content-derived chunk ids help you here#

The retrieval logs in this incident recorded chunk ids and not chunk text, and the ids are derived from the content — sha256(chunk_text), truncated. That choice was made for deduplication, and it cuts both ways in a remediation.

It helps in three concrete ways. The 212-retrieval count was computable at all because the id in the log is stable and resolvable, and it stayed computable after the chunk was deleted, since the id in the log does not depend on the chunk still existing. Identical text ingested into a second corpus produced the identical id, which is how the wiki-page copy was found in minutes rather than by a fuzzy text search across two stores. And the log itself is far lower-sensitivity than a log of chunk text would have been, which is why the four months of retrieval history existed to be read: nobody would have retained four months of raw retrieved passages.

It hurts because a content-derived id is a hash of personal data, and a hash of personal data is pseudonymised personal data rather than anonymous data. Anyone holding the log can test a guess: compute sha256 over a candidate chunk text and see whether the id matches. For a chunk of prose the candidate space is large, so this is not a practical inversion, and for short structured content it is trivial — a chunk whose text is a formatted address in a known city has few enough plausible variants to enumerate. The log therefore stays in scope after the chunk is gone, which means the erasure request covers it too and the "we only log ids" claim in the security questionnaire was too strong.

The fix is the same construction chapter 4 used for surrogates: derive the id with a keyed HMAC rather than a bare digest.

def chunk_id(text: str) -> str:
    return hmac.new(CHUNK_ID_KEY, text.encode(), hashlib.sha256).hexdigest()[:16]

Determinism inside your estate survives, so deduplication and the retrieval-log join both still work. The offline verification oracle does not, because an attacker with the log and a perfect guess cannot recompute the id without the key. What you take on in exchange is key management: rotating CHUNK_ID_KEY changes every id in the system, which means a full reindex and a break in the continuity of your retrieval logs, so rotation becomes an event you schedule rather than a hygiene task.

Closing the loop rather than closing the incident#

The last step is the one that determines whether this happens again, and it is not a control. The corpus was added by a team who did not know the ingest redaction existed, which is a discoverability failure: the redaction was a step in one pipeline instead of a property of the ingest interface. The durable fix is that a corpus cannot be registered without a policy binding — an entry in the same policy file that carries the tiers and the thresholds — and that ingestion refuses to run for a corpus with no binding. Then a new team's first attempt fails loudly on day one instead of succeeding quietly for four months.

You now have the pieces this topic exists to assemble: a placement argument that puts redaction at boundaries, a harm ranking that says which entities earn which treatment, a measurement that produces per-entity recall on your own traffic with an interval attached, a surrogate scheme that keeps the corpus usable, a canary that produces evidence instead of assurance, and a data map whose last two rows you cannot undo. The way to start is to pick the pipeline that already sends the most free text to a provider, spend an afternoon enumerating its sinks, and run one canary through it. Whatever that canary finds in the first hour will tell you which of these chapters you need next.