Deleting the row leaves the vector, the cache, the trace and the backup

A user of a health-adjacent scheduling assistant asked, in writing, that the assistant forget that she was undergoing fertility treatment. Support ran the deletion, the primary row went, the confirmation email said the information had been removed, and the memory screen from chapter 7 showed nothing. Nine days later she asked the assistant to move a Thursday appointment and it replied that it had avoided the morning slot "since your clinic appointments are usually early." The sentence had never been in the deleted record. It was in a compacted summary written six weeks earlier by the job from chapter 3, which had folded four memories into one durable statement, and the summary had its own row id and no link back to what it came from.

The vector for the original record was also still in the index, because the delete had gone to Postgres and the index was a separate system with its own client. It had stopped being reachable through the primary store and had not stopped being reachable through similarity search, which is the retrieval path the agent actually used.

Why deleting by id could not have worked#

Both halves of that failure follow from properties the system already had, and walking them is what turns "delete the derived copies too" from advice into a design.

Take the summary first. The compaction job reads a set of memory rows and writes one new row containing a condensed statement. The new row is a row in the same table: it has its own id, its own timestamp, and its own embedding, and nothing about it records which rows it was condensed from. So after the job runs there are two independent carriers of the same fact, related to each other by an event that happened in the past and left no trace. A delete addressed to the original id can therefore reach the original and nothing else, no matter how carefully it is implemented, because the information needed to find the second carrier does not exist anywhere in the system. This is the same constraint chapter 7 hit with provenance, arriving from the other side: a relation that is only true at the moment of writing has to be recorded at the moment of writing.

The second half is simpler and more common. The delete was a statement against Postgres. The vector index is a different process with a different client and its own copy of the record's text alongside the embedding, and no statement issued to one database removes anything from the other. What made this consequential rather than untidy is which store the agent reads from. Retrieval runs as a similarity search against the index, so the index is the authoritative source for what the agent can recall, and the primary table is the store that stopped serving. Deleting from the table removed the record from the screen and left it in the only place that was feeding the model.

Put the two together and the shape of the failure is that the observable moved and the danger did not. What the user, the support agent and the confirmation email were all watching was the primary row, and the primary row count went to zero exactly as expected. The quantity that actually determined whether the fact was gone is the number of stores still able to serve it, and that number varies with how long the record had been alive and how many jobs had touched it — six weeks in this case, long enough for one compaction cycle and every re-embedding pass in between. A record deleted an hour after it was written would have produced a clean result on the same code, which is why this class of bug survives testing.

Every copy the fact made on its way through the system#

A memory record does not stay a single row. It is embedded, cached, summarised, logged, replicated and snapshotted, and each of those produces a copy with its own lifecycle, its own owner and often its own retention policy set by someone who has never thought about deletion.

Most of that diagram is a fan-out from one node, and a fan-out is easy to handle: enumerate the arrows, call each target. The two edges that are not from P are the ones that make it a graph rather than a list. T --> Q says a copy that has already left your system spawns another copy outside it, at a provider whose retention you configure rather than control. S --> V is the edge that produced the nine-day failure, because a derived record is a first-class record that gets embedded like any other, so the derived row acquires its own derived copies and the depth of that chain is not fixed — a summary can be compacted into a later summary, which is embedded in turn. Deletion has to traverse this graph rather than the single node at its root, and traversal rather than enumeration is exactly what the recursive query below is for.

Splitting the table into rows you can purge and rows that only age out is the substance of the design, and the split is visible in the second column rather than the fourth. Five rows name an action — delete by id, lineage lookup, targeted eviction — and four name a clock. Purging is a positive action you take and can verify. Ageing out is a promise about a clock, and the only thing that makes it credible is that the clock is short, documented, and enforced by configuration you can point at. Read the third column against that split and the gap is the whole exposure profile: everything in the purge half completes within minutes, and everything in the age-out half runs to weeks. A design decision worth taking deliberately is therefore which side of the split each store sits on, since moving a store from the age-out half to the purge half is usually a matter of building a delete path rather than of accepting a longer promise.

A deletion is a workflow with a receipt, not a statement#

Because the fan-out crosses systems that fail independently, the delete cannot be a sequence of calls inside a request handler. It is a durable workflow, keyed by the receipt id from chapter 7, retried per target, with per-target status the user can see.

async def delete_memory(memory_id: str, actor: Principal) -> Receipt:
    record = await store.get_by_id(memory_id)
    if record is None or record.org_id != actor.org_id:
        raise ToolError("No such memory.")          # chapter 4's resolve-then-check

    async with db.begin() as tx:
        await tx.execute("SET LOCAL app.current_org = :o", {"o": actor.org_id})
        # The tombstone outlives the row and is what makes restores safe.
        await tx.execute(
            "INSERT INTO memory_tombstones (memory_id, org_id, subject_user_id, "
            "content_hash, requested_by, requested_at) VALUES (:m, :o, :s, :h, :a, now())",
            {"m": record.id, "o": record.org_id, "s": record.subject_user_id,
             "h": sha256(record.text), "a": actor.user_id},
        )
        descendants = await tx.fetch(
            "WITH RECURSIVE lineage AS ("
            "  SELECT id FROM memories WHERE id = :m "
            "  UNION ALL "
            "  SELECT m.id FROM memories m JOIN lineage l ON l.id = ANY(m.derived_from)"
            ") SELECT id FROM lineage", {"m": record.id})
        await tx.execute("DELETE FROM memories WHERE id = ANY(:ids)",
                         {"ids": [d.id for d in descendants]})

    await workflow.start("purge_derived", receipt_id=..., ids=[d.id for d in descendants])
    return receipt

Two things in there carry the weight. The derived_from array column is the lineage that the scheduling assistant did not have, and adding it is the difference between a delete that reaches the compacted summary and one that leaves it behind — the compaction job from chapter 3 must record its inputs, which costs one array column and nothing else.

The recursive query is what turns that column into a traversal, and it is worth reading rather than skimming, because the recursion is what handles the S --> V edge from the diagram. The first branch selects the record the user clicked. The second branch selects any row whose derived_from array contains an id already in the result, and UNION ALL feeds those rows back into the same condition, so the query keeps going until a pass adds nothing. That is the mechanism that reaches a summary of a summary: the recursive query finds the whole subtree, because summaries get summarised and a fact can be two generations removed from the row the user clicked. A single join would find only the first generation, and a fixed number of joins would find a fixed number of generations, whereas the depth of the chain is set by how many compaction cycles a record has survived and is not known when the query is written.

The tombstone is the other, and it is a small marker row written in place of the record that was removed, saying that a fact with this identity used to exist and has been deleted. It stores the id, the tenant, the subject and a hash of the content, deliberately not the content itself, so the deletion record does not become a copy of the thing that was deleted. A hash is enough for the three jobs it has to do and useless for reconstructing the text, which is exactly the property wanted from a row that will outlive everything else. It outlives the row on purpose, and it is consulted in three places: by the retrieval path as a suppression filter, so a straggler in the vector index cannot be returned even if the purge job has not run; by the restore procedure, which replays the tombstone log against any snapshot before the restored database serves traffic; and by the audit trail, which needs to show that a request was made and honoured.

Choosing to delete descendants outright rather than recompute them is a decision worth taking explicitly. Deleting is safe and loses the other facts that went into the summary; recomputing from the surviving parents preserves them and costs a model call plus the risk that the regenerated summary re-derives the deleted fact from a neighbouring record that still implies it. Recomputation with the deleted content hash checked against the output is the thorough version, and deletion is the version most teams should ship first.

Telling the user what is true#

The honest statement is short, and writing it before you build the workflow is a good way to discover which rows in the table you have not solved. The scheduling assistant's revised confirmation read: "We have deleted this from the assistant's memory and from everything it searches, including summaries built from it. Copies may remain in diagnostic logs for up to 30 days and in encrypted backups for up to 35 days; these are not searchable by the assistant, and if we ever restore a backup, the deletion is reapplied automatically before the system serves traffic. Our model provider may retain the original request for up to 30 days under its own policy."

Each clause maps to a row of the table, which is what makes it defensible rather than reassuring. The backup position — that snapshots may age out rather than being surgically edited, provided restores re-apply deletions — is the mainstream regulatory reading of erasure rights, and it holds only because of the tombstone replay. A team that promises the same wording without implementing the replay has written a sentence that is false the first time they restore.

The row with no mechanism at all is the last one. If memory content ever enters a fine-tuning corpus — the dataset a base model is trained further on so that it behaves the way a particular product needs — deletion becomes structurally unachievable, since the fact is distributed across weights and no retraining schedule you can afford makes that promise real. That row is different in kind from the age-out rows above it, and the difference is worth stating plainly: an age-out row has a clock that eventually reaches zero, and this one has no clock at all. The practical policy is to exclude memory content from training corpora entirely and to say so in the same paragraph, because it converts an unfixable problem into a boundary you never cross. Provider-side logging is the neighbouring case with a better answer: zero-retention or short-retention modes exist on major APIs, and turning one on removes a row from your table that you otherwise have to explain.

Knowing the deletion worked#

A workflow with per-target status reports what it attempted. Confirming that a fact is actually gone needs an independent check, and the fixture machinery from chapter 6 already provides it: write a memory containing a fresh canary token, let it flow through embedding and one compaction cycle, delete it, then search every store for the token — primary table, vector index, full-text index, cache, and a live agent turn engineered to want exactly that fact. The compaction cycle in the middle is the part that must not be skipped, since a probe that deletes immediately after writing tests the fan-out from P and never creates the S node at all. Running this as a scheduled probe rather than only in the pipeline is what catches the case where a new derived store was added by another team and nobody updated the purge workflow, which is the realistic way this regresses.

Sampling the tombstone log gives the same assurance over real deletions without touching their content. Pick fifty tombstones from the last week, query each store by the memory id, and assert absence; the content hash on the tombstone additionally lets you check that no surviving record has the same hash, which catches a duplicate written by a parallel path. A discrepancy rate above zero is an incident rather than a metric, since every entry represents a request someone made and was told had been honoured.

Volume changes the shape of the problem at the account level, and the reason is worth seeing rather than accepting. An individual record is one workflow; a user exercising deletion across their whole history at the scheduling assistant meant 3,400 memory records with 11,000 descendants, and an organisation offboarding meant 640,000. The per-record workflow issues one delete call to the vector index per record, so the number of calls is the number of records, and the index's delete API is rate-limited per call rather than per identifier. Running that many times took the API well past its limit and the run stalled at 38 percent. The version that works batches by store rather than by record — one bulk delete per index with a list of ids, chunked to whatever the client accepts — so the number of calls becomes the number of chunks rather than the number of records, and it writes tombstones in a single statement. That brought the 640,000-record case from an estimated eleven hours to nineteen minutes. The suppression filter on the retrieval path is what makes the intervening time safe, since a bulk purge that takes nineteen minutes is nineteen minutes during which the primary rows are gone and the index is still catching up.

Putting the topic to work comes down to an order. The controls in it are cheap to add while the schema is young and expensive afterwards, and they are also independently useful, so the order to build them in is the order in which the next incident becomes impossible. Assign every record an owner and put the tenant in the key. Take identity from the session and delete the parameters that let anything else supply it. Write the path inventory down, including the jobs, and push the constraint into the database where those paths cannot avoid it. Seed two canary tenants and let a hostile suite run in the pipeline. Record provenance and lineage from the first write, because neither can be reconstructed. And write the deletion paragraph you would be willing to send to a user, then build until it is true.

Two of those six are irreversible and the rest are not, which is the ordering advice underneath the list. Row-level security can be switched on next quarter over a table that has been running without it, a hostile suite can be written against a system that already exists, and an index can be relaid out under a working product. Provenance and lineage cannot be added to records that were written without them, and a deletion promise made before the workflow exists cannot be un-made once it has been sent. Build the two that only go one way first.