Layering a write path that never stops to ask about 'call me Dave'

The first version of the hardened write path sent every candidate to a review queue. It was correct, it caught everything, and it lasted nine days. At 4,000 candidates a day and a measured two minutes of reviewer time each, the queue needed 133 person-hours a day to stay level, which is seventeen full-time reviewers for a personalisation feature. That arithmetic is worth doing in the open, because it is the whole reason the rest of this chapter exists: four thousand candidates at two minutes each is eight thousand minutes of work arriving every day, and a working day is eight hours, so the staffing requirement follows without any judgement about how good the reviewers are.

The queue was never level. Median dwell time reached eleven hours, which meant a user who said "call me Dave" was greeted as David for the rest of that day and often the next, and the support team started filing bugs against memory for not working. A control that makes the feature worse than not having it is not a control anyone keeps.

The distribution is what makes a better design possible. Of those 4,000 daily candidates, 3,680 were names, languages, timezones, units, and communication preferences — content whose worst case is mild annoyance. The dangerous fraction was small and, more usefully, identifiable by structure rather than by meaning, which is what lets a pipeline route rather than inspect.

Five layers, each with one job#

The percentages on the two edges out of the fourth decision are the design in one place. Almost everything takes the left edge and is written without any model or human being consulted, and the right edge is what the expensive layers see. Nothing was thrown away to get there — the candidates on the left edge were all checked, by layers two and three, using tests that do not read for meaning.

Layer one attaches the write log record from chapter 5 and filters nothing. Its position at the front is not ceremony: origin is an input the layers below it depend on, and a candidate that reaches the schema check without knowing whether it came from a user turn or a fetched page has already lost the cheapest discriminator in the system. Making provenance mandatory to construct a candidate — a constructor argument rather than a field set later — is what stops a new code path from quietly appearing that writes without it.

Layer two is chapter 4's schema, and it is the layer that does the most work per microsecond. It runs deterministically in about 200 µs, rejects on structure rather than on meaning, and cannot be argued with by any text, which matters because the input is text written specifically to argue with software.

Layer three is the one chapter 4 left open, and it asks a question the schema cannot: whose profile is being modified, and by whom. Every candidate carries a subject_user_id and an actor_user_id, and in the ordinary consumer case they are equal and the check is trivial. They diverge in exactly the configurations where poisoning gets interesting.

def authorise(c: Candidate, ctx: SessionContext) -> Decision:
    if c.subject_user_id != ctx.authenticated_user_id:
        # A support agent, another tenant's agent, or an extractor that
        # inferred a subject from text it read. None may write here.
        return Decision(admit=False, reason="cross-subject write", alert=True)
    if c.slot in ENTITLEMENT_BEARING:
        return Decision(admit=False, reason="entitlements are read from the IdP")
    if c.origin == "tool_result" and c.slot in HIGH_RISK:
        return Decision(admit=False, reason="untrusted origin into high-risk slot")
    return Decision(admit=True)

ctx.authenticated_user_id is the value the whole function turns on, and it comes from the session rather than from the candidate, which is what makes the first comparison meaningful — comparing two fields the extractor filled in would prove nothing. The first branch is the one that stops a poisoned page from writing to somebody else's profile, which is the version of this attack that scales: a payload on a page fetched by a support agent's copilot describes a customer, and an extractor with no subject discipline will happily create a record on that customer's account.

The second branch is a structural refusal, not a filter, and it is the reason no wording of an entitlement claim gets through — there is no slot that accepts it and no code path that writes one. What the user is entitled to is resolved when it is needed from the identity provider, the IdP named in the code above, and from the billing service, which are the systems where that answer is actually true. The third branch is chapter 4's origin rule, restated here because it is an authorisation question as much as a schema one: free text from a source outside the trust boundary is text nobody was authorised to write.

Layer four is where the classifier lives, and it sees 3.1% of traffic because slot and origin have already decided that the rest is either safe or refused. Human review sits behind the classifier on that same 3.1%, which at this volume is 124 candidates a day and roughly four hours of one person's time — one hundred and twenty-four candidates at two minutes each is about two hundred and fifty minutes, against the eight thousand the unrouted queue was receiving. Median dwell fell to 4.2 hours, and nobody noticed, because nothing a user says conversationally about their own name or language ever enters that queue.

Layer five accepts that the four above will be beaten and makes the aftermath cheap. Writes are versioned rather than overwritten, deletes are tombstones, meaning the row is marked dead and kept rather than removed, so that a revoked record cannot be silently re-derived by a summary job, revocation works by source_sha256 as chapter 5 showed, and a sweep re-scores stored rows against the current policy version whenever that version changes. The sweep is the layer people forget: a policy tightened today does nothing about the 360,000 rows admitted under the old one, and re-scoring them costs one batch job rather than an incident.

Ordering by what survives being defeated#

The drop order is not the cost column sorted downwards, and the gap between those two columns is the thing to read. Cost per write runs from 0.3 ms to two minutes, which would put the classifier and the review queue at the bottom on price alone. What actually decides the order is the fourth column. Three layers are defeated by conditions that are either detectable or structural — a code path you can find, a payload that is merely false, a compromised session — and three are defeated by conditions that leave no trace at the moment they occur. The two orderings happen to agree here, and it is worth knowing they agree for a reason rather than by coincidence, because the reason is that a layer which reads for meaning is both slow and quiet when beaten.

The classifier goes first under time pressure, and the reason is its failure mode rather than its accuracy. When a deterministic schema check is defeated the write is refused and the attacker sees a rejection; when a classifier is defeated it returns instruction_shaped: false, score: 0.19, the write proceeds, and the write log records that the candidate was checked and found clean. You have paid 90 ms per high-risk candidate to obtain a false assurance that is now durable in your audit trail.

That last consequence is the one worth being precise about, because it makes the classifier's contribution potentially negative rather than merely small. A missing check leaves a row whose provenance says nothing about whether anyone looked at it. A defeated check leaves a row whose provenance says somebody looked and found it clean, and the score is a low number that reads as evidence. An investigator six weeks later treats those two rows differently and should not, since the second one is the output of a component that an attacker was tuning against. It is also a model reading text that was written to influence models, which is the same structural weakness that chapter 4 identified in asking the extractor to police its own input, one layer further along. Against that, its unique coverage is narrow: on the forty payload variants, the classifier caught two that the origin rule and slot schema had not already refused, and both of those were freeform notes from user turns, which is the least valuable case because a user poisoning their own profile is mostly harming themselves.

Human review goes second, and the honest reason is that reviewers cannot judge what the queue is asking them to judge. A reviewer looking at Works at Acme, on the billing team has no way to determine whether it is true, and no reviewer in a support organisation will refuse a plausible sentence at 4pm. The test that separates useful review from theatre is whether the reviewer has an independent source to check against: an entitlement claim can be checked against the billing system, a source URL can be opened, and a plausible sentence about somebody's employer can be checked against nothing at all. Review is effective where the reviewer has an independent basis for a decision and it is theatre where they do not. If you keep review at all, keep it only for candidates where the write log gives the reviewer something to check.

Provenance, schema and authorisation are the three that stay, and the interesting thing about them is that they cost about 1.5 ms combined and none of them involve a model. They also account for all six of the successful payloads from chapter 3: the origin rule refuses free text from tool results, the slot schema has nowhere to put an entitlement claim, and the subject check stops writes aimed at other people's profiles.

The layer that gets bypassed is the one you did not enforce in the database#

A stack like this has one failure mode that has nothing to do with attackers. Six months later a feature team needs to store something the slot schema does not model, finds the gate inconvenient, and writes an INSERT from their own service. The pipeline is still there, still correct, still tested, and no longer the only way in. Every write path built in application code has this property, and the fix is to make the database refuse.

Revoking INSERT and UPDATE on agent_memory from the application role and exposing a single SECURITY DEFINER function — one that runs with its owner's privileges rather than the caller's, so it can write to a table the caller has no rights on — that takes the candidate together with its lineage fields turns the policy into something Postgres enforces. Follow the permissions rather than the code and the guarantee is easy to check: the application role cannot write to the table at all, the function's owner can, and the only way to reach the owner's privileges is to call the function. A write with no write_id, no origin or no policy_version fails at the boundary rather than passing quietly, and a new service that wants to store memory has to call the function, which means it inherits every layer. A team that adds one next quarter does not have to know the policy exists, because the alternative to calling the function is an error.

This is also what makes the provenance guarantee real rather than conventional: memory_write_log gets its row inside the same function and the same transaction, so a record and its lineage cannot exist separately by construction. That is the invariant chapter 5 argued for, now enforced by the database instead of by everyone remembering.

The stack concedes, and says so#

None of this is a firewall, and describing it as one in a design review invites a false sense of completion. Every layer has a stated defeat condition in the table above, and the two that stay are beaten by the same thing: a well-formed, correctly slotted, subject-consistent record that is simply false. Works at Acme, on the billing team written from a poisoned page passes all three deterministic layers, because nothing at write time can establish truth. What the stack achieves is a reduction in the class of things that get through — from arbitrary text including imperatives, secrets and authorisation claims, down to plausible declarative facts in bounded slots — plus a lineage record for whatever does get through and a mechanism to revoke it in bulk.

Notice what kind of achievement that is, because it is easy to overstate in either direction. The stack does not lower the probability that a determined attacker succeeds so much as it changes what success looks like: not a stored instruction, not a stored credential, not a stored entitlement, but a stored fact that happens to be untrue. That is a narrowing of the output class rather than a reduction in the attempt rate, and it is worth exactly as much as the difference between what a false fact can do and what a stored instruction can do.

That residual is not nothing. A false fact in a employment slot still shapes future sessions, and if the agent's tools can act, the model may act on it. Which means the record that survives the write path has to be survivable when it comes back in, and the read boundary is the only place left to make that true.