Isolation is proved by labelled fixtures and hostile inputs

The vendor from chapter 3 had eleven tests named after tenant isolation, all green, on the day the plumbing distributor's summary came back containing an electrical wholesaler's warehouse hours. Reading them afterwards, every one had the same structure: create org A, create org B, write a memory to each through MemoryRepository.write, search as A through MemoryRepository.search, assert that B's record is absent. They exercised one path, the one written by the people who wrote the tests, and they asserted on the return value of a function that had never been the problem. The compaction job did not appear in any of them, and neither did the vector index, the cache, or the model's final answer.

Those eleven tests were not badly written, and it is worth being precise about what went wrong, because the same shape recurs everywhere this kind of property is tested. A unit test establishes that a component honours its contract. MemoryRepository.search has a contract — given a namespace, return records in that namespace — and it honoured it, on the day of the leak as on every other day. Tenancy is not a property of that contract. It is a property of the conjunction of every path that touches the table, and a conjunction cannot be established by testing one of its terms. Adding a twelfth test of the same shape would have raised the count and left the property exactly as unproven as before.

A test suite that proves isolation has to be organised around two things the naive version leaves out: the inventory of paths from chapter 3, and an assertion surface that reaches the text the user finally sees.

Fixtures that are recognisable on sight#

The seed data does the work that assertions cannot. Give each tenant records containing a canary token, meaning a distinctive string chosen so that it appears nowhere else in the system and its presence anywhere is therefore proof that a particular tenant's data reached that place. A leak then becomes detectable by substring search in any artefact — a JSON response, a log line, a trace payload, a generated paragraph — without knowing in advance which field it would arrive in.

That last clause is the reason for the design rather than a convenience. A leak's arrival channel is not known ahead of time: it can come back as a field in a JSON body, as free text the model wrote, as a span attribute, as a cache entry, or as a substring of a summary that fused two tenants' facts. An assertion written against a named field can only fail on the channel its author anticipated. An assertion written against the serialised artefact fails on every channel, and it is sound because uniqueness makes presence imply provenance: if the token appears nowhere else in the system, then a token found in an artefact can only have come from the tenant whose record contains it. The whole method rests on that implication, which is why the uniqueness of the strings is a correctness requirement rather than a stylistic one.

CANARY = {
    "org_alpha": "ZANZIBAR7F31",
    "org_beta":  "QUETZAL9C08",
}

@pytest.fixture
def two_tenants(store):
    seed = {}
    for org, token in CANARY.items():
        seed[org] = [
            store.write(org_ns(org), f"Contracted support tier is Premium ({token})."),
            store.write(user_ns(org, f"{org}_u1"), f"Prefers replies in German ({token})."),
            store.write(thread_ns(org, f"{org}_t1"), f"Restarting the router failed ({token})."),
        ]
    return seed

One record per scope from chapter 1 is deliberate, because the scopes have different enforcement paths and a suite that only seeds user-scoped records will not notice that organisation-scoped reads skip the namespace check. The three constructors are the ones from that chapter, so each seeded record's key already carries the tenant in its first position, and a prefix listing for one tenant cannot name the other's rows by omission. The tokens are pronounceable and distinctive so that a human reading a failure message can tell instantly which tenant's data escaped, which matters when the failing assertion is three layers deep in a generated response.

The assertion that catches the most is the coarsest one, applied at the outermost boundary:

def assert_no_foreign_canary(payload: str, acting_org: str) -> None:
    for org, token in CANARY.items():
        if org != acting_org:
            assert token not in payload, f"{org} data leaked into a response for {acting_org}"

Run that over the serialised HTTP response, over the model's final message, over every span the request emitted — the timed records your tracing system keeps of each step, including the inputs and outputs of a retrieval — and over the contents of the cache after the request completes. The trace check is the one that finds the post-filtering problem from chapter 5, because that is a leak into observability rather than into the answer, and no assertion on the response body would ever see it.

Assert on three surfaces, not one#

Isolation can fail at three depths, and each needs its own assertion because a control at one depth can mask a failure at another.

At the store, ask directly: can a connection or client authenticated as tenant A retrieve, update, or delete a record belonging to tenant B, by id, by search, or by prefix listing? These tests bypass the repository layer on purpose and talk to the database and the vector index with tenant A's credentials, because that is what verifies the row-level policy rather than the application code sitting on top of it.

At retrieval, ask whether the candidate list — before re-ranking, before deduplication, before truncation — contains any foreign record. Asserting on the final result set hides post-filtering entirely, since post-filtering produces a clean final list from a contaminated candidate list.

At the response, run a real turn through the agent with a prompt engineered to want the other tenant's data ("summarise everything you know about support tiers across all customers") and assert on the generated text. This is the only surface that catches a leak arriving through the model's own summarisation of context that was assembled by some path other than the memory tool — a system prompt built from a cached org profile, for instance.

The reason all three are needed rather than the outermost one alone is that the layers mask each other in a specific direction. A downstream filter can clean a contaminated upstream result, so a pass at the response says nothing about the store or the candidates. The masking does not run the other way: contamination at the store shows up in the candidates unless something filters it, and contamination in the candidates shows up in the response unless something filters it. So each assertion covers the failures that its own layer and everything above it can produce, and only the innermost assertion sees the innermost failure. Three assertions is the smallest number that leaves no depth unwatched.

The hostile half#

Each case below tests a control that an earlier chapter specified, which is what makes the suite reviewable: a failure names the mechanism that regressed rather than just reporting that something got out.

The middle column is what makes this a suite rather than a list of paranoid ideas. Every entry in it names a mechanism specified earlier in this topic, which means a red case reports which design decision stopped holding rather than that something leaked, and it means a control added later arrives with its own row rather than with a vague instruction to test more. Read the third column downwards and a second pattern shows up: every expected result is a statement about what happens when the input is hostile, and in six of the nine rows the correct behaviour is that nothing at all comes back. A suite whose expected results are mostly absences needs the canary tokens to be meaningful, since absence of a foreign token is the only observable that distinguishes a control working from a request that failed for some unrelated reason.

Two rows deserve their own note. The fourth deletes a predicate on purpose and expects zero rows, which is the assertion that distinguishes a database policy from a convention — if the answer is "all rows", the policy was never enabled or never forced. The concurrency case is the one most often skipped and the one that catches the connection-pool bug, which no sequential test can reach. Fire the requests with real interleaving against a pool sized deliberately small — four connections for fifty requests forces reuse — and assert per response rather than in aggregate, since a single crossed response among fifty is exactly the production signature. Aggregating would hide it: fifty responses of which forty-nine are correct averages to a pass on any measure that averages.

@pytest.mark.parametrize("path", ALL_MEMORY_PATHS)  # from the chapter 3 inventory
async def test_no_cross_tenant_read(two_tenants, path):
    async with session_as("org_alpha", "org_alpha_u1"):
        result = await path.execute(query="support tier")
        assert_no_foreign_canary(serialise(result), acting_org="org_alpha")

Parametrising over the inventory rather than writing one test per path is what keeps the suite honest as the system grows, because adding a new job means adding it to ALL_MEMORY_PATHS, and a path that is not in the list is visibly not in the list. That visibility is the whole benefit. With one test function per path, a job that nobody wrote a test for is indistinguishable from a job that does not exist; with a parametrised list, the list is a single object a reviewer can read against the inventory, and the diff that adds a background worker without adding a line to it is a diff somebody can object to.

The same fixture, running in production#

A suite in the pipeline tests the code. It does not test the deployed configuration, and several of the controls in this topic are configuration: whether row-level security is actually enabled on the production table, whether the production vector collection has a payload index on the tenant field, whether the cache in front of the store is the one the tests exercised. All three can be right in the test container and wrong in the cluster.

The cheap way to close that gap is a pair of permanent synthetic tenants in production, seeded with their own canary tokens and exercised by a scheduled job that runs a subset of the suite — one read per path, the forged-identifier case, and the smallest-tenant retrieval probe from chapter 5. Keep the synthetic tenants small and asymmetric on purpose: the reason is the arithmetic from that chapter. Post-filtering shows itself only when a tenant's share of the index is small, so a balanced pair — two tenants of equal size in a store containing only them — has each holding half the index, which is the one configuration in which post-filtering and pre-filtering return the same thing. One tenant holding thirty records against a production index of millions reproduces the failure; a balanced pair never will, however carefully the rest of the test is written. Route their canary tokens into an alert that fires on any appearance outside their own responses, including in log aggregation, and you have a detector that covers the paths a code test cannot reach.

Two operational cautions come with this. The synthetic tenants must be excluded from billing, analytics and model training pipelines, or they will show up in a quarterly report as two customers with strange usage patterns. And the canary tokens must be genuinely unique strings rather than words like "CANARY", because a token that collides with real content produces alerts nobody trusts, and an alert nobody trusts is worse than no alert. That second caution is the uniqueness requirement from earlier arriving as an operational rule: a colliding token breaks the implication the whole method rests on, so a match no longer proves provenance and every alert has to be investigated by hand.

Proving the tests can fail#

A green isolation suite is evidence only if the tests would have gone red under the conditions they claim to cover, and that is a separate thing to check. The reasoning is the same as for any other measurement. A test that is green when the system is sound and also green when the system is broken has the same reading in both worlds, so seeing it green tells you nothing about which world you are in. Evidence requires that the two cases look different, and the only way to find out whether they do is to build the broken world and look.

Take the original defect and reintroduce it deliberately behind a flag — restore the batch-by-row-count compaction query, restore the user_id parameter on the memory tool, switch the vector search back to post-filtering — and confirm each one turns a specific test red. Every mutation that leaves the suite green identifies a path with no assertion on it, and that exercise typically finds two or three on the first run. This is worth automating as a small mutation set that runs weekly against a branch, since it is the only mechanism that detects a test which stopped meaning anything after a refactor.

The vendor's eleven green tests are the same failure caught late rather than early. Nothing about them was wrong on the day they were written; what changed is that the system grew paths they did not cover, and their greenness stayed constant the whole time. That is the property to be suspicious of. A count of passing isolation tests is a quantity that does not move when the number of unprotected paths moves, so watching it tells you about the tests and nothing about the system, and a suite parametrised over an inventory is the arrangement that couples the two back together.

Coverage of paths, not of lines#

A passing suite establishes that the controls you specified are present on the paths you enumerated, under the inputs you thought of. It does not establish that memory cannot leak, and the gap between those two statements is the entire value of keeping the path inventory as a first-class artefact — coverage here is coverage of paths, not of lines, and a line-coverage number on the repository layer says nothing about a job that never imports it.

Three categories sit outside what this suite can reach. Anything that leaves the system boundary is gone: the trace assertion checks what you emitted, and what your observability vendor does with it afterwards is governed by a contract rather than a test. Timing and volume side channels are not addressed, and in a shared index a query that is measurably slower when another tenant holds a similar record is a real if minor disclosure. And a suite driven by a model — the response-level assertions — is nondeterministic, so a case that passes because the model declined to comply once has not proved that it will decline again; those cases need repetition and should be read as sampling rather than as proof.

Isolation testing answers whether other people can see a user's memories. It says nothing about whether the user can, and a system that guards its memory perfectly while offering no way to look at it has solved half of the privacy problem and left the half the user actually experiences.