An eval inside one session tests context, not memory

A write-then-read eval is one where the fact under test is written in one session, the session ends, and a later session is scored on whether the fact came back. That structure sounds like a description of any memory test, and most memory suites do not have it. What they have instead is a conversation in which the fact is stated and then used, which is a test of the model's ability to read its own context window and says nothing about the store.

The reason this matters more than it sounds is that the two failure modes look identical from the outside. A suite that tests context returns a high number, rises when the model improves, and falls when the model regresses, so it behaves like a working instrument in every way except the one that counts. Nothing in the previous chapter's arithmetic can detect it either: a confidence interval computed over cases that do not depend on the store is a perfectly correct interval around zero.

The 94% that survives an empty store#

The memory team's own suite — 180 cases, all of them labelled memory tasks — reports a 94% pass rate. Truncate the memory table, point the retriever at the now-empty store, and run the same 180 cases again. The pass rate is 91%.

Read that pair carefully, because the shape of it is the whole diagnosis. The store went from millions of records to nothing, which is the largest change anyone could make to a memory system, and the number that is supposed to report on memory barely moved. A quantity that stays put while the property it claims to measure is destroyed is not measuring that property. Five cases' worth of pass rate changed out of 180, which means at most 175 of the suite's cases are graded on behaviour that a system with no memory at all produces identically — at most, because a three-point net move is compatible with a few cases flipping each way, and the per-case flip counts from the paired table are the honest version of the count.

The same defect turns up outside memory, and one comparison is close enough to be worth stating as an analogy rather than a claim. Evaluating this suite is like evaluating a cache by measuring response correctness on requests that carry their own answer in the payload: the cache is never consulted, the responses are right, and the dashboard is green. The analogy holds for the defect and breaks on the repair. A cache has an origin behind it, so a correct cache is one that returns what the origin would, and you can always check it against that. A memory store has no origin — nothing else in the system knows that the fiscal year starts in April — so there is nothing to compare a retrieval against, and the only available check is whether removing the store changes the outcome.

The null-store ablation is that check: leave the read path in place, empty the store underneath it, and re-run. Leaving the read path in place is the part that matters, because it holds everything else constant. Retrieval still runs, still embeds the query, still costs its latency, and still produces a memory block in the prompt — an empty one. The only thing that differs between the two runs is the content of the store, so the difference between the two pass rates is exactly what the stored content contributed. Disabling the retriever outright would also change the shape of the prompt, and then the difference would carry two causes.

It costs one afternoon and tells you what fraction of your memory suite is about memory, so it is worth running before any of the sizing arithmetic from the previous chapter is applied to a number that turns out to be measuring something else.

Why the first version of the eval always passes#

The natural way to write a memory test is to write a conversation. The user says "always report revenue net of tax", the assistant acknowledges, and three turns later the user asks for a revenue report; the assertion checks that the generated query excludes tax. It reads like a memory test and it is a context test.

Four steps take you from the design to the failure, and each is checkable on its own. The model's only input is the context window handed to it with the request. In a multi-turn conversation that window contains the transcript so far, because that is how a model is given the earlier turns at all. So an instruction stated in turn one is physically present in the request that carries turn four. The assertion inspects the output, and the output is a function of what was in the context, so an assertion that passes when the instruction is in the context will pass whether or not anything else put it there. The retriever can return an empty list, the store can be unreachable, the write path can be throwing exceptions into a swallowed log, and the case passes.

The fix is structural rather than a matter of writing better assertions, and it follows from the same four steps. If the transcript is the channel carrying the fact, the transcript has to go. The write has to be persisted and the session has to actually end, meaning the conversation id changes, the message history is dropped, and the only path by which the earlier fact can reach the model is retrieval from the store.

Three of the steps in that diagram are the ones that get skipped, and each of them fails in a different direction. The flush and index-commit wait matters because a write that lands asynchronously in the vector index — the ANN structure from the previous chapter, holding one embedding per stored record — may not be searchable for several seconds, and a harness that starts session 2 immediately will produce retrieval misses that have nothing to do with the read policy. That one makes a good system look bad, which at least gets investigated.

The other two make a bad system look good. Rotating the conversation id is obvious once stated and easy to get wrong when the harness reuses a single agent object across cases, since the message list often lives on that object; a harness that constructs one agent and loops over cases has not ended a session at all, whatever it named the variable. Discarding the rolling summary is the one that catches experienced teams. Many agents maintain a compacted summary of the conversation so far, precisely so that a long conversation does not overflow the context window, and that summary is by design a durable carrier of what was said. If it survives the session boundary it is a second, unmeasured memory channel carrying the fact straight into session 2's context. Either ablate it with the store or count it as part of memory and ablate both together, but do not leave it running while claiming to have turned memory off.

A case is only memory-dependent if the memory-off arm fails it#

The null-store ablation belongs inside case construction, not only in the final run. A case earns its place in the suite when the memory-off arm fails it, because that is the definition of depending on stored information.

def validate_case(case: MemoryCase) -> str:
    with_mem = case.success(run_arm(case, memory=True))
    without  = case.success(run_arm(case, memory=False))
    if with_mem and not without:
        return "memory_dependent"      # keep
    if with_mem and without:
        return "guessable"             # answerable from priors — rewrite or drop
    if not with_mem and not without:
        return "broken_or_hard"        # inspect: bad reference, or a real failure
    return "memory_harmful"            # keep, and read this one carefully

Roughly a third of candidate cases come back guessable, for a reason that is easy to miss: the model guesses correctly from priors, meaning from what is usually true in the world rather than from anything this user established. Ask what currency a workspace reports in, when the workspace name is a UK company and the seed data is in pounds, and the model will answer GBP without any stored fact at all. It is not retrieving; it is completing a pattern that holds across most of its training data. The case looks memory-dependent and grades as a pass in both arms, so the branch above catches it, and the branch is the only thing that will — an author reading the case will see a question about a stored preference and a correct answer, and both of those are true.

The last branch is worth keeping rather than discarding. A case that passes without memory and fails with it is the clearest single instance of the harm the topic's later chapters quantify, and a suite built by throwing those away reports a systematically flattering number, since it has removed exactly the evidence that could argue against the feature.

There is also a clock to fake, and the reason is arithmetic. Cases that test whether a fact is still true, or whether a record has aged out under a retention policy, assert something about the gap between when a record was written and when it is read. A harness that lets both of those come from the wall clock produces a gap of two seconds, which is the time the harness took to run, and no retention rule fires on two seconds. Worse, the gap depends on when the suite happens to run relative to when the seed data was authored, so it drifts. Storing an explicit occurred_at on each seeded write and letting the harness set the scored session's notion of now — six days later for a stale-preference case, eleven months later for a retention case — makes the gap a parameter of the case rather than an accident of the calendar. A harness that uses wall-clock time will pass a retention case in January and fail it in December.

Making a case genuinely dependent usually means adding a distractor — a plausible wrong answer planted where the model will find it — rather than making the question harder. If the stored fact is "fiscal year starts in April", the seed data should contain a plausible alternative — a January-aligned sibling workspace, or a report elsewhere in the account that uses calendar quarters — so that a model without the fact has something wrong to reach for. Without a distractor the model often abstains and asks, which some graders score as a failure and others as correct behaviour, and the case ends up measuring the grader's opinion about abstention rather than anything about the store.

A store with five records is not a store#

A case validated in isolation runs against a store containing exactly the facts the case seeded. Production stores contain thousands of records per workspace, most of them irrelevant to any given request, and retrieval has to find the right one among them.

That difference changes what the read policy is being asked to do. With five records the question is whether anything scores above the threshold, and almost any scoring function puts the one relevant record at the top of four unrelated ones. With thousands, the question is whether the relevant record outscores every irrelevant one, and each additional record is another chance for something topically adjacent to land above it. Adding records cannot lower the target's own score, so the failure is always the same shape: the target is still there, still above threshold, and no longer inside the top k.

Running the same 420 memory-dependent cases — which score 91% against a store holding only their own seeded records — against a store seeded with 4,000 unrelated records per workspace, sampled from other workspaces' real writes with names and identifiers replaced, dropped the pass rate to 68%. The read policy did not change. The haystack did.

Building that distractor corpus is cheaper than it sounds and has one rule worth stating. The records have to come from the same distribution as real writes, and the reason is geometric. Retrieval ranks by distance in embedding space, so a distractor only competes with the target if it lands near the query. A corpus of synthetic filler written by a model clusters somewhere the real query embeddings never go, so every filler record scores far below the target and the target wins by default, however many of them there are — an unnaturally easy corpus to rank against. Sampling real writes from other workspaces, then substituting names, identifiers and figures, preserves the phrasing and the topical overlap that put the distractors in the same neighbourhood as the query, which is the only place they can do any work.

That 23-point gap is the difference between an eval that tests whether retrieval can find a fact and one that tests whether it can find the right fact, and it is the single largest correction most memory suites need. It also has a maintenance consequence, because the distractor volume has to track production; a suite calibrated against 4,000 records is not measuring the same system a year later when the median workspace holds 30,000.

The competency categories in the LongMemEval benchmark are worth borrowing wholesale at this point, because they name the cases teams do not think to write: reasoning that requires combining facts from two different sessions, temporal reasoning about when a fact was true rather than what it is, knowledge updates where a later session contradicts an earlier one and the old record must lose, and abstention where nothing relevant is stored and the correct behaviour is to ask. The knowledge-update category is the one that most often exposes a write policy that only appends, since an append-only store answers "what is the fiscal year" with two records and no rule for preferring the newer. Its runnable harness scores a read-and-write policy against 500 curated cases, which is a reasonable calibration exercise even for a team whose real evaluation has to be built on its own traffic.

Four ways the fact reaches the model without being remembered#

The single-session version is the most visible leak and not the most common one in a mature harness. The others are variations on the same theme, which is that the information gets into the context by some route other than the read policy under test.

The third row is worth separating from the first two because it survives the null-store ablation. Empty the store and the agent can still read the fact out of a document or a profile field, so the case passes in both runs and looks memory-independent — which it is, but for a reason that a team reads as "the model is good" rather than as "there is a second channel". The check for it is not the ablation but an audit of what the seed data contains.

The last row is the softer version referred to in this chapter's description, and it is the most common way a memory evaluation quietly fools its authors. A team builds the eval by reviewing conversations where memory did something, tunes the extractor's instructions and the similarity threshold against those conversations until the writes look right, then scores the system on the same material. Nothing about the resulting number is a lie; it is a training-set score reported as a test-set score, the distinction being that data used to shape a system always flatters it and only data the system has never been tuned against measures it. The tell is a suite that scores far higher than production complaint rates suggest, and the repair is the ordinary one from supervised learning: partition by conversation before any tuning happens, and never let a conversation that informed the write policy appear in the scored set.

A suite that survives all four leaks now reports on memory rather than on context, and the number it reports is lower and means something. None of this changes the shape of the result. A case now either passes or fails, and when it fails, the trace says the assistant produced a report in the wrong currency and nothing else. Whether the currency preference was never written, was written and never retrieved, or was retrieved and sat in the context while the model ignored it are three different defects with three different owners, and one bit of output cannot tell them apart.