Retrieval hit rate is activity; task success is the measurement

Memory, in the sense this topic uses, is a store of facts about a user that outlives the conversation they were mentioned in. A model sees only its context window, the block of text handed to it with each request, and that block is discarded when a session ends, so an assistant without a store meets the same analyst as a stranger every Monday and has to be told again that the fiscal year starts in April. The machinery that avoids this has two halves: a write policy that decides what is worth keeping out of a conversation, usually an extractor model proposing candidate facts and a gate accepting or rejecting each one, and a read policy that decides which stored records come back into the prompt, usually by computing an embedding of the incoming request — a list of numbers positioning a piece of text in a space where texts that mean similar things sit near each other — and searching for the stored records nearest to it. Readers meet the arrangement wherever a product claims to remember them, in the saved-memory feature of a consumer chat assistant, in a support agent that knows which plan a customer is on, or in a coding agent that recalls a repository's conventions.

Evaluating memory, the subject of this topic, means establishing whether those two policies earn what they cost. Retrieval adds latency to every request, injected records add input tokens, and the accumulated facts are personal data that has to be retained, protected and deleted on request. The benefit on the other side is much harder to see, because a fact recalled correctly looks from the outside like an assistant doing its ordinary job. Nobody files a ticket saying the currency was right again. That asymmetry is why a memory feature is usually measured by the numbers that are easy to collect rather than by the one that answers the question, and why the numbers that are easy to collect are the ones that cannot fail.

The dashboard that cannot report a failure#

The memory dashboard in the quarterly review looks healthy. Retrieval fired on 61% of assistant turns, the store holds 4.1 million facts across roughly 40,000 active workspaces, the median retrieving turn injects 3.2 memories into the prompt, and the write gate accepts 88% of the candidate facts the extractor proposes. The assistant in question lives inside an analytics product: users ask it to build reports, and it remembers things like which currency a workspace reports in, that this customer's fiscal year starts in April, and that a particular analyst always wants revenue net of tax. The head of product asks whether the feature is working. Nothing on the board answers that, because every number on it would look the same or better if memory were making the assistant worse.

That last claim is the one to establish rather than assert, and the quickest way to establish it is to break the system on purpose and follow the arithmetic. Retrieval decides that a stored record matches by comparing its similarity score against a threshold — the score a record has to beat before retrieval will treat it as a match — and a turn counts as a hit when at least one stored record clears that bar. The scores themselves are produced by the embedding model and do not move when the threshold moves. So lowering the bar cannot take away a record that already cleared the old one, and it admits every record whose score sits between the new bar and the old. The set of hitting turns after the change therefore contains the set of hitting turns before it, and hit rate can only go up.

Lower the threshold on this system from 0.78 to 0.62 and the hit rate goes from 61% to 89%. The extra somethings are, by construction, the weaker matches — the ones the old threshold rejected — and nothing in the arithmetic above consulted whether any of them was relevant to the request. The write gate behaves the same way for the same reason. A gate is a filter over the extractor's proposals, so removing a condition can only admit proposals the old conditions rejected. Loosen it from "store only user-asserted stable facts" to "store anything the extractor labels a preference" and the fact count roughly triples inside a month, most of the increase being restatements, one-off instructions and things the user said sarcastically. Both changes move both headline metrics up. Neither is evidence that a single report came out closer to what the analyst wanted.

The general form is worth stating, because it is what makes a metric safe or unsafe to put on a board. Hit rate is a monotone function of the threshold — loosen the bar and it rises, tighten the bar and it falls — and the direction is fixed by the definition rather than by the quality of the retrieval. That licenses a genuinely useful prediction: given a configuration diff, you can say which way the number will move without running anything. The residual is that monotonicity fixes the direction and not the size, which depends on how the scores happen to be distributed, and that it holds only for changes to the bar. Swap the embedding model and the scores themselves move, so the hit rate can go either way and the argument above says nothing.

An activity metric is one whose value is determined by the configuration of the feature rather than by the outcome of the work. The distinction matters because activity metrics are the ones that are easy to instrument, so they arrive first, get on the dashboard first, and become what the team means by "how memory is doing".

Only the last row can move in the direction that tells you to stop. That is the property to look for when choosing what to put on a dashboard: a metric that cannot go down when the feature is bad is not measuring the feature, it is measuring whether the feature is switched on.

Why a good retrieval score is not a defence#

Retrieval quality metrics sit awkwardly between activity and outcome, and the obvious objection here is that hit rate is a crude number while the retrieval literature already has better ones. It does. Recall@k is the share of the records that should have been found which appear in the top k results; mean reciprocal rank rewards putting the right record first; and nDCG, normalised discounted cumulative gain, grades the whole ordering against an ideal one. All three are computed over a labelled set of query-to-memory pairs. They are real measurements, and a bad score on them is genuinely bad news.

The belief worth taking apart is that a good score on them defends against a bad memory system. Write recall@k out as the fraction it is: the number of relevant records that appear in the top k results, divided by the number of relevant records. Both halves of that fraction are counted over the labelled set, and the labelled set is built by looking at what is in the store and marking which records ought to come back for which query. A fact the extractor never wrote is in neither half. It cannot be in the numerator, since it is not in the store to be returned, and it is not in the denominator, since nobody labelled a record that does not exist. Remove it from the top and the bottom of the same fraction and the fraction is unchanged.

So the score is constant while the property that endangers the user varies. Delete four facts in ten from the store and the retriever still gets full marks for finding everything that exists, because "everything that exists" is what the denominator counts. If the extractor never wrote down that the fiscal year starts in April, recall@k is computed over a labelled set that does not contain the fact. The mem0 write-up on 2026 memory benchmarks makes this point sharply: retrieval scores hide a bad write policy, because they are conditioned on what was written.

What the score does license is narrow and worth keeping. It bounds the retriever's competence over the records that are actually there, which is exactly what you want once a task has already failed and the question is which stage failed. Recall@k earns its place later in this topic as that diagnostic, which is a different job from deciding whether the feature is worth having.

The comparison that answers the question#

The measurement memory owes is a comparison between two runs of the same work: a held-out set of tasks — tasks reserved for measurement and kept out of every tuning decision — executed with memory enabled, and the identical tasks executed with it disabled, scored by the same success criterion. Everything else — model, temperature, system prompt, tool definitions, retry policy — is held fixed, so the only difference between the arms is whether retrieved memories were placed in the context.

A task here is larger than a single prompt. It is a scenario with a history: one or more earlier sessions that seeded the store, then a final session containing the request that gets scored. The reason for that structure follows in three steps. The model sees only what is in its context window. A final request that already contains everything needed to answer it hands both arms the same context, since the memory-off arm is missing only the injected block and the block was not carrying anything the request lacked. Two arms with the same context produce the same distribution of answers, so the task cannot come out differently in the two arms except by chance. A task without a history is therefore a task that contributes nothing to the comparison, however much it looks like a memory test. The multi-turn scoring machinery from the earlier topic on conversational evaluation carries over directly here, including the trajectory-level success criterion and the rule that a conversation is scored on its outcome rather than turn by turn; there is no need for a second grading harness.

@dataclass
class MemoryTask:
    task_id: str
    seed_sessions: list[list[Turn]]   # replayed to populate the store
    final_session: list[Turn]         # the request that gets scored
    success: Callable[[Trace], bool]  # from the multi-turn eval harness
    slice: str                        # "explicit_pref" | "inferred_fact" | "control"

def run_arm(task: MemoryTask, *, memory: bool) -> Trace:
    store = MemoryStore(namespace=f"{task.task_id}:{memory}")   # isolated per arm
    agent = Agent(model=MODEL, temperature=0.2, prompt=SYSTEM_PROMPT,
                  retriever=store.retriever() if memory else NullRetriever(),
                  writer=store.writer())                        # writes always run
    for session in task.seed_sessions:
        agent.run_session(session)        # writes happen here in both arms
        store.flush()                     # persist before the session ends
    return agent.run_session(task.final_session)

def paired_run(tasks: list[MemoryTask]) -> list[tuple[str, bool, bool]]:
    return [(t.task_id, t.success(run_arm(t, memory=True)),
                        t.success(run_arm(t, memory=False))) for t in tasks]

Two details in that harness are design decisions rather than requirements, and it is worth separating them. The requirement is only that the two arms differ by exactly one thing. The decision is which one thing, and here it is the read path: the write path runs in both arms, and only the read path is ablated — switched off so that the effect of its absence can be measured — so the arms differ by whether stored context enters the prompt rather than by a whole subsystem. Ablating writes as well is a different and also useful experiment, because it tells you the cost of the extractor's inference calls, and it is worth running separately so the two effects do not get mixed into one number.

The second detail is the per-arm namespace, and it is a requirement rather than a decision. Run both arms against the same store and the memory-on arm's writes contaminate the memory-off arm on the next task: they are still sitting there when it reaches that task, so the memory-off arm's store is not the store it would have had on its own. The difference the harness then reports belongs to neither arm.

Task success also has to mean something specific enough that two people scoring the same trace agree. For the analytics assistant it is a conjunction of three checks on the final artefact: the generated query returns the reference row set, the currency and tax treatment match the reference spec, and the date range matches the reference. Two of those three are deterministic comparisons against a stored expectation and cost nothing to run; the third, which covers cases where several query shapes are legitimately correct, is graded by a model against a written criterion. The mix matters for what comes later, because a deterministic comparison returns the same verdict every time it is run and a model-graded one does not, so the deterministic part of the criterion contributes no variance and the model-graded part does.

The prompt must also stay identical apart from the memory block itself. It is tempting to give the memory-on arm a slightly richer system prompt explaining how to use retrieved facts, since that is what production does, but then the experiment compares two prompts and attributes the whole difference to memory. If production's prompt has memory-specific instructions, keep them in both arms and let the memory-off arm receive an empty memory block, which is what it will see in production whenever retrieval returns nothing anyway.

Held-out means held out from the tuning#

The set has to be built before the write gate, the ranker and the injection prompt are tuned, and it has to stay out of that tuning afterwards. Threshold values, k, chunk sizes and the extractor's instructions all get adjusted by looking at cases, and every case you look at while adjusting them stops being evidence.

The reason is worth walking rather than repeating as a slogan. Tuning is a search: several candidate configurations are tried, each is scored on the cases in front of you, and the highest scorer is kept. Each case's score has two parts — how well the configuration handles cases of that kind, which is what you want to measure, and the accidents of that particular case, which is the wording the user happened to use and where the relevant record happened to sit in the ranking. The search cannot tell those apart, so it selects for both, and the configuration it returns is the one that fits the accidents best as well as the kinds best. Run it again on the same cases and the accidents are still there, so the score comes back high. Run it on cases it has never seen and the accidents do not recur, so the score drops. The gap between the two is what "held out" buys, and it exists whether or not anybody intended to overfit.

Teams that build the eval set after shipping usually build it by sampling conversations where memory did something interesting, which selects for cases the current configuration already handles. That is the same defect arriving from a different door: the cases were chosen by the configuration.

Each task also carries a slice label, because the aggregate number is going to be the first thing that disappoints and the slice breakdown is what survives. Explicit user-set preferences ("always net of tax") behave very differently from facts the extractor inferred from conversation ("this user seems to work on EMEA accounts"), and averaging them together produces a number that describes neither.

Three nearby questions the comparison is silent on#

The with-and-without comparison answers one question and does not answer several that sit next to it. It does not tell you which part of the system to fix when a task fails, since a failure could be a fact never written, a fact written but never retrieved, or a fact retrieved and then ignored by the model — three causes that a single pass/fail flag cannot separate, because one bit of output has one bit of information in it and three causes need more.

It does not tell you whether the memory system is well designed relative to alternatives. The comparison of twelve memory architectures in the "no architecture wins" write-up is a reminder that the ranking of designs shifts with the workload, so a result on this task set is a result about this product on this traffic. A design that loses here may be the right design for a product whose sessions are shorter and whose relationships are longer.

It also says nothing about engagement, retention or how users feel about being remembered, which are real considerations that require different instruments and a much longer time base. A feature can be neutral on task success and still change whether people come back, in either direction, and the eval above has no way to see that.

The limitation that bites first, though, is arithmetic rather than conceptual. The difference between 152 and 148 is four tasks out of two hundred. Re-running the memory-on arm the following morning, with the code, the data and the model all unchanged, produced 149 — a movement of three tasks with nothing happening at all.

A comparison whose result moves by three tasks when nothing happens cannot resolve a difference of four. Until the size of that movement is known and written down, no argument about whether two points justifies the extra latency and tokens has anything solid underneath it, and every later chapter of this topic is an argument of exactly that shape. Computing the noise band is the next chapter's work, and it determines how much of the rest of this topic is worth doing at all.