Memory keeps its place only by beating a re-retrieval baseline

Every argument in this topic so far has been made from properties — how durable a claim is, who owns the fact, what a wrong record costs to repair. Properties are how you decide what to build before there is anything to measure, and they run out at exactly the point where somebody asks whether the thing you built is worth keeping.

The measurement that answers that has one shape. Run the same tasks twice, once with the write policy live and once with no persistent memory at all, and compare how often the agent got the answer right. Everything else about the design is in service of making that comparison honest: where the tasks come from, how hard the baseline is allowed to try, and how the result is broken down when it arrives. This chapter reports one such evaluation at Arbor, and the part worth carrying away is the breakdown rather than the totals.

Building tasks that memory cannot rig#

The design is the same shape as any A/B comparison, with the arms chosen to answer the question that has been open since chapter 8. Arm A runs the stacked gate from chapter 3 at its 4.4% write rate, with retrieval from the store. Arm B runs no persistent memory at all: the same model, the same tools, plus the account lookups and transcript search from the previous chapter. Everything else is held constant, including the prompt, the tool definitions and the token budget, so the only difference is whether facts were extracted and stored ahead of time or found at read time.

The tempting way to build the task set is to look at what the store contains and write questions whose answers depend on those records. Follow that through and see what it measures: every task has, by construction, a record in the store that answers it, so Arm A's job is to retrieve a record that is known to exist and Arm B's job is to reconstruct the same fact from scratch. That measures whether retrieval can find what the gate wrote, which is a component test dressed as an evaluation, and it will report a large win for memory every time — including on a store full of records that should never have been written, since the tasks were derived from them.

Arbor built the set the other way round, from evidence produced before anyone looked at the store. Two support engineers read 220 sessions from accounts and a date range the gate had never been tuned on, and wrote a task wherever a session showed a user relying on something established earlier — a re-ask, a correction, an "as I mentioned last time", or a question that only makes sense with prior context. That yielded 140 tasks, each a question posed in a fresh session with no transcript carryover, with a rubric answer and a list of the prior sessions the fact came from. The sessions, not the store, are the ground truth, which is what makes the set neutral between the two arms.

The ratio between the two counts is a finding in itself. Two hundred and twenty sessions produced 140 tasks, so a little under two sessions in three contained a moment where the user leaned on something established earlier. That is the size of the opportunity the whole subsystem is competing for, and it is worth measuring before anything is built, because a workload where one session in twenty shows that pattern cannot repay a store however well the store is built.

@dataclass
class Task:
    id: str
    account_id: str
    question: str
    rubric: str                 # what a correct answer must contain
    depends_on: list[str]       # prior session ids establishing the fact
    fact_class: str             # "owned" | "user_established" | "preference" | "none"
    consequence: str            # "informational" | "acted_on" | "authorising"


def run(arm, tasks):
    rows = []
    for t in tasks:
        out = agent.session(account_id=t.account_id, memory=arm.memory,
                            tools=arm.tools).ask(t.question)
        rows.append({
            "task": t.id, "arm": arm.name,
            "pass": judge(t.rubric, out.text),      # validated against human labels
            "latency_ms": out.latency_ms,
            "prompt_tokens": out.prompt_tokens,
            "wrong_claim": judge_contradiction(t.rubric, out.text),
        })
    return rows

fact_class is what makes the result readable rather than a single blended number, and the three values partition the set. Of the 140 tasks, 31 depended on a fact the user had established that no system of record holds — the warehouse conventions, project constraints, working agreements. Forty-two depended on facts owned elsewhere, which is the class chapter 8 argued should never be memory. The remaining 67 needed no prior context at all and were included deliberately, to measure whether memory does harm on the turns where it is irrelevant. That last group is the one most evaluations omit, and omitting it means measuring only the turns where a subsystem might help.

The consequence field alongside it does a job that only becomes visible when the results arrive, and it has to be filled in when the task is written. It records whether a wrong answer to this question would merely misinform, cause the user to do something, or commit the company to something. Assigning it in advance is what makes the later stratification of errors a measurement rather than a story: a class chosen after seeing which tasks went wrong can always be drawn around them.

Grading used a model judge, which is a second model shown the rubric and the answer and asked whether the answer satisfies it, scored against 40 human-labelled outputs and agreeing on 89% of them, which is good enough to compare arms and not good enough to publish a two-point difference as a finding. The reason a judge of that quality can still settle a large gap is that its errors fall on both arms, so they add noise to each measurement rather than a bias to their difference — but the noise does not shrink just because the difference does, which is why the same judge is decisive at thirty points and useless at four.

Handicapping the baseline is the other discipline the design needs, and it is a design decision rather than a requirement — nothing forces you to make the arm you expect to lose as strong as you can. It is easy to build a re-retrieval arm that loses by giving it a weak transcript search, a small retrieval budget or no tools, and a comparison won that way tells you nothing except that you can lose on purpose. Arm B got the same token budget as Arm A spent on memory plus retrieval, a hybrid search over transcripts, combining keyword matching with embedding similarity rather than using embeddings alone, and tool descriptions written with as much care as the memory prompt. Where the two arms could not be matched — Arm B necessarily makes more tool calls — the difference was recorded as a cost rather than hidden, which is why latency and tokens are rows in the same table as success.

The result, with costs in the same table#

The four-point overall lead is not the result. It is a blend of three different effects that point in different directions, and reporting it alone would have been the same mistake as reporting the write rate.

It is also worth doing the arithmetic that connects the top row to the three below it, because the three strata partition the task set and an overall figure ought to be their average weighted by task count. Carry that out and the two arms come out within about a point of each other, with the baseline fractionally ahead rather than four points behind. The top row and the strata are therefore not two views of one measurement, and nothing in the table adjudicates which of them is off. What the disagreement does establish is that no weight of evidence sits in the overall row: it cannot be rebuilt from the parts, and the parts each have a task count and a mechanism behind them.

On the 31 user-established tasks, memory won by 32 points, which is the entire case for the subsystem. Arm B's transcript search found the relevant conversations often enough but had to re-derive the convention from a scattered argument each time, and it got it wrong in nearly half the cases, exactly as chapter 8 predicted.

On the 42 owned-fact tasks, memory lost by 19 points, because Arm A answered from stale records where Arm B called the billing and ticketing tools and got current data. This is the chapter 8 argument reproduced as a measurement, and it is the finding that changed the product: those records were not tuned, they were deleted, and the gate was changed to refuse claims that match an owned schema.

It is worth being clear about what that deletion does to the table, because it is the only change in the run that improves a stratum without a trade. Once the owned class is refused, Arm A stops answering those 42 questions from a stale copy and starts calling the same tools Arm B called, so its result on that stratum becomes Arm B's result on it. The nineteen points Arm A was losing there are not recovered by better records; they are recovered by having no records. That is the shape of a finding worth acting on, and it is invisible in a blended figure where a nineteen-point loss on one stratum is being netted against a thirty-two-point win on another.

On the 67 no-context tasks, memory cost four points. Irrelevant records were retrieved into the prompt and occasionally pulled an answer sideways, which is the same distraction effect that the seeded run in chapter 5 measured as inert-until-it-isn't. The mechanism is the retrieval-weighted precision result from earlier in this topic seen from the other end: a store contains records that match a wide range of queries, so on a query no record should bear on, the broadest records win by default and arrive in the prompt looking like context. Adding a conditional retrieval step — skip the store entirely unless the query resembles something a durable user fact would bear on — recovered that subset to 82% and left the 31-task subset unchanged, at the cost of one small classifier on the read path. The two halves of that result are what make it worth having: a change that had recovered the harm and also cost some of the benefit would have been a trade, and this one was not.

The wrong-claim row is the one to look at hardest, because it moves the opposite way from success. Arm A produced more confidently wrong statements even while passing more tasks, which is the signature of a system answering from belief rather than from a lookup. The two are not in tension: a system answering from a stored belief will assert where a system that has to look something up will either fetch it or decline, so the same mechanism that raises the pass rate raises the rate of confident error. Stratifying that by consequence made the decision sharper. A wrong-claim rate of 6.4% over 140 tasks is about nine wrong claims in the run, and three of them were on tasks marked authorising, meaning the assistant made or denied a commitment on the strength of a record. A third of the confident errors landed in the class where a confident error is worst, which is not a proportion any aggregate would have shown. A 6.4% wrong-claim rate that beats the baseline is still not an acceptable rate for that class, which is why the authorising class ended up on the explicit gate from chapter 3 regardless of what the aggregate said.

The cost side of the table deserves a sentence about proportions, since it is where intuition is most often wrong. Storage at $28 a month is noise, and inference is actually cheaper in Arm A than in Arm B because 1,500 fewer prompt tokens per request outweighs everything the store costs to read. The dominant cost of memory in this system is 0.35 FTE of human time, which at a loaded rate is somewhere around €4,500 a month, or roughly 160 times the storage bill. Any argument about whether to keep a memory subsystem that is conducted in terms of gigabytes is being conducted in the wrong currency, and the repair trail from chapter 6 is where the real money is.

Three limits on what the table proves#

Three limitations are worth stating, because they bound how much the table proves, and two of the three push in the same direction.

The tasks were built from sessions where users relied on prior context, so facts a user gave up on restating are invisible to the measurement, which biases against memory. The window was three weeks of held-out traffic — held out in the sense that the tasks come from accounts and dates nobody used while building or tuning the gate — and the value of a store grows with tenure while transcript search degrades with it, so a longer study would likely widen the 31-task gap. Both of those understate the case for the subsystem, which is worth knowing when the result is close.

The third limitation cuts the other way and constrains what may be quoted. The judge agrees with humans 89% of the time, which is fine for a 32-point difference and worthless for a four-point one. Between that and the arithmetic above, the overall row has two independent reasons to carry no signal: it should be read as "no meaningful difference", and it is only the strata that carry anything.

The comparison is also not a one-off, and the reason is that the baseline moves. Model changes and retrieval changes move it quietly. New integrations move it sharply: an integration that gives you an API for a fact class has just created a system of record where there was none, which is an event that should retire records rather than sit alongside them, because the class has crossed the boundary from chapter 8 in the middle of its life. Arbor reruns the two arms quarterly on a refreshed task set, and the standing rule is that any fact class failing to beat re-retrieval gets deleted at the next run.

Committing to that rule in advance is what makes the rerun a decision procedure rather than a report. A quarterly measurement with no pre-agreed consequence produces a slide, and the class that lost survives because removing it is somebody's work and keeping it is nobody's. The rule also has to name the unit it acts on, which is the fact class rather than the subsystem, because the outcome is almost never that memory wins or loses everywhere — it is that one class has quietly crossed the ownership boundary from the previous chapter and nobody noticed.

The evaluation that justifies a memory subsystem is therefore this one and not a proxy for it: task success with the write policy live, measured on held-out tasks, beating a re-retrieval baseline that has been given every advantage you can give it, reported by class of fact so that the classes which lose can be found and removed. Each clause is load-bearing. Held out, or the tasks were drawn from the store and the answer is decided in advance. Task success, or you are measuring the component rather than the product. Against a strong baseline, or you have measured how badly you can build the alternative. By class, or three effects of different signs arrive as one number and the classes that should be deleted survive inside it. Store size, write rate, record counts and user-visible "memories saved" tiles do not answer the question, because none of them is downstream of whether an answer was right.

What the write decision leaves you with is a policy you can state in a paragraph and defend with a table: which classes of fact are admitted, on what evidence, at what precision, with what expiry, and what happens when one turns out to be wrong. Everything downstream assumes that policy exists — how records are organised and indexed, how competing versions are reconciled, how a store is consolidated as it grows, and how retrieval decides what to surface into a prompt. Those are the next problems, and they are worth solving only for the facts that earned admission here.