Two points of task success against 400ms and 1,500 tokens

The review has forty minutes and eight people, two of whom spent a year building the memory system. The slide says 74% to 76%. The first comment is that 400 milliseconds is nothing compared to a two-point quality gain, the second is that the store is costing $3,700 a month and that seems cheap, and the third is that the legal team has been asking about the retention policy for a while now. All three are reasonable and all three are being made in the wrong order.

What makes an order right or wrong here is not taste. Each of the three questions takes the answer to another as an input, so they form a chain rather than a list, and asking them out of chain order means asking a question with a hole in it.

Follow the dependencies and the diagram stops being a flowchart and becomes an argument. The second gate compares the annual cost against the value of the gain, which is the gain multiplied by what a point is worth — so it is a ratio with the gain in it, and a ratio with an unknown in it has no value. Open the meeting on cost and the discussion becomes whether $178,000 a year is a lot, which depends entirely on what it buys, and eight people will answer that from their own budgets. The third gate asks whether a set of standing obligations is worth carrying in exchange for a benefit, which again names the benefit. Open on the retention policy and the argument becomes about risk appetite in the abstract, which is a real conversation and not this one. Significance goes first because it is the only one of the three whose question contains no term from the others.

The one arrow that leaves the chain and comes back is the loop from the first gate's "not a gain yet" box to the measurement at the top. It is there because failing gate one is not a verdict; it is a statement that the instrument was too small, and the response is to make it bigger and measure again rather than to move down the chain with an unknown. That loop is also where a decision goes to die if nobody bounds it, which is why the previous chapter's equivalence bound matters: the loop terminates when the interval becomes narrow enough to sit entirely on one side of the threshold, and it does not terminate on its own.

Gate one: is there a difference at all#

The measured +2.0 points sits inside a ±2.5-point band with a 95% paired interval of [-0.2, +4.2]. That interval does not support "memory improves task success by 2 points" and it equally does not support "memory does nothing", since a genuine 4-point gain lives comfortably inside it. The correct output of gate one, on that evidence, is that the experiment was too small and the decision has to wait for one that is not.

Enlarging the set to 2,350 paired tasks was six weeks of work using the construction process from the first chapter, with the break-even threshold written down beforehand so that the result could not be interpreted after the fact. The re-measurement came back at +0.4 points with an interval of [-1.0, +1.8]. That is a stronger statement than the first result in both directions: the gain is not distinguishable from zero, and any gain that exists is smaller than 1.8 points. An interval with a useful upper bound is what converts a flat result from an unresolved question into a finding.

The two-point figure from the smaller run was not a lie, and understanding where it went is worth a paragraph rather than a sentence, because the mechanism recurs. The first set over-represented the explicit-preference slice relative to production traffic, because the engineer building it naturally reached for the vivid cases — the ones where a user had said something memorable and the assistant had visibly used it. The previous chapter established that the aggregate is a slice-size-weighted average of the slice differences, so over-weighting the slice with a 13.3-point gain pulls the aggregate up without any slice's own behaviour changing. The enlarged set was sampled to match the production mix, the weights moved towards the slices that lose or do nothing, and the aggregate fell. The slice effects themselves did not change; their weights did.

Gate two: what would the gain have to be#

Break-even converts points of task success into money so it can be compared with the ledger, and the conversion is a chain of four multiplications worth walking rather than accepting. One point of task success means one percent of tasks that used to fail now succeed. So the number of failures avoided in a year is the monthly task volume, times one percent, times twelve. Multiplying that by what a single avoided failure is worth gives the annual value of one point. Dividing the annual cost by that figure gives the number of points the feature has to buy to pay for itself.

The one term in that chain that cannot be read off a dashboard is what a failed task costs, and for this product it was measured rather than assumed. Sampling 400 failed tasks showed an average of 0.9 additional assistant turns spent recovering, at $0.026 each, and a 4% rate of escalation to a human support contact costing about $6.20 fully loaded. That is roughly $0.27 per avoided failure.

Those two terms are worth comparing before moving on, because they are not the same size. The recovery-turn term is a fraction of a turn at the price of a turn, and it comes to a couple of pennies. The escalation term is a small rate multiplied by a cost hundreds of times larger than a turn, and it comes to about ten times the recovery term. Almost the whole value of avoiding a failure is the escalation it avoids, which means the break-even figure is sensitive to the escalation rate and the loaded cost of a support contact, and barely sensitive to token prices at all. A team that halves its inference cost has not moved this calculation; a team whose support contact costs twice as much has moved it a lot.

TASKS_PER_MONTH   = 1_200_000
VALUE_PER_AVOIDED = 0.27          # $ per task that fails less often
ANNUAL_COST       = 178_000       # inference + infra + 26 eng-days/quarter

value_per_point = TASKS_PER_MONTH * 0.01 * VALUE_PER_AVOIDED * 12
break_even_pp   = ANNUAL_COST / value_per_point

print(f"${value_per_point:,.0f} per point per year")   # $38,880 per point per year
print(f"break-even at {break_even_pp:.1f} points")     # break-even at 4.6 points

Memory has to buy about 4.6 points of task success to pay for itself at this scale, and the measured interval tops out at 1.8. The entire plausible range of the effect is below the threshold, which is the shape of evidence that settles a decision rather than deferring it: whichever value inside the interval is the true one, the answer comes out the same way, so the remaining uncertainty about the effect has stopped mattering. Note that this is only possible because the threshold was fixed before the measurement; a break-even computed afterwards can always be argued down to whatever the result happened to be.

The conversion carries one assumption that has already been seen to fail once, and it is worth stating rather than leaving buried. A point on the eval set is being treated as a point on production traffic, so the calculation is only sound if the eval set's mix of slices matches production's. That is precisely what the first, smaller set got wrong, and the correction moved the aggregate by more than the whole break-even margin. Anyone quoting a break-even figure therefore owes a statement about how the set was sampled, and a set built from vivid cases invalidates the arithmetic above rather than merely making it optimistic.

Two costs resist this arithmetic and belong in the discussion anyway. The 410 milliseconds were never converted into dollars, and the relevant fact is that p95 task latency, the time the slowest one task in twenty takes, moved from 4.8 to 5.9 seconds against a product service level objective of 6 seconds — an SLO being the performance target a team commits to holding and reports against — so memory consumed most of the remaining headroom and any future feature that wants latency has to take it from somewhere. The 51 regressions against 71 gains are not captured by a net figure at all, and the asymmetry described in the correction-turn analysis means they cost more per instance than the gains return.

Gate three: obligations at this size of gain#

If memory were buying eight points, the erasure fan-out, the residency duplication, the retention sweep and the incident blast radius would all be worth carrying, and the sensible response would be to invest in making them cheaper. At a bounded gain under two points, they are a standing commitment with nothing on the other side. The relevant question is not whether the obligations are tolerable in the abstract but whether they are tolerable in exchange for what has been measured.

The obligations also have a property the other costs do not: they persist after the feature is switched off. Deleting the retrieval path does not delete 4.1 million records, and a store nobody reads from is still retained personal data with the same erasure and access duties. Its disclosure risk is larger rather than smaller, because the component is now unwatched: nobody is looking at its dashboards, nobody is on call for it, and a leak from it produces no wrong answers to notice. A rollback plan has to include the data, with customer notice and a defined deletion date, or the worst line in the ledger survives the decision.

What survives the rollback#

Removing the feature wholesale would throw away the one part of it with evidence behind it. The slice analysis said explicit user-set preferences gained 13.3 points while extractor-inferred facts lost 3.3, and those two things share almost nothing except the word "memory": one is a user typing an instruction they expect to persist, the other is a model guessing what to retain.

Two of the three keeps have no measured gain in the evidence column at all, and that is deliberate rather than inconsistent. The eval suite and the trace instrumentation are instruments rather than features: they produced every number in this topic, they cost nothing per task, and discarding them means that any future attempt starts from the position the first chapter opened in.

The kept preference path is deliberately not a memory system. Preferences become typed rows against a user, set through an explicit command or a settings panel, rendered into the prompt as a 40-token block on every task with no embedding, no ANN lookup and no ranking. Removing the ranking removes the failure mode along with the machinery: there is no top-k to fall out of, so a preference that exists is always in the prompt. The user can see and edit them, which removes the stale-record failure mode by giving the correction to the person who knows the answer, and it makes access requests trivial because the data is already legible.

Re-running the 1,000-task eval with only that path is a fair comparison for a reason worth checking rather than assuming: the memory-off arm is untouched by the change, so its score should come back where it was, and it does — 74.0%, the same figure the full system was measured against. A memory-off arm that moved would mean something other than the read policy had changed between the two experiments, and the comparison would be worthless. Watching the control arm reproduce itself is the cheapest available audit of a re-measurement.

That run gave 76.4% against 74.0%, a difference of +2.4 points with 45 gains and 21 regressions, a 95% interval of [+0.8, +4.0] and a McNemar p of 0.005. That interval is worth checking against the identity from the second chapter rather than taking on trust: the standard error of a paired difference is the square root of the discordant count over the number of tasks, and 45 plus 21 discordant tasks over 1,000 gives a standard error a little over eight-tenths of a point, so stepping the usual two standard errors either side of +2.4 lands almost exactly on the published bounds. The added cost is 40 tokens on the 30% of tasks that have a preference set and 8 milliseconds of a primary-key lookup, and the maintenance is ordinary product code rather than a stateful subsystem, which puts the annual cost near $22,000 and the break-even near 0.6 points. The narrow version clears its own threshold by a margin the wide version never approached, and it does so because the expensive machinery was serving the slice that was losing.

Retiring something the team is proud of#

The engineering part of this decision is easier than the human part, and a few things make the human part survivable. The decision rule goes in writing before the enlarged measurement runs, including the break-even figure and what would happen at each outcome, so that the meeting afterwards is about a pre-agreed criterion rather than about whether the number is good. The people who built the system run the re-measurement themselves, because a result produced by a sceptic is contested and a result produced by the authors is not. The announcement names the criterion and the evidence rather than a verdict about the work, and it names what is being kept, which on this rollback was three of six components and included the two most reusable ones.

The memo should also say what would justify trying again, in terms specific enough to check. Everything in gate two scales with quantities that are properties of the traffic rather than of the code, so a change in the traffic changes the answer without anybody writing a line. For this product the condition written down was a shift in workload: if the median customer's sessions per project rise above twenty, or the share of requests whose answer depends on something established more than a session ago rises above the 12% discordance measured here, the calculation changes and the suite already exists to redo it.

That is a real possibility rather than a consolation. Memory's value is a property of the traffic more than of the architecture, and the comparative studies of memory systems keep finding that the ranking of designs moves with the workload — which means a decision reached here is a decision about this product now, and stating the conditions under which it expires is part of making it honestly.

What generalises out of this topic is a standing practice rather than a verdict. Every stateful feature that injects context gets a paired held-out set before it ships, a measured noise band, a write-then-read design that a null-store ablation cannot pass, traces that attribute failures to a stage, a regression count reported next to the net difference, and a ledger that includes the engineering time and the obligations. Run that on the next feature and the meeting takes ten minutes instead of forty, because the numbers arrive in an order that can settle something.