Write rate is a cost with an error rate attached

Write rate — the share of assistant turns after which the gate admits at least one record — is the one memory metric every system already has, because it is a counter on a code path that already exists. That is exactly why it ends up standing in for quality. It is free, it is continuous, and it moves in a direction that is easy to narrate.

What it actually measures is how fast the store is accruing obligations. Every admitted record is something that can be retrieved into a prompt, something a user can demand to see and to have removed, and something a later extraction pass can read back as though it were evidence. None of those depend on whether the record is correct, which is why the rate on its own is silent about quality: it counts the cost side of every write and says nothing about the benefit side of any of them. Reading it as a personalisation win requires an assumption about the error rate that nobody on the slide has stated.

Four numbers, and what each one costs to get#

Turning a write rate into a decision needs four measurements, and they differ enormously in how expensive they are to obtain, which is why most teams have the cheapest one and none of the others.

The table has six rows and the heading promises four numbers, which is not a miscount but a claim about which of them decide anything. Write rate decides nothing on its own, as the paragraphs above laboured; it is the context the others are read against. Restatement rate is a live proxy for one of the four rather than a fifth measurement, standing in for recall between the quarterly replays that produce a real recall figure. What is left — precision, recall, added latency and review burden — are the four that carry an action threshold you could act on today, and they are ordered in the table by how much they cost to obtain rather than by how much they matter.

Precision is the one to build first, and the sampling has to be stratified — drawn in fixed quantities from each gate and each record kind rather than uniformly from everything written — or the number will be dominated by whichever gate and record kind happen to be most common. The reasoning is worth stating because it decides the shape of the sampler. A uniform sample estimates the average over the population it was drawn from, so if one gate produces most of the writes, a uniform sample measures that gate and reports the answer as though it described the store. What you want instead is a per-stratum estimate you can then combine, which requires enough records in every stratum to estimate anything at all, and that is what a floor buys.

Arbor sampled 200 records a week, drawn proportionally across gates and across the fact / preference / situation kinds with a floor of 20 per stratum, and had two support engineers label each one against four criteria: is the claim true, is it durable, is the scope right, and did the user state it rather than the assistant. A record fails if any criterion fails, which is a deliberate choice rather than an obvious one: a record that is true but wrongly scoped will produce wrong answers on the turns outside its real scope, so treating the four as a conjunction rather than scoring them separately keeps the metric aligned with what goes wrong.

Inter-labeller agreement ran at a Cohen's kappa of 0.71, a score that discounts the agreement two labellers would reach by chance alone and runs from 0 for coin-flipping to 1.0 for identical verdicts, which is good enough to act on and not good enough to publish precision to two decimal places — at n=200 the 95% interval is roughly ±7 points, so a move from 0.62 to 0.66 is not a move. That last point is the one people skip, and it has a consequence for how often the number is worth looking at: a weekly figure whose interval is that wide will wander by several points from week to week with nothing having changed, so the thing to act on is a level that persists across several samples, not a week-on-week difference.

import random
from collections import Counter

def weekly_sample(records, n=200, floor=20):
    """Stratify by (gate, kind), with a floor so rare strata are visible,
    then weight the remainder by how often each record was retrieved."""
    strata = Counter((r.gate, r.kind) for r in records)
    picked = []
    for key in strata:
        pool = [r for r in records if (r.gate, r.kind) == key]
        picked += random.sample(pool, min(floor, len(pool)))

    remaining = n - len(picked)
    rest = [r for r in records if r not in picked]
    weights = [1 + r.retrieval_count for r in rest]   # see below
    picked += weighted_sample_without_replacement(rest, weights, remaining)
    return picked


def precision(labels):
    ok = sum(1 for l in labels if l.true and l.durable and l.scoped and l.user_stated)
    return ok / len(labels)

Recall is the expensive one, because it requires knowing what should have been written, which only a human reading whole conversations can say. There is no way around that: a gate's misses are by definition invisible in the gate's own output, so the reference has to be produced by something that read the same input and does not share the gate's blind spots. The 500-conversation labelled set from the previous chapter cost about three person-days to build and can be replayed against a new gate in an afternoon, so the sensible pattern is to build it once, treat it as a fixture, and refresh it quarterly as the product changes.

Between refreshes you need a live proxy, and the one that works is the restatement rate: a small classifier watching for turns where a user asserts a fact the store already has a taxonomy slot for, having asserted it before. The logic is that a missed write has an observable consequence — the user has to say the thing again — so counting the consequence estimates the cause. Users restating things is measured continuously and for almost nothing, and unlike recall it needs no labelled set. Its weakness is the other direction of the same reasoning: a user who has given up restating a fact is a missed write that generates no signal at all, so the proxy undercounts exactly the users who have stopped expecting the product to learn.

Latency has a subtlety that trips people up when they read the dashboard, and it comes from the two rates having different denominators. The gate runs on every turn, because deciding not to write requires reading the turn first. The write happens on 31% of them. So a gate adding 310 ms at p50 is adding it to all 12,000 daily turns rather than to the 3,720 that produce a record. Deciding not to write is exactly as expensive as deciding to write.

Moving extraction off the hot path removes it from the user's turn entirely, at the price of a staleness window — Arbor's background writer ran 4 to 9 seconds behind, which is invisible across sessions and useless within one, since a fact stated in turn 3 is not in the store when turn 4 is assembled. That is usually fine, because the current session's transcript is already in context and supplies the same fact by a different route. It stops being fine for agents that spawn parallel sub-sessions expecting a shared store, because a sub-session has no transcript to fall back on and the store is the only channel between them.

Review burden is arithmetic that most teams do not do, and it is where the write rate finally acquires a meaning. Start from the volume. At 12,000 assistant turns a day and a gate firing on 31% of them, the system adds 3,720 rows a day to a store nobody reads. At the model gate's measured precision of 0.62, thirty-eight in every hundred of those 3,720 daily writes are wrong, which is roughly 1,414 records a day that should not exist. Over a year that is half a million wrong claims, each one carrying a deletion obligation, each one eligible to be retrieved into a prompt, and each one a candidate to be re-extracted into a second-generation record by a summariser that reads the conversations it distorted. Spread over 3,400 active users, the total write volume alone is a little over one new claim per user per day, every day, arriving faster than any human could look at them and faster than most users generate genuinely new durable facts about themselves.

Now put a rate against that volume. At a realistic audit rate of 25 records an hour, reviewing every wrong record would take 56 person-hours a day. Reviewing every record, wrong or not, would take 149. Neither is a plan, which is the actual finding: at a 31% write rate there is no staffing level at which humans keep the store clean, so precision has to be high enough that the store does not need cleaning. Sampling 1% of a day's writes for measurement is about thirty-seven records, which at the same rate is 1.5 hours and is affordable; treating human review as a correction mechanism rather than a measurement mechanism is not.

The review arithmetic has a second reading that is easy to skip past. Human review capacity is roughly fixed, so it sets a ceiling on the write rate the organisation can support at any given precision, and the two numbers move together. Hold the volume at 3,720 daily writes and raise precision to 0.95, and and the wrong-record count falls to 186 a day, which one person can work through in about seven and a half hours if they do nothing else — still not a plan, but at least a conceivable one for a regulated subset. At 0.62 it is not conceivable at any headcount you would get approved. Write rate and precision are therefore not independent dials that you tune separately; they multiply into a single quantity, wrong records per day, and that quantity is what any correction process has to be sized against.

That identity — turns per day, times write rate, times one minus precision — licenses exactly one conclusion, and it is worth being precise about which. It tells you the size of the workload a given operating point creates, in the unit a staffing conversation happens in. What it does not license is any statement about how bad the workload is, because it weights every wrong record the same. That residual is not a small one, and the next section is entirely about it.

Precision on written records is the wrong denominator#

There is a flaw in the number above that matters more than the confidence interval, and it is a flaw in the reference population rather than in the measurement. Precision measured over written records treats every record as equally consequential, and they are not. A wrong record that is never retrieved into a prompt costs you storage, a row in the user's memory settings page, and a deletion obligation. A wrong record retrieved into 61 responses over six weeks costs all of that plus 61 distorted answers. Sampling uniformly over what was written measures the health of the store, and what you care about is the health of what reaches the model.

Those two populations only coincide if wrong records are retrieved at the same rate as correct ones, and there is no reason for that to hold. The fix is to weight the sample by retrieval count, which is the 1 + r.retrieval_count line in the sampler above, and the one is there so that a record retrieved zero times still has some chance of being drawn rather than none.

Arbor ran both and the numbers diverged sharply: uniform precision was 0.62, and retrieval-weighted precision was 0.48. The direction of that gap is the finding. Weighting can only pull the figure down if wrong records carry more retrievals on average than correct ones, so the store was not merely producing junk, it was producing junk that the retriever preferred. That sounds surprising until you look at what the model gate over-produced — broad, general-sounding claims about how the user likes things, phrased in language that matches a wide range of queries. Records like "user prefers detailed technical explanations" match almost any embedding you throw at them, and records like "pnpm in the billing-service repo" match almost none. The gate's failure mode and the retriever's similarity function were aligned in the worst possible direction: the same vagueness that made a claim unjustifiable made it easy to retrieve.

That divergence is also the argument against reporting store size as a health metric. Two stores of 500,000 records, one where 4% of records serve 80% of retrievals and one where retrieval is spread evenly, are different systems with different risk profiles, and the row count cannot tell them apart. The retrieval-weighted number is not free of the same problem one level down — it counts retrievals rather than influence, and a record can be placed in a prompt and ignored — which is the gap the next chapter is built to measure.

What the numbers are supposed to trigger#

Measurement is only worth the cost if the thresholds are agreed in advance, and the useful ones are specific about which way you move.

Precision below about 0.85 with no working correction path is the signal to tighten the gate, and tightening means the durability-framed prompt first, since it is a prompt edit rather than an architecture change, and a heuristic gate for the record kinds where the errors concentrate second. The order is chosen by cost of change rather than by size of effect, on the grounds that a cheap change you can reverse is worth trying before an expensive one you cannot.

Moving to an explicit gate is a different trigger, and it is about consequence rather than accuracy. Facts that change what the assistant does — an approval threshold, a deployment target, a billing contact, anything that will be acted on rather than mentioned — should be confirmed by the user regardless of how good the model gate looks in aggregate. The reason is that an aggregate hides the units: 0.9 precision on a class of records that authorise actions is a 10% wrong-action rate, and one in ten is a number nobody would sign off if it were written that way. The same applies to anything that would be sensitive to show back on a settings page, where the cost of the record being wrong includes the cost of the user seeing that you believed it.

None of these thresholds mean much unless somebody owns the dashboard, which is the quiet failure mode of memory metrics generally, and it has a mechanism rather than being bad luck. Write rate goes on a product dashboard because it is free and it goes up. Precision goes on nobody's dashboard because it costs three hours a week and it goes down. Six months later the only number anyone has watched is the one that carries no information, and no decision was ever taken to make that happen. The cheapest defence is to put write rate and retrieval-weighted precision on the same tile, so that neither can be quoted without the other, and to make the weekly sampling a scheduled job with an owner rather than an activity someone does when they have time. That is a design decision about incentives rather than about measurement, and it is the one that determines whether any of the measurement survives contact with a quarter.

Rising restatement rate while precision holds steady is the signal to loosen instead, which is worth stating because a team that has internalised the asymmetry from chapter 2 can over-correct into a store that admits nothing. Users repeating themselves is the visible price of a tight gate, and when that price shows up in the data, the gate has room to give. The pairing matters: restatement rising with precision falling means something else is wrong, and the two signals are only interpretable together.

A write rate quoted without an error rate beside it is a cost quoted without a price. The arithmetic that turns one into the other is three multiplications, it fits on the same tile, and it is the difference between a memory system whose operating point somebody chose and one that arrived at its operating point because the number was green.

Everything so far has been measured on the write path — what was proposed, what was admitted, how right it was. That leaves the question the retrieval-weighted number hinted at without answering: whether a bad record in the store actually changes what the agent says and does in a later session, or merely sits there. Reasoning about that is weaker than causing it, and the next chapter deliberately poisons a store to find out.