Model-decided, heuristic and explicit gates trade recall for precision

The rules for what belongs in a memory store say nothing about who applies them. A model can be shown a transcript and asked which sentences are worth keeping. A set of regular expressions and policy rules can be run over the user's turns. A button can be put in the interface and the user can press it. All three implement the same policy in the sense that they are trying to admit durable, well-evidenced, in-scope facts and reject the rest, and they disagree wildly about which sentences those are.

They disagree in a way that reproduces, which is what makes the comparison worth making rather than running once and forgetting. This chapter reports one replay of three gates over real conversations at an analytics product called Arbor, and the numbers in every row of it will differ in your system. What will not differ is the ordering, and the ordering is what a design decision has to be made against.

What the replay measured, and what the two numbers mean#

Arbor's team settled the argument by replaying it. They pulled 500 real conversations, 6,100 assistant turns in total, and had two support engineers label every candidate fact in them against the boundary from chapter 1 — durable, well-evidenced, scoped. The labellers agreed on 410 facts that should have been written and disagreed on 47 more, which were dropped from the set rather than adjudicated. Then they ran three admission gates over the same transcripts offline and compared what each one produced against that reference.

Two numbers carry the comparison, and they answer different questions about the same store. Precision is the share of the records a gate wrote that the labellers agreed deserved to be written, so it measures how much of the store is junk. Recall is the share of those 410 reference facts that the gate actually captured, so it measures how much the store is missing. The two have different denominators, which is the whole reason a single accuracy figure cannot stand in for them: precision divides by what the gate produced, recall divides by what existed to be found, and a gate can move one without touching the other. Added latency is reported at p50 and p95, the median and the ninety-fifth percentile of the delay the gate contributes to a turn, since a gate that is usually fast and occasionally slow is felt by users as a slow gate.

The last column is not an independent measurement, and seeing that it is derived is the quickest way to understand what the table is saying. Wrong records per thousand turns is the write rate times a thousand, multiplied by the share of writes that were wrong, which is one minus the precision. The top row writes on 31% of turns, so three hundred and ten writes per thousand turns, of which thirty-eight in every hundred are wrong, giving 118. The second row writes a hundred and eighty per thousand at twenty-one in a hundred wrong, giving 38. The heuristic writes sixty at seventeen in a hundred wrong, giving 10. Every row reconstructs the same way, including the explicit gate's 0.1 and the confirmed gate's 1.9. So the last column adds no information the first three do not carry; what it adds is a unit, and the unit is the one that matters, because a correction process is staffed against wrong records per day rather than against a precision score.

That identity licenses a specific move: it lets you convert any point on this curve into a workload. Its residual is that it counts records and not their reach. A wrong record that is never retrieved and a wrong record that is retrieved into dozens of answers both count as one, and the next chapter is largely about why that difference is the one you actually care about.

Read the rest of the table as gaps rather than as levels, because the levels are properties of Arbor's data and the gaps are properties of the gates. Moving from the first row to the second costs seven points of recall and buys seventeen points of precision, which cuts wrong records by roughly two thirds. Moving from the second row to the third costs a further twenty-six points of recall and buys four points of precision. Moving from the third to the fourth costs twenty-nine points of recall and buys fourteen. The first of those three steps is the only one where the trade is lopsided in your favour, and it is also the cheapest to make, since it is a prompt edit rather than a new system.

The ordering itself is the result to remember, because it reproduces across products and does not depend on which model or which regexes you used. A model asked to judge has the highest recall and the lowest precision. A user pressing a button has the highest precision and the lowest recall. Everything else lands between them, and the numbers in any given row will differ in your system while the shape stays.

One caveat belongs with the precision column before it is used to decide anything. The reference set was built by dropping the 47 facts the two labellers disagreed about, which means every gate is being scored on the subset of candidates that two humans found unambiguous. Ambiguous candidates are precisely where a gate's judgement is hardest and where its errors concentrate, so a precision figure computed this way is an upper bound rather than an estimate. The number is stable enough to compare gates against each other, which is what it is for here, and it is not the number to quote when somebody asks how good the store is.

What each gate is actually good at#

The model gate earns its recall on the facts no pattern will ever match. A user who writes "we finally got off Snowflake last quarter, the migration was miserable" has told you their warehouse changed without using any of the words a warehouse-detection rule would look for. Facts assembled across several turns behave the same way: the user mentions a repository in turn 3, says "we use pnpm here" in turn 9, and only a reader tracking the conversation connects the two into a scoped preference. Of the 410 reference facts, 118 were of this kind, stated obliquely or spread across turns, and the heuristic gate found 4 of them.

Its errors are equally characteristic. Roughly half of the model gate's false positives at the 31% setting were situational statements promoted to durable ones, which is the chapter 1 boundary being crossed by a judge that was never asked about durability. Another large group were the assistant's own claims read back as user facts, because the extraction prompt received the whole transcript and did not distinguish speakers reliably — records like "user's export runs at 02:00 CET" traced back to a turn in which the assistant proposed that schedule and the user said nothing. The rest were inventions of scope, where a preference expressed about one dashboard came out as a global rule.

The second row in the table is one prompt edit, and it moves more than most model swaps would. The reason is in the shape of the question rather than in the model's capability. "Is this worth remembering?" asks whether a fact has value, and almost any fact is arguably worth remembering, so there is nothing in a candidate that can supply evidence against it — the judgement has no negative side to weigh and collapses into a yes. Asking instead whether a future session would need the fact, and whether it will still be true in three months, gives the judgement two ways to fail that a candidate can actually exhibit, and forces it onto the axis that matters.

EXTRACT_PROMPT = """You are deciding what to store in long-term memory about a user.

For each candidate fact, answer three questions before proposing it:
1. Will this still be true in three months? If it has a natural end
   (a trip, a deadline, a sprint), it is a situation, not a preference.
2. Did the USER state it, or did the assistant assert it and the user
   not object? Only user-stated facts are admissible.
3. Would a session next month be worse without it? Not "nice to have" —
   worse.

Reject anything that fails any of the three. Rejecting is the default.
Return JSON: [{"statement", "kind", "scope", "expires_at", "quote", "turn"}]
An empty array is a normal and frequent answer.
"""

The heuristic gate is the opposite trade. It is a set of patterns and slot rules over the user's turns only — a slot being a named thing the store has an opinion about, such as the warehouse or the preferred units, so that two records filling the same slot are talking about the same question — and it does exactly what is written in it:

import re

PATTERNS = [
    # (regex, kind, scope, claim template)
    (r"\b(?:our|we use|we're on)\s+(?:warehouse\s+is\s+)?"
     r"(bigquery|snowflake|redshift|databricks)\b", "fact", "global",
     "Data warehouse is {0}."),
    (r"\b(?:always|from now on|in future|every time)\b.{0,60}?"
     r"\b(metric|imperial|utc|local time)\b", "preference", "global",
     "Standing instruction: {0}."),
    (r"\bcall me ([A-Z][a-z]+)\b", "preference", "global", "Prefers to be called {0}."),
]


def heuristic_gate(user_turn: str, ctx) -> list[dict]:
    out = []
    for pattern, kind, scope, template in PATTERNS:
        m = re.search(pattern, user_turn, re.I)
        if not m:
            continue
        if negated(user_turn, m.start()):      # "we're not on Snowflake"
            continue
        if hypothetical(user_turn):            # "if we were on Redshift..."
            continue
        out.append({"kind": kind, "scope": scope,
                    "statement": template.format(*m.groups()),
                    "quote": m.group(0), "turn": ctx.turn})
    return out

Everything attractive about this gate follows from it being ordinary code, and the chain is worth making explicit because it is what people mean when they say a heuristic is "simple". A regular expression is evaluated in the process that received the turn, so no network call is involved, which is why it runs in about two milliseconds and can sit on the hot path — the work that happens while the user waits for the turn to come back — without anyone noticing. It is deterministic, in the sense that the same input produces the same output every time, so a record that appeared can be explained by pointing at a line rather than by re-running a sampler and hoping. Because the rules are text in a repository, they have unit tests, they show up in code review as a diff, and when the gate produces a bad record you fix the rule and the class of error is gone rather than reduced in probability. What it cannot do is exceed its own pattern list, and the list grows one incident at a time — Arbor's had 34 patterns after a year, each traceable to a specific complaint, with recall still under 0.4.

The pattern count is worth reading as a rate rather than as a total. Thirty-four patterns after a year, each traceable to a specific complaint, is roughly a pattern every week and a half, and recall was still under 0.4 at the end of it. That tells you the shape of the curve the heuristic is climbing: each new pattern covers one more way of saying one more thing, and the ways of saying things do not run out. A gate whose recall improves by a constant amount per incident will reach a good number eventually, and a gate whose incidents each cover a shrinking slice of remaining ground will not, which is why a year of diligent pattern-writing moved recall so little.

The explicit gate barely writes at all, and its numbers are worth reconstructing because they are the smallest in the table. Users pressed the "remember this" affordance on 0.4% of turns, which over 6,100 turns is 24 records, and 23 of them were correct — twenty-three out of twenty-four is the 0.97 in the precision column, computed from a sample small enough to fit in a paragraph. The one error was a user who asked the assistant to remember a project code name and typed it wrong, which is a failure of the user rather than of the gate and has no fix on the gate's side. That is as close to a clean signal as this problem offers, and it covers less than a tenth of what a well-run store should contain, because remembering to press a button is not a thing people do while trying to get work done.

Proposing the write and asking for confirmation recovers a good part of the recall. "Want me to remember that your warehouse is BigQuery?" was offered on 6.1% of turns and accepted 54% of the time, and those two figures multiply into the 3.2% write rate in the last row, at near-explicit precision. It is not free: those prompts interrupt, and Arbor found session abandonment measurably worse once the assistant asked more than once per session, so the confirmation gate has to ration itself. The usual rationing rule is impact — confirm facts that change what the assistant does, let style preferences accumulate silently through the promotion log from chapter 2.

The three gates also differ in something the table cannot show, which is what happens when one of them misbehaves at three in the morning. A heuristic gate that starts writing garbage has a cause you can find by reading a regex and a fix you can ship in an hour. A model gate that starts writing garbage after a provider updates the model behind an alias has no diff to read, and the remedy is prompt work evaluated against a reference set you hope somebody kept. An explicit gate cannot really misbehave, since it only writes what a user asked it to. Teams underweight this at design time and then discover it during the first regression, which is usually noticed weeks late for the reasons the previous chapter set out.

Stacking them beats choosing one#

The arrangement that works in practice uses all three, in a pipeline where each stage can only remove or downgrade:

def admit(turn, ctx, store):
    candidates = model_extract(turn, ctx)          # high recall, noisy
    candidates += heuristic_gate(turn.text, ctx)   # cheap, precise, narrow

    kept = []
    for c in dedupe(candidates):
        if not durable(c) and not c.get("expires_at"):
            continue                                # chapter 1 boundary
        if c["source_speaker"] != "user":
            continue
        if weak_evidence(c):
            log_observation(c)                      # chapter 2 promotion path
            continue
        kept.append(c)

    high_impact = [c for c in kept if c["kind"] == "fact"]
    quiet = [c for c in kept if c["kind"] == "preference"]
    store.put_all(ctx.user_id, quiet)
    return propose_confirmation(high_impact[:1])    # at most one per session

The ordering matters, and each of the three constraints on it comes from a different property. Generation comes first because a later stage can only discard what it was handed, so recall lost at the top of the pipeline cannot be regained lower down, whereas precision lost at the top can be. The policy layer runs after generation rather than inside it because by then the candidates are structured objects with a kind, a scope and an expires_at, and checking a field is a line of code where checking the same thing in raw text would be another model call. Confirmation comes last because it is the only stage that costs the user something, so it should see the smallest possible queue, which is what every stage above it has been shrinking. Deduplication before the policy loop is the fourth piece: without it, the model's and the heuristic's overlapping proposals for the same fact would otherwise be counted as two independent observations, which lets a single sentence promote itself on the strength of having been read twice.

Where on the curve you are allowed to sit#

Reading the table as "the model gate is inaccurate" gets the shape wrong. Every row is a point on the same curve, and moving along it trades one error for the other. The reason the curve has a preferred direction here is the cost asymmetry from the previous chapter: recall failures are paid once, in a re-ask, and precision failures are paid indefinitely. That is what makes 0.62 precision at 31% write rate a bad place to sit even though it captures the most facts, and it is why the durability-framed prompt at 0.79 and 18% is a better system despite finding fewer true facts.

Where you sit on the curve should also move with what you have built around the store, and this is the condition most teams leave implicit. The asymmetry argument assumes bad records are permanent. A team with user-visible records, a working delete path, sampled weekly audits and contradiction detection has broken that assumption, so bad records have a bounded life and the extra recall is worth having; such a team can afford the model gate at 18%. A team whose store is a table nobody can see, with no audit and no correction flow, has not broken it, and has no business running anything looser than the heuristic, since every mistake it makes is permanent by construction. The same product can therefore be at two different points six months apart without anyone changing their mind about anything, which is the sense in which this is a position rather than a preference.

The intuitions from reranking do not transfer, and it is worth being explicit about why, since the two problems look superficially similar. Ranking recall failures at read time are recoverable — the next query re-runs the whole selection and a document that lost today can win tomorrow. Admission recall failures are recoverable too, by the user restating the fact. Admission precision failures are the ones with no recovery path, because nothing downstream re-examines whether a stored record deserved to be stored.

None of the three gates settles where records live, how they are indexed, or how conflicts are resolved at read time. A gate answers one question — does this candidate enter the store — and a system with a beautifully tuned gate and no reconciliation still ends up with two records disagreeing about the warehouse. Nor is the ordering a statement about model quality. A stronger model raises the model gate's whole curve, and it does not turn the model gate into the explicit gate, because the gap is about who holds the ground truth. The user knows whether they always want metric units. The model is inferring it from a sentence.

The choice among the three is therefore not a choice about which is best at judging sentences. It is a choice about which error you have built the machinery to survive, and the honest form of the decision names both halves: this is the write rate we run at, this is the precision we have measured on it, and this is what happens to the records we get wrong.

Turning a live gate's behaviour into precision, recall, latency and review burden — and deciding what those figures should trigger — is the next chapter.