Agent memory and unavailable tools

Agent memory here means everything the model already has in its context: earlier turns of the conversation, the system prompt, a product description it read two minutes ago, and whatever it retains from training. It is the material a model draws on when a tool does not answer, and it is the reason unavailability is not a neutral event. A missing inventory count degrades an answer; a fabricated price is a commercial statement your company did not make. The two need different handling, which is the argument against the reflex of wrapping every server in the same retry-with-backoff and letting the model sort out the consequences.

The measurements below assume the hosted endpoints of chapter 2, on local and hosted server builds, which run 140 ms at p50 and 2.1 s at p99.

Why agent memory fills the gap a tool leaves#

The mechanism is worth seeing at the message level, because the fabrication is not mysterious once you look at what the context contains:

[
  {"role":"user","content":"How much is the winter jacket, JKT-114?"},
  {"role":"assistant","tool_calls":[{"id":"c1","name":"pricing_quote",
     "arguments":{"sku":"JKT-114","currency":"EUR"}}]},
  {"role":"tool","tool_call_id":"c1",
     "content":"Error: request to pricing-mcp timed out after 30000ms"}
]

That last message is text in the context window, and text in the context window is an input the model reasons over rather than an instruction it obeys. A tool error competes with everything else present — the customer's question, the system prompt's instruction to be helpful, an earlier turn where a price appeared in a product description — and the model resolves the conflict by continuing. Some models will say they could not retrieve the price; the failure mode of interest is the one where they do not, and across a large enough volume of conversations both behaviours occur.

Spelled out, the steps are each checkable on their own. A model is given a sequence of messages and produces the continuation it judges most likely. A tool result is one of those messages, and it carries no more authority than any other message present. So a message reporting a timeout competes with the customer's question, with a system prompt instructing helpfulness, and with any earlier turn in which a price appeared — and it competes on level terms, because nothing in the format marks one message as binding and another as advisory. The continuation satisfying the most of what is present is an answer that contains a price, and a price is available, since the model read one in a product description two minutes ago. No step in that sequence is a malfunction. Each is a component doing what it was built to do, which is why the remedy has to be a different arrangement of components rather than a firmer instruction to this one.

The belief this defeats is that telling the model about the failure is a control. It is tempting because it is true of people: a colleague told the pricing system is down says so rather than guessing. It is false here for a reason that has nothing to do with the model being careless — the message is an input rather than a directive, and the format contains no mechanism by which an input becomes binding. Some models do report that they could not retrieve the price, which is exactly what keeps the belief alive, and across enough conversations both continuations occur. A control that holds most of the time, with nothing forcing it, is a tendency, and a tendency is not what belongs between a customer and a price.

The design conclusion is that unavailability is an orchestration decision. The code around the model chooses what happens when a server is down, and the model is never handed a decision it can reinterpret.

type Policy =
  | { kind: "omit"; disclose: string }
  | { kind: "cached"; maxAgeMs: number; disclose: (asOf: Date) => string }
  | { kind: "hard_fail"; message: string; handoff: "human" | "retry_later" };

async function withPolicy(name: string, call: () => Promise<Result>, p: Policy) {
  try {
    return { ok: true as const, result: await call() };
  } catch (err) {
    recordUnavailability(name, err);
    switch (p.kind) {
      case "omit":
        return { ok: false as const, toolRemoved: true, note: p.disclose };
      case "cached": {
        const hit = await cache.get(name);
        if (!hit || Date.now() - hit.asOf.getTime() > p.maxAgeMs) {
          return { ok: false as const, terminate: FALLBACK_UNAVAILABLE };
        }
        return { ok: true as const, result: hit.value, note: p.disclose(hit.asOf) };
      }
      case "hard_fail":
        return { ok: false as const, terminate: p.message, handoff: p.handoff };
    }
  }
}

The terminate branch does not ask the model to apologise and stop. It ends the turn in code and emits a fixed message, because a request to stop is still a request the model weighs.

Three servers, three policies#

An agent answering customer questions over an inventory server, a pricing server and an internal search server has three dependencies with three different consequences of being wrong, so a uniform policy is either too strict for one or too loose for another. The retry column can be filled in at all only because these calls are idempotent reads, meaning a repeat has no effect beyond returning the answer a second time.

Inventory tolerates omission because "I can't confirm stock right now, the item is listed at €209" remains a useful and truthful answer, and the customer discovers availability at checkout regardless. Pricing does not tolerate omission or improvisation, because a number is what was asked for and any number offered without a source is a claim the business will be held to; there is no partially correct price. Internal search sits at the other end, since its results improve an answer that the model can still give from the conversation and the other tools, and disclosing its absence to a customer communicates nothing they can use.

The uniform alternative — one retry-with-exponential-backoff wrapper and a circuit breaker around all three, with the tool error passed to the model to judge — is attractive because it is a single well-understood component and because the model does hold the conversational context. Both halves of that argument fail on inspection. A shared retry policy sets one timeout, and the value that is right for internal search at 5 seconds spends twice the acceptable budget on pricing before failing; a shared circuit breaker — the component that stops calling a dependency once failures pass a threshold and lets calls through again after a pause — opens on the failure count of whichever server is flakiest and takes the others' behaviour with it. Delegating the decision to the model puts the choice in exactly the place that cannot enforce it, since the error arrives as text and the model's job is to produce a helpful answer from the text it has. Retries and breakers are still useful mechanics underneath, configured per server; what does not survive is one policy for three consequences.

The timeouts are set from measured behaviour rather than from a round number. With hosted endpoints running 140 ms at p50 and 2.1 s at p99, a 2.5-second ceiling accepts essentially every healthy call and rejects the tail. A 30-second timeout is more than fourteen times the p99, so it converts a fast failure into a slow one and spends half a minute of a customer's patience before failing anyway. The whole turn gets a budget too — around 8 seconds of tool time in a chat context — because three servers each allowed 2.5 seconds plus a retry can otherwise stack into 15 seconds of silence.

Both of those multiples are worth checking rather than taking on trust, because the check is the argument. Thirty seconds against a p99 of two point one seconds is a ratio of more than fourteen, which puts the ceiling far outside the range healthy calls occupy, and a ceiling outside that range cannot distinguish a slow call from a dead one. Every call it eventually catches was already lost; the extra time buys nothing except the customer waiting through it. The fifteen seconds of silence comes from the other direction. Two and a half seconds for a call and two and a half for its retry is five seconds per server, and five seconds across three servers is fifteen. Set against an eight-second budget for the whole turn, fifteen is nearly twice what the turn is allowed to spend, which is why the budget has to sit above the per-call ceilings rather than being implied by them.

The table's own figures give a different worst case, and the gap between the two is the point of writing a policy per server. Inventory allows two seconds and one retry, which is four. Pricing allows two and a half and one retry, which is five. Internal search allows five and takes no retry, which is five. Four, five and five is fourteen seconds — under the uniform policy's fifteen, still comfortably past the eight-second budget, and reached only in the turn where all three time out together. The budget enforced across the turn is what converts that possibility into a bound, and it is the reason the per-call ceilings on their own do not amount to a policy.

Closing the routes around the policy#

A policy the model can bypass is a suggestion. Four implementation properties make it binding.

The tool is removed from the turn rather than left callable. When a server is unavailable, the tool list sent to the model for that turn omits its tools, so there is no call to attempt and no error to reinterpret. This matters more than it sounds: a model that can see pricing_quote in its tool list will keep calling it, accumulating failures and tokens, and each failure adds another error string to the context.

No raw error text reaches the context. Failures are recorded to logs and metrics where operators read them; the model receives either a structured absence or nothing. The disclosure string, where there is one, is written by you, in the product's voice, and is a fixed input rather than a description of an exception.

Fallback data arrives with provenance attached. A cached value is passed as a structured result carrying its as_of timestamp and a source field, and it is served only while it remains inside its time-to-live (TTL), the maximum age at which a stored copy still counts as usable, so the response the model builds on is self-describing, and the template that renders the answer can require the timestamp to be shown.

The fourth property is a check after generation, not before. For claims that must be sourced — a price, a delivery date, a balance — the response is scanned for the claim type before it is sent, and any price token that does not match a value returned by a tool in this turn stops the message. That check is what catches the case the first three properties miss, where the model volunteers a price nobody asked it to look up. It costs a regular expression and a set lookup, and it is the only mechanism in this chapter that acts on the output rather than on the input.

Those four properties are worth reading as a conjunction with one term that is unlike the others. Removing the tool, withholding the error text and attaching provenance all act on what the model is given, so all three are silent together whenever the failure is that the model produced something it was never given — a number from a product description it read earlier in the session. Counting them as three independent defences overstates the protection badly, because a single design decision, the uniform wrapper that hands everything to the model, sets all three at once, and reversing that one decision unsets all three. Only the output check acts on what the model produced, which makes it independent of the other three in the way they are not independent of one another. That is the argument for keeping it in spite of being the least elegant piece: an independent check is worth more than a third correlated one.

When a cached price is acceptable#

The stakeholder asking for a cached fallback on pricing is not being careless; a stale price is often better than a dead conversation, and the conditions that make it acceptable are statable. The cache has to be populated from the authoritative server rather than from a parallel source, so it is a delayed copy of the truth and not a second opinion. Its age has to be bounded well inside the rate at which prices actually change: if the catalogue reprices weekly, a fifteen-minute TTL is generous, and if promotional pricing rotates hourly then no TTL is safe and the answer is no. The age has to be disclosed in the customer-facing answer, with the timestamp, because "€209 as of 14:05 today" is a different statement from "€209". The quote must not be binding — no checkout, no contract, no order confirmation reads from that path. And prices that are customer-specific, negotiated or regulated fall outside it entirely, since a cache keyed by SKU cannot represent them and a plausible generic price is exactly the failure this chapter exists to prevent.

Those conditions are conjunctive, and the way they are usually lost is by satisfying four of them and treating the fifth as a detail. Four are properties of your implementation — where the cache is filled from, how long a value may live, whether the age is shown, whether the quote binds — and each of them can simply be built. The fifth is a property of the data rather than of the code: a customer-specific, negotiated or regulated price cannot be represented by a cache keyed on a SKU, however carefully the cache is written, because the key does not carry the customer. That is the condition to check against the catalogue rather than against the design, and it is the one a stakeholder asking for a cached fallback will not have in mind, because from the outside every price looks like a number attached to a product.

Where those conditions hold, the cached path is a genuine improvement over a hard failure. Where any one of them fails, the hard failure is the correct product behaviour, and it is worth building the handoff well — a message that names the problem and offers a human — so that the correct behaviour is not also the worst experience in the system.

Exercising a path that has never run#

None of this makes the vendor available. Timeouts, policies and disclosures are about behaving correctly during an outage; the outage itself is the vendor's, and the approval conditions in chapter 4, on what has to be true before a hosted endpoint goes in a request path, are where you decide whether to accept it at all. A degraded path is also only real if it runs, so the reasonable practice is to exercise it deliberately — disable a server in staging on a schedule, or run a small share of production turns with a forced unavailability — because a fallback that has never executed is a hypothesis.

The honest limitation is that fabrication is reduced rather than eliminated. Removing the tool, withholding error text and constraining the output make the invented number much less likely and much more likely to be caught, and a sufficiently determined phrasing of a question can still produce a confident sentence with nothing behind it. The enforceable part is the output check, which is why it is worth having even though it is the least elegant piece: it does not care why the model produced the number.

The rule that survives all of it is that a model must never be the component deciding what unavailability means. It is handed either a result or a structured absence, chosen by code, per server, from consequences somebody wrote down — because the cost of being wrong about stock, about a price, and about an internal search result differs by orders of magnitude, and one retry wrapper cannot encode three different answers.

Across this topic the pieces now stand in a sequence you can run on a real integration this week. Take one server you already depend on, verify the three claims its registry listing does not settle, connect from the environment that will use it and commit the snapshot, replace whatever mutable reference is in your configuration with an immutable one and write down who reviews the next release, schedule the drift check, put the argument and result guards around the calls your client makes, and write the unavailability policy for that one server before you write it for all of them. The first server takes an afternoon. The sixth takes twenty minutes, because by then the snapshot directory, the policy table and the review cadence already exist and the work is filling in a row.