The prompt store you just pinned is now a serving dependency

A control plane at 99.9% monthly availability is unreachable for about 43 minutes a month by definition, and a service that resolves its prompt from one at request time has just adopted that number. The question a design has to answer is not how to avoid it but what serves during it, and the answers available range from "the revision that was serving a minute ago" to "an empty string", with nothing in the API to indicate which one you have chosen.

That first figure is a definition rather than a measurement, and doing the arithmetic is what makes it usable in an argument. A thirty-day month holds roughly forty-three thousand minutes. A tenth of one per cent of that is about forty-three of them, so a provider publishing three nines is publishing permission to be unreachable for that long every month while owing nobody anything. It is not a forecast of a bad month; it is the floor the contract guarantees. And availability multiplies down a serving path, because a request that must reach two systems is answered only when both are up, so a service resolving its prompt from that control plane at request time cannot be more available than the control plane is. Adopting the dependency is adopting the floor.

The two numbers that decide the shape of the answer are the fetch's latency and its failure mode. A cold fetch from a hosted prompt platform measures around 38 ms at the median and 210 ms at the ninety-ninth percentile — the slowest call in a hundred — so putting it inline adds 210 ms to the tail of every slow request for information that changes perhaps twice a week. And an SDK — the vendor's client library your service imports — typically returns an empty string when a fetch fails, which the application concatenates into the message list without complaint. This chapter is about paying that bill deliberately rather than discovering the terms during an outage.

The fetch must not be on the request's critical path#

The resolution path that survives contact with production has three layers, and the request only ever touches the first:

class PromptResolver:
    def __init__(self, bundled: dict[str, Prompt], ttl_s: int = 60):
        self._cache: dict[str, CacheEntry] = {}
        self._bundled = bundled          # baked into the image at build time
        self._ttl_s = ttl_s

    def get(self, name: str, env: str) -> Prompt:
        entry = self._cache.get((name, env))
        if entry is not None:
            return entry.prompt          # serve even if past TTL; refresher owns freshness
        return self._bundled[name]       # cold start, control plane unreachable

    async def refresh_loop(self) -> None:
        while True:
            for (name, env) in self._watched:
                try:
                    revision = await control_plane.pointer(name, env)
                    body = await revision_store.get(name, revision)   # immutable, cacheable
                    self._cache[(name, env)] = CacheEntry(
                        prompt=Prompt(revision=revision, body=body),
                        fetched_at=now(),
                    )
                    metrics.gauge("prompt.cache.age_s", 0, tags={"prompt": name})
                except Exception as exc:
                    metrics.increment("prompt.refresh.failure", tags={"prompt": name})
                    log.warning("prompt refresh failed, serving cached", error=str(exc))
            await asyncio.sleep(self._ttl_s / 4)

The refresher runs on a timer and owns all the failure handling, so a 503 from the control plane changes nothing about what serves — the cache keeps its last good entry indefinitely, deliberately, because a stale approved prompt is a working service and a missing prompt is not. The _bundled dictionary covers the one case the cache cannot: a process that starts while the control plane is unreachable, which is exactly what happens when an outage coincides with an autoscaling event or a rolling restart.

Why the fetch cannot sit on the request path is a comparison of two rates, and it is worth doing rather than asserting. The pointer changes perhaps twice a week. Requests arrive tens of times a second. Fetching per request therefore pays the fetch's cost — 38 ms in the ordinary case, 210 ms on the slowest call in a hundred — hundreds of thousands of times over, to learn something that was already true the previous hundred thousand times it was asked. A cache with a background refresher pays that cost a few times a minute and arrives at the same answer. What is given up is freshness bounded by the refresh interval, which is seconds. What is bought is latency on every request, and it is bought at the tail, which is where slowness is visible to a person.

Bundling means the build embeds a specific revision, recorded in the image's manifest:

{
  "image": "orders-agent:v2024.11.7",
  "bundled_prompts": {
    "order_agent.system": "pr_9d41c60e",
    "order_agent.summariser": "pr_31ba7702"
  },
  "built_at": "2024-11-12T08:02:10Z"
}

The fallback is a named revision with the same provenance as any other, not "whatever is in the repo at build time" resolved lazily and not an empty default. That distinction is what makes the degraded mode explainable: during an outage the service serves pr_9d41c60e, which was reviewed, approved and previously live, and the logs say so.

The TTL — time to live, the age at which a cached copy is treated as due for a refresh — is the one number in that resolver worth arguing about, and the argument has two sides that pull in opposite directions. A short TTL makes pointer changes propagate quickly, which is what the rollback story depends on, and it multiplies load on the control plane: twelve instances refreshing every fifteen seconds is 48 requests a minute, which is nothing, while the same code at 200 instances is 800 requests a minute against a vendor endpoint that may well be rate-limited and is certainly billed. Somewhere past a hundred instances the polling model stops being appropriate and the sensible arrangements are a long TTL with an explicit invalidation channel — a pub/sub message or a streaming watch that tells instances to refresh now — or a sidecar per node that polls once and serves the rest of the processes locally. The number to keep constant through any of those redesigns is the worst-case propagation time, because that is what appears in your incident timelines.

The multiplication behind those two figures is worth a line, because it decides which redesign a team needs. A refresher waking every fifteen seconds wakes four times a minute, so the load a single instance puts on the control plane is four requests a minute and nothing else — twelve instances give the forty-eight, two hundred give the eight hundred. Load is linear in the instance count and inversely proportional to the interval, which means the two levers pull against each other: halving the interval halves the propagation time and doubles the load. The asymmetry is what settles it. Adding instances raises the load and improves propagation not at all, and services are scaled out for reasons that have nothing to do with prompts, so a design whose control-plane load grows with the fleet has a bill arriving on somebody else's schedule.

Failing to an empty prompt fails in the most expensive direction. An agent with no system prompt does not stop working; it works enthusiastically, without the constraints that were the whole point of the prompt. If a bundled fallback is genuinely unavailable — a prompt added after the current image was built, for instance — refusing the request is the correct behaviour, because a 503 from your service is a problem you will notice within minutes and an unconstrained agent is one you notice in a review weeks later.

The belief that makes an empty default survivable is that a missing prompt would be obvious, and it is worth taking apart, because every property of this failure runs the other way. A system prompt supplies no capability. It removes options — which persona, which tools, which subjects to decline, which shape the answer takes. Take it away and none of the model's ability leaves with it; what leaves is every restriction, so the service answers more questions than before, more readily, in a register nobody chose. No exception is raised, no response is empty, and no error rate moves, because from the runtime's point of view nothing failed. The only signal is the content of answers that nobody is reading while they are being written, which is why the gap between the start of an outage like this and its discovery is measured in the time it takes a person to look.

The fifth row is the one that surprises people. A bad pin does not fail loudly, because running instances keep serving from cache and only newly started instances fall back, so the fleet splits into two behaviours along an axis nobody is watching.

That axis is instance age, and no dashboard groups by it because nothing else has ever needed it to. Every long-lived instance holds a cache entry from before the bad pin and keeps serving it, correctly, since the resolver serves past its TTL by design rather than emptying itself when a refresh fails. Every instance started after the pin has no entry and falls through to the bundled revision. Both behaviours are the design working exactly as specified. What creates the split is that the two populations are answering with different text at the same moment, and which population a given customer reaches is decided by a load balancer. Error rate stays flat, latency stays flat, and the only way to see any of it is a request-log field naming which revision answered — the field the last section of this chapter argues for on entirely separate grounds.

Validating the revision exists at pin time — a single existence check inside promptctl pin before the pointer write — removes the class entirely, and it is cheap enough that there is no reason to skip it.

A pointer flip is not instantaneous, and you should know the number#

The rollback in the previous chapter reported "12/12 instances within 30s". That number is a consequence of the refresh interval, and it is worth being able to derive it rather than trusting the tool's output. With a 60-second TTL and a refresher waking every 15 seconds, an instance picks up a pointer change somewhere between 0 and 15 seconds after the write, plus the fetch itself. Across twelve instances with independent timers, the last one converges at about 15 seconds in the common case and later if a fetch fails and waits for the next cycle.

Two consequences follow. During the convergence window the fleet is genuinely serving two different prompts, so a dashboard split by revision id will show both, and an engineer watching an error rate recover will see it fall over fifteen seconds rather than dropping cleanly. More importantly, a request that has already started does not change prompts mid-flight, and for a multi-turn agent loop that can run 30 to 60 seconds across several model calls, that is a decision you have to make rather than inherit. Resolving the revision once at the start of the loop and threading it through every turn keeps a single conversation coherent; resolving per model call means turn 1 can be governed by one set of tool instructions and turn 4 by another, which produces transcripts that are very hard to read.

Whichever you choose, the request log records the revision the request actually used, taken from the resolver's return value rather than from the pointer. Those two disagree for fifteen seconds after every flip and for the entire duration of a control-plane outage, and the log is worthless in exactly the cases you care about if it reports the intention instead of the fact.

What stays in the repository regardless#

The bundled fallback only exists because the prompt source is still in the repository, and that is the general answer to what the platform does and does not take over. The repository keeps the authoritative text and its history, so the diff, the blame and the approval record from the first chapter of this topic remain intact. It keeps the template code and the escaping rules from topic 4.9, on rendering user input into a prompt safely, because those are executable behaviour and belong with the code that runs them. It keeps the eval fixtures and the CI job that runs them, so a prompt change is still gated before it can become a publishable revision. And it keeps the build-time export that turns the current approved revisions into the bundled dictionary above.

The control plane owns exactly one thing: which of the published revisions is currently designated for each environment and slice. That is a small surface, and keeping it small is what makes the degraded mode benign — losing the control plane costs you the ability to change which prompt is live, and costs you nothing about the ability to serve. Teams that adopt a platform without keeping this split end up with the prompt existing only in the vendor's database, which is the arrangement where a vendor outage is a total outage, where a vendor's data-retention policy is your change history, and where the answer to "what did this prompt say in March" depends on a support ticket.

The fallback will lie to you eventually#

The honest limitation in this design is drift between the bundled revision and reality. An image built on 12 November bundles pr_9d41c60e; if the pointer has since moved forward through four approved revisions and the service has not been rebuilt in six weeks, then a cold start during a control-plane outage serves a prompt that is six weeks and four decisions old. Nothing is broken and nobody is alerted, because the fallback is doing exactly what it was designed to do, and the behaviour of newly started instances is quietly different from the rest of the fleet.

A second, quieter cost comes with putting the prompt in a third-party store. System prompts routinely encode business rules that nobody thinks of as sensitive until they are outside the perimeter: escalation thresholds, the conditions under which a discount may be offered, the phrasing legal insisted on, the internal names of tools and the systems behind them. Once the text lives in a vendor's database it inherits that vendor's access model, its retention policy and its breach surface, and the people who can read it are whoever has a seat on the platform rather than whoever has a checkout of the repository. Keeping the repository authoritative for the text means the review that decides who may read a prompt is the same review that decides who may read the code, which is one fewer access model to reason about.

The mitigations for drift are all unglamorous: emit the age of the bundled revision as a metric and alert when it exceeds the deploy cadence, rebuild on pointer changes that stick for more than a day, and treat a rising prompt.source=bundled count as an incident signal rather than a debug line.

What none of them do is remove the ambiguity when a request from that period turns up in a complaint six weeks later. The pointer says which revision was designated; the bundled manifest says which one a freshly started pod would have used; the two disagree, and neither of them is a record of what this particular request was actually given. The only thing that answers that question is a record attached to the request itself, which needs to carry more than a revision id, and which is where the next chapter starts.