Staging a prompt rewrite the way you would stage a schema migration

Staged exposure is the practice of letting a change reach a growing fraction of traffic, with a defined stopping rule at each step, rather than switching all of it at once. It is standard for schema migrations and feature flags, and it arrives late to prompts because a prompt change looks like an edit rather than a deployment. The pointer machinery from earlier in this topic is what makes it affordable: a rollout that can only be undone by a release cannot be staged, since each stage would carry a fourteen-minute exit.

The measurements that gate the stages are guardrail metrics — the quantities that must not get worse, as distinct from the success metric the rewrite was written to improve. Naming them separately is the whole discipline, because a rewrite is tuned against one number and does its damage in the others: which of six tools gets picked when two are plausible, how arguments are filled when the user was vague, when the agent decides it cannot help and escalates, how often it refuses, how long it takes, and what it costs.

Four points on a frozen set is a statement about 340 examples#

An eval set is a sample of the paths a system takes, chosen by people who were thinking about the cases they knew to worry about. A rewrite changes behaviour across every path, and the categories where it does damage are systematically the ones the set under-represents. Those categories are all consequences of wording, and none of them is what the set was built to measure.

The measurement problem is worse than "we did not think of that case", because the success metric and the harm metrics have different statistics. Task completion sits around 71%, so a few hundred examples resolve a four-point move. An unauthorised refund sits at maybe 0.4% of conversations, and detecting a move from 0.4% to 0.8% at 80% power — an eight-in-ten chance of noticing the move if it is genuinely there — needs roughly 9,000 conversations in each arm of the comparison. Any rollout designed around the metric the prompt was tuned for will be sized for the wrong quantity, which is the specific reason "run it as an A/B for two weeks and ship whichever arm wins on task completion" fails: the arm that wins on the tuned metric is exactly the arm most likely to be losing quietly somewhere else, and the experiment is not powered to see it.

Why a rare metric costs so much more to measure is worth having in plain terms, because the stage sizes look arbitrary without it. Task completion is observed on every conversation: each one either completed or did not, so a few hundred of them already pin the rate tightly enough for a four-point move to show. An unauthorised refund is observed on almost none of them. At four in a thousand, a sample of a thousand conversations contains about four, and a count of events wobbles by roughly the square root of itself, so a handful of events is consistent with a wide spread of underlying rates. Doubling the rate doubles the expected count; telling the doubled count apart from the original needs enough conversations that the two expectations are far apart compared with that wobble, and for counts this small far apart means thousands rather than hundreds. Sparse metrics are not harder to reason about than dense ones. They are harder to see, by a factor that grows as the metric gets rarer, which is the reverse of the intuition that a rare problem is a small one.

Stages sized to what they can detect#

Staged exposure is not one idea repeated at different percentages. Each stage detects a different class of problem, and the stage durations follow from the arithmetic above rather than from a template. The volumes below assume a customer-facing agent handling 120,000 conversations a day.

Every volume in that table is an exposure multiplied by a duration, and doing one of them makes the rest read themselves. A hundred and twenty thousand conversations a day is five thousand an hour. One per cent of that is fifty an hour, and two hours of it is the hundred in the canary row. Ten per cent for a full day is a tenth of a hundred and twenty thousand, which is the twelve thousand; half for a full day is the sixty thousand; and shadow sees all of the traffic because nothing is being split, so four hours at five thousand an hour is the twenty thousand. Set those against the paragraph above and the stages sort themselves. A hundred conversations cannot see a change in a metric that fires four times in a thousand, and sixty thousand can, so each stage's duration follows from what it is able to detect rather than from how nervous anybody is.

Shadow mode needs a note on mechanics and on price. The candidate runs on a copy of the real input and its output is discarded, which means tool calls must not execute — the shadow run gets a stub tool layer that returns the same results the live run received, so a shadowed agent that decides to issue a refund records that decision without issuing anything. Running both prompts doubles input tokens for the shadowed slice, and at four hours of full traffic that is roughly 20,000 extra conversations at $0.011, about $220 for the stage. Cheaper than the alternative by two orders of magnitude, and worth budgeting deliberately rather than discovering on the invoice.

Traffic assignment has to be sticky at the level of a conversation or a user, not a request. Bucketing per request means a single conversation is governed by the candidate on turn one and the baseline on turn three, which produces incoherent transcripts and metrics that measure a mixture rather than either arm. Hashing the conversation id into a bucket is enough, and the bucket assignment belongs in the request record from earlier in this topic so that every measurement can be grouped by arm afterwards without reconstructing who saw what.

Those are two separate requirements and both are easy to skip. Hashing the conversation id, rather than drawing a random number per request, is what makes the assignment stable — the same id hashes to the same bucket every time, so every turn of a conversation lands in the same arm with nothing stored anywhere to remember it. Recording the arm on the request is what makes the measurement possible at all, because a metric computed a day later has no other way of knowing which prompt produced which outcome. Reconstructing it afterwards from the pointer history fails precisely in the periods that matter, which are the ones where two pointers were live at the same time.

The 1% canary — the first sliver of live, user-visible traffic the candidate is allowed to answer — is included knowing what it cannot do. A hundred conversations will not tell you whether unauthorised refunds doubled, and it will tell you within two hours if the rewrite broke JSON parsing for a downstream service, which is the failure that would otherwise reach every user at once. Skipping the shadow stage is the more common mistake: it is the only stage where the two prompts see identical inputs, which makes the comparison free of traffic-mix noise, and for an agent it also surfaces the tool-plan divergence — the fraction of conversations where the two prompts choose different first tools — which is the earliest available signal that behaviour moved.

The gates themselves are expressed as thresholds against the current baseline rather than against zero, and declared with the rollout rather than agreed in a meeting:

# rollouts/booking_agent_2024-11-25.yaml
prompt: booking_agent.system
candidate: pr_c81b3f04
baseline:  pr_9d41c60e
guardrails:
  unauthorised_write_rate:   { baseline: 0.004, max_relative: 1.10, blocking: true }
  tool_error_rate:           { baseline: 0.004, max_absolute: 0.010 }
  escalation_rate:           { baseline: 0.061, max_relative: 1.20 }
  refusal_rate:              { baseline: 0.011, max_relative: 1.50 }
  p95_latency_ms:            { baseline: 6200, max_absolute: 8000 }
  cost_per_conversation_usd: { baseline: 0.011, max_relative: 1.25 }
success:
  task_completion: { baseline: 0.711, min_relative: 1.02 }
abort:
  action: pin_baseline          # pointer move, no deploy
  automatic: true
  evaluation_window_min: 10

Only one guardrail is marked blocking, and choosing which one is a decision worth making explicitly rather than by declaring everything important. On an agent that can move money, the rate of write actions taken without the conditions that authorise them is the metric that blocks a launch, because it is the only one whose damage is not recoverable by turning the prompt off. Latency and cost are worth a conversation and a delay; an escalation rise is a support-staffing problem; a doubling of unauthorised refunds is money out of the door that no pointer flip brings back.

Aborting must not require a human to be awake#

The abort path is the pointer move from earlier in this topic, and the convergence numbers say the fleet honours it in about fifteen seconds with in-flight conversations finishing on the candidate.

Automating the abort matters more than automating the promotion. Promotions can wait for someone to look at a dashboard on Tuesday morning; a breach at 03:00 on a 50% ramp costs 30,000 conversations if it waits for the same person. The evaluation window in the config exists because guardrail metrics lag — a ten-minute rolling window means a genuine breach is confirmed ten minutes after it starts, and shortening the window to react faster trades against aborting on noise, which trains the team to disable the automation.

The thirty thousand comes straight out of the table above. Half of a hundred and twenty thousand a day is sixty thousand conversations, which is two and a half thousand an hour, so thirty thousand is the twelve hours between three in the morning and the middle of the next working day. Staging does not make a breach smaller. It makes it shorter, and the cost of one is a rate multiplied by a time, of which only the time is available to a design. Promotions have the opposite shape, because a promotion that waits costs nothing at all — which is why the automation budget belongs on the abort even though the promotion is the step people find themselves impatient with.

Not every guardrail can be measured at the speed the state machine wants to run. Tool-error rate, refusal rate, latency and cost are available within seconds of a conversation ending. Whether the customer's problem was actually solved usually is not: a complaint takes days, a refund reversal takes a billing cycle, and a satisfaction score arrives only from the small fraction of users who answer. Gating a 24-hour stage on a metric with a three-day lag stalls the rollout or, more often, gets quietly ignored. The workable arrangement is to gate the stages on the fast proxies and to keep the slow metrics as a post-launch review that can still trigger a pointer move a week later, which is the one advantage a prompt rollout has over a schema migration — the rollback stays available indefinitely.

The held state earns its place too. A metric drifting toward its threshold without crossing it is the common case, and a state machine with only "promote" and "abort" pushes an engineer into deciding under time pressure. Holding at the current exposure while the numbers accumulate is usually the correct move, and it needs to be a first-class option rather than an absence of action.

When the whole ceremony is waste#

A four-stage rollout with 52 hours of ramp is expensive in calendar time and in attention, and applying it to every prompt change is how the process gets abandoned. Three properties decide how much of it a change earns.

The first is what the agent can do. A prompt with no tool access, producing text a human reads before anything happens, has no irreversible outcome to protect; the worst case is a bad answer, which the abort path fixes for everyone in fifteen seconds. The second is exposure per unit time. An internal tool with 40 daily users generates about two conversations an hour, so a 1% canary is statistically meaningless and a 24-hour ramp accumulates fewer samples than the shadow stage would on the main service; the honest version of a rollout there is to ship it to everyone, watch it, and keep the pointer ready. The third is time to abort. If the abort is a pointer flip and traffic is low, total exposure to a bad revision is bounded by detection time, which makes a slow ramp a way of spending days to avoid a risk already bounded at minutes.

Where all three point the other way — a customer-facing agent with write tools, tens of thousands of conversations a day, and effects that land in a payments ledger — the ramp is not ceremony. The migration comparison is exact in an uncomfortable way at that point: like a backfill, some of what a prompt rewrite does cannot be rolled back. Pinning the baseline stops new unauthorised refunds and does not recall the ones already issued, and the conversations where the agent gave a customer a wrong commitment about a booking have already been read.

That irreversibility is the whole reason the stages are sized the way they are. Everything else in this topic reduces the time between noticing a problem and undoing it, and staging is the only thing that reduces the number of people who meet the problem before anyone notices. A shadow stage costs $220 and four hours; the refund episode above cost about a third of 5,600 daily refunds for as long as it took somebody to read a dashboard.

Everything in this chapter assumes there is a reviewed artefact with a revision id to stage, a baseline to pin, and a record of what the prompt said yesterday. Plenty of services arrive without any of that, with the production prompt living in a vendor console that shows only the current text, and the first job there is not a rollout plan.