Sequencing a model migration

Sequencing a model migration means choosing the order in which four things happen — qualification, shadowing, cutover and burn-in — and how much calendar sits between them, given a date on which the version you run today stops being served. That date is the shutoff, or retirement: the day the provider stops resolving a model id, after which every request naming it fails outright. A successor is usually available months ahead of it, which makes the whole thing look like a scheduling problem with a great deal of slack. The slack is not spare time; it is the width of the window in which a rollback still has somewhere to go.

Three words in that list are worth fixing before they are used. Shadowing runs a candidate model over real traffic without returning its output to anyone. A cohort is a named group of tenants that moves onto the candidate together, so that a cutover happens four or five times rather than once. Burn-in is a stretch of ordinary running on the candidate with the incumbent still deployable.

Four phases, each of which can say no#

Read the diagram for its edges rather than its boxes. Every forward transition has a gate, and two of them have a return edge back to the incumbent. Those return edges are the entire design, and they are available only while the incumbent still resolves.

That availability is not a property of your deployment pipeline, which is the assumption worth dislodging before the rest of the plan is built on it. Reverting a pin is a one-line config change and a deploy, all of it inside systems you own, and the speed of that is what makes rollback feel like something you control. What the change actually does is put a different string in the model field, and a string in that field is a name the provider has to still be answering. If the id has been retired, the deploy succeeds, the config is correct, and every request it produces comes back as an error. The return edges on the diagram are therefore drawn by somebody else's calendar rather than by your tooling, and no amount of deployment engineering buys them back once the date has passed. Finishing the cutover with weeks in hand is how you stay inside a window you have no way to extend.

Qualify is chapter 4's suite — the eval run cut along the slices you care about and scored against a gate file. It costs a few hundred dollars and a week, and it happens now rather than in October because a failure here changes the plan rather than the schedule. Everything downstream assumes this phase produced a yes, or produced a named, accepted no.

Shadow runs both models over a sample of live traffic and compares the outputs offline. The candidate's output is never returned to a user; it is scored, diffed against the incumbent's, and written to a table. This is where the eval suite's blind spots surface, because the traffic is real and the distribution is not one you curated. Five per cent of 380,000 daily requests is 19,000 candidate calls a day, about $180 a day at the candidate's cost per request, so ten days of shadowing costs under two thousand dollars — a number worth computing in advance and watching on the per-feature cost tracking you already have, because a shadow arm someone forgets to turn off is a line item that survives for months.

What makes that comparison available at all is a property of the request rather than anything you built. A call to a hosted model carries everything it depends on in its own bytes — the prompt, the parameters, the model id — and leaves nothing behind on the provider's side for the next call to read. Sending the same input to a second model therefore cannot disturb the first, because there is nothing shared between them to disturb, and the user's response has already been computed by a call that had no way of knowing the second one was coming. Two conclusions are commonly read out of that property and neither follows from it. Tokens are billed on both arms and both arms draw on the same rate-limit budget, so a shadow arm is off the user's path without being off your invoice. And it is off the request path only for as long as you keep it there.

One implementation detail decides whether shadowing is safe. The candidate call must not be in the request path. Fire it onto a queue, or dispatch it after the response has been returned, and give it its own timeout and its own error budget; if you await it inline, you have coupled your production p99 to an unqualified model and the migration's first incident is caused by the safety mechanism.

async def summarise(req):
    result = await call_model(MODELS["summariser"]["incumbent"], req)   # serves the user
    if shadow_enabled(req.tenant_id) and sample(0.05):
        shadow_queue.put_nowait(                                       # never awaited here
            {"request_id": req.id, "tenant_id": req.tenant_id,
             "input_uri": persist(req), "incumbent_output": result.text,
             "candidate": MODELS["summariser"]["candidate"]}
        )
    return result

The line that holds that separation is put_nowait. It writes the job onto the queue and returns without waiting for a consumer to be ready, which is the property that keeps the candidate's timing out of the caller's timing: the function reaches return result after doing an amount of work that does not depend on the candidate at all. The comparison still happens, later, in whatever reads that queue. What the snippet is really demonstrating generalises past shadowing, and is worth carrying to the next mechanism of this shape — an arrangement that protects one path by watching it is only a protection while the two do not share a resource, and the resource in question here is the time the caller spends blocked.

Cutover moves tenants onto the candidate in cohorts, ordered by blast radius ascending — how much damage a bad answer does to that group, smallest first — rather than by size. Internal workspaces first, then free-tier accounts, then paying accounts with low-consequence usage, then everyone else, with the highest-consequence account last.

That ordering has a reason behind it worth being able to reconstruct. Every cohort you complete converts something you were guessing at into something you have observed, and the observations accumulate — nothing learned from the internal workspaces stops being true when the free tier lands. The cost of being wrong moves in the other direction, rising with each cohort's consequence. Ordering by consequence ascending is the only sequence in which the amount you know is largest at the moment the stakes are highest, and ordering by size, or by whichever tenant is easiest to schedule, gives that alignment up for nothing. Where the rule stops holding is representativeness: a cohort whose traffic looks nothing like the one behind it teaches you very little about it, so blast radius decides the order while the resemblance between cohorts decides what a clean result is worth.

# config/models.yaml
summariser:
  incumbent: claude-sonnet-4-5-20250929
  candidate: claude-sonnet-5
  default: incumbent
  cohorts:
    - name: internal
      selector: {org_type: internal}
      use: candidate
      since: 2026-08-24
    - name: free_tier
      selector: {plan: free}
      use: candidate
      since: 2026-09-07
    - name: standard
      selector: {plan: standard}
      use: incumbent          # flips 2026-09-21
    - name: enterprise_de
      selector: {tenant_id: [t_4410]}
      use: incumbent          # flips last, after burn-in on standard

Burn-in is the phase teams skip, and skipping it is what turns a migration into a rolling incident. Two weeks on the candidate with the incumbent still deployable, watching the per-slice production metrics rather than the aggregate, is what catches the regression that only appears in end-of-month batch traffic or in the retry path.

Burn-in reads as padding, and that reading is why it gets cut. The belief underneath it is that once the last cohort has flipped and nothing has broken, the migration is finished — which would hold if the traffic seen since that flip were a fair sample of the traffic the feature gets. It is not. A month-end batch runs once a month, a quarterly report once a quarter, the retry path only when something upstream fails, and a rare document layout only when the customer who sends it happens to send one. Waiting is not what burn-in costs you; the waiting is the mechanism, because the only way to observe the slow parts of an input distribution is to still be watching when they arrive.

Each gate needs a number and a person#

A phase diagram with unspecified gates degrades into a schedule, and a schedule under deadline pressure advances whether or not the previous phase went well. Each transition therefore needs its criteria written down before the phase starts, and a named person who reads them and says yes.

Shadow is the gate people define worst, because it produces a mountain of comparison data and no obvious threshold. The criteria that work are the ones already established for the eval suite, recomputed on shadow traffic: per-slice output quality where the task is scorable without a reference, field-level agreement where it is structured, plus the operational metrics the offline suite cannot produce — candidate error rate, candidate p99, and cost per request measured rather than estimated. The additional thing shadow gives you, which the eval suite cannot, is a sample of real disagreements to read. Fifty of them, read by a person who knows the product, is worth more than any aggregate on this phase, because the failure you are looking for is the one nobody thought to write a metric for.

Cutover gates are simpler and should be boring: seventy-two hours on the cohort with per-slice production metrics within the thresholds from the gate file, no new error classes in the logs, and no support tickets attributable to the change. The last one needs support to know the migration is happening and to have a tag for it, which is a five-minute conversation that teams routinely skip and then regret when a ticket about "weird summaries" sits in a general queue for a week.

The person who says yes matters as much as the numbers. On a cohort-by-cohort migration the decision is made four or five times over five weeks, and it degrades into a formality by the third one unless it is somebody's explicit job. Make it the pin's owner from chapter 3 (the named individual recorded in the model registry beside the retirement date), give them the per-slice dashboard rather than the aggregate, and record the decision — a two-line note per cohort, with the numbers as they stood — so that a problem found in cohort four can be checked against what cohort three actually looked like.

There is one more gate that is easy to leave out, which is the decision to stop shadowing. A shadow arm left running after cutover is paying for a comparison against a model that no longer serves anyone, and it will keep doing so until an unrelated cost review finds it. Put its removal in the plan as a task rather than as an intention.

Rollback means something different after the shutoff#

The word "rollback" is doing two different jobs in this plan, and the difference is a date.

Once the shutoff passes, the incumbent returns a 404 and the second and third rows stop existing. Whatever problem you find the following week has to be solved forward — by fixing the prompt, by moving to a different model you have not qualified, or by turning the feature off — and all three of those are slow under pressure. That is the whole argument for finishing the cutover with weeks in hand instead of days: the value of the slack is not that the work might overrun, it is that a working rollback path has an expiry date and you want to still be inside it when the surprise arrives.

When the successor fails re-qualification#

Chapter 4's German-invoice slice — the nineteen-point collapse that a three-point aggregate gain concealed — is the case where the plan's first phase says no and the shutoff does not care. The mistake here is treating the shutoff as forcing the migration to that particular successor, when what it forces is a migration. There are more options than the conversation usually admits, and they have very different costs:

Ordering matters as much as the contents. The top three are cheap enough to run in parallel with the clock still ticking, and prompt rework in particular deserves a serious attempt before anything structural, because generation bumps frequently move behaviour that prompting can move back — instructions written to compensate for an older model's habits often over-apply on a newer one that follows instructions more literally, and deleting them is as likely to help as adding more. The bottom four all consume the slack you built, which is the argument for starting the qualification in week one rather than week eight: the option set is wide in August and narrow in October.

What narrows the set, rather than merely shortening the list, is visible in how the rows are sorted. Each one keeps a different amount of the system you already have. Prompt and scaffold rework keeps all of it and changes the text of an instruction. A larger sibling keeps the client library, the request shape and the lifecycle calendar, and changes one string. A different provider keeps the idea of the feature and replaces the SDK, the failure modes, the pricing and the calendar underneath it. Self-hosting replaces the party serving the model at all. How much a row keeps is what sets both of its cost columns at once, because the time to know whether it works is the time to re-establish whatever it did not keep, and the permanent cost is whatever it added that somebody now maintains forever. Read that way the table is reconstructible rather than memorable, which is what you need when the option that fits is one it does not list.

The escalation is worth naming explicitly too. When qualification fails and the remaining options all cost more than a sprint, the decision stops being the on-call engineer's. Somebody has to choose between a quarter's roadmap and a feature, and that choice needs the person who owns the roadmap, informed early enough that they have options.

A cutover is not a blue/green deployment#

The cohort mechanics look like a blue/green deployment — two complete versions of a service running side by side while traffic is switched from the old one to the new — and the resemblance misleads. Blue/green assumes the two sides are functionally equivalent and the only question is infrastructure. Here the two sides produce different text on the same input by design, so "both are up and healthy" is not the completion criterion; per-slice output comparison is.

State the incumbent already produced#

A cutover changes the next request and nothing already stored. Summaries you cached, classifications you materialised and embeddings you indexed were all produced by the incumbent, and none of them is touched by flipping a cohort. Mixed-vintage artefacts are usually tolerable for generated text. They are actively broken for embeddings — the numeric vectors a model produces to stand in for a piece of text — because vectors from two models occupy incompatible spaces and similarity search degrades silently rather than failing.

The silence is the part that needs a mechanism, since a failure that raised an error would be a nuisance rather than a hazard. An embedding is a list of numbers, and no position in that list carries a meaning fixed in advance — what the number at a given position stands for is settled during the model's own training, by that training run and nothing else. Two models therefore produce lists whose positions were assigned independently, and a similarity computation compares two lists position against position. That arithmetic is perfectly well defined on inputs which agree about nothing: it reads two lists of the right length and returns a number in the usual range. Nothing throws, no query fails, and the store goes on returning results that look like results. A mixed-vintage index is worse than either vintage alone for exactly that reason — every comparison crossing the two is arithmetic on incomparable axes, and in the output it is indistinguishable from one that is not.

Re-embedding a corpus of stored documents is therefore its own migration, with its own backfill plan, its own cost model and its own cutover, and it wants to be discovered in August alongside everything else rather than in November.

The plan also assumes someone runs it. Ninety days with four phases and a named owner works when there is one migration in flight and it is the thing the owner is doing this quarter. A service with four pinned versions across three providers has four of these on overlapping clocks, most of them dormant most of the time, all of them capable of becoming urgent while everyone is busy with something else.

Keeping four dates owned rather than merely written down is a different problem from running one migration well, and the mechanisms teams reach for first — a required field, a failing build, a threshold in CI — tend to be aimed at the wrong person entirely.