Model deprecation and version pinning

Read these first

• Cost tracking per request, user, feature and tenant

What you will know

• Pin a model version explicitly rather than riding a floating alias

  Aliases can move without a code diff, making rollback and incident attribution ambiguous.

• Find the deprecation date for a model you depend on

  Provider notices, support windows, and actual shutdowns can differ, so record the date with an accountable owner and buffer.

• Re-qualify a new model version against an eval suite rather than by eyeballing

  A replacement can improve aggregate scores while breaking specific languages, tool calls, safety cases, or latency targets.

• Plan a migration before the shutoff date rather than during

  A workable plan needs parallel testing, a rollback path, traffic sequencing, and enough time to investigate regressions.

• Reproduce a floating-alias change in a staging harness and trace the resulting regression to the model change

  Controlling prompts, inputs, dependencies, and captures distinguishes provider drift from application, retrieval, or randomness changes.

Chapters

1. 1

   Family aliases and dated snapshots

   Two call sites can name the same model and send different requests: one carries a family alias the provider re-targets at request time, the other a dated snapshot id fixed in your own configuration until you change it. This chapter defines both forms, shows why the model field echoed back on the response is the only trustworthy record of which weights answered, and settles the rule that decides between them — float only where the served version is recorded per request, which admits a nightly eval arm and one named low-consequence surface and nothing else.

   Read chapter
2. 2

   The replay harness

   When quality drops over a weekend and the model version, the retrieval index and the traffic mix all moved, pinning back to last month's snapshot and watching the failure rate cannot tell them apart. A replay harness re-runs captured requests offline against two model versions with the prompt, the retrieval context and the sampling parameters held to what was actually sent, so the model is the only thing that varies. This chapter covers the capture record that makes it possible, the cohort split that separates a model change from an input shift, and how to reduce twelve thousand outputs to a number worth reading.

   Read chapter
3. 3

   Deprecation dates and internal deadlines

   A provider publishes an announcement, a deprecation status, an end of support and a retirement date, and only the last of those makes a request fail. This chapter separates the four events, says where each one is published and how far the date can be trusted, and works backwards from a 15 May 2027 shutoff through burn-in, cutover and one failed re-qualification attempt to an internal deadline of 15 January 2027, recorded beside the pin with a named owner, a source URL and a verification date that decays.

   Read chapter
4. 4

   Per-slice re-qualification

   A weighted mean lets 1,500 English items carrying 63% of the weight move an aggregate from 0.86 to 0.89 while German invoices fall nineteen points. Cut the run along the axes that break independently — language, document type, tool-call schema validity, refusals, latency percentiles, cost per request — compute each slice's margin of error before reading any delta, and write the pass criteria down before the run. Includes the case where price per token is unchanged and the bill rises 30% because the successor is more verbose.

   Read chapter
5. 5

   Sequencing a model migration

   Ninety days to a shutoff with a successor already available is a scheduling problem of a specific shape, and the ordering proposed first spends nine of those days and puts its only decision point at the moment of maximum ignorance. Four phases — qualify, shadow, cut over by cohort, burn in — each carry a gate that can say no, and two of them carry a return edge to the incumbent that expires on the shutoff date. Covers how much slack to keep, what rollback means once the old model is genuinely retired, and the seven options available when the successor fails re-qualification.

   Read chapter
6. 6

   Forcing functions for model pins

   One config file, a required deprecation-date field and a CI failure 60 days out is the right instinct aimed at the wrong person: it blocks every unrelated change from whoever happens to push that morning, and the cheapest way to satisfy it is to edit the date. A forcing function that is cheapest to satisfy by falsifying its own input has been designed to be defeated. Covers what to enforce instead — registration at the moment a pin is written, a reconciler that reads the provider rather than your repository, and a blocking gate kept for the terminal case — so that four pinned versions across three providers stay owned rather than merely listed.

   Read chapter