Data drift, prompt rot, and a new model need different evidence

Three incidents land in the same fortnight and none of them was preceded by a deploy.

Invoice extraction, which has held an F1 of 0.94 on production-sampled cases since February — F1 being the combined precision-and-recall score, a single number that falls when the extractor misses fields and also when it invents them — comes in at 0.81 on Monday. A summariser used by the customer success team starts producing answers the team describes as "longer and more careful", and the change is sharp enough that people can name the weekend it happened. A support-ticket classifier that routes into five product tiers begins dropping tickets into the catch-all bucket at three times its usual rate, and the affected tickets are not obviously unusual.

The reflex is to reach for input statistics in all three cases, because "the data changed" is the explanation that has been available to anyone monitoring a machine learning system since long before any of this involved a vendor's API. That reflex is right about one of the three. In the second, nothing about the inputs moved and the model behind a stable alias was replaced. In the third, the inputs are exactly what they have always been and the prompt has aged out of agreement with the product it describes. Investigating all three as data drift means two investigations that find nothing, reported as "no anomaly detected", after which everyone waits for it to happen again.

The taxonomy is older than the API you are calling#

Monitoring literature for classical models separates covariate shift, where the distribution of inputs moves while the relationship between input and correct answer holds, from concept drift, where the correct answer for a given input changes — fraud patterns adapting to your detector, a credit model meeting a new interest-rate regime. Both assume the thing in the middle is yours and therefore fixed: you trained those weights, you have the artefact, and it does not change unless you retrain it.

Calling an alias on a vendor's endpoint removes that assumption, and the taxonomy needs a third entry for it. The function itself can be replaced while its name, its interface, its latency profile, and its price stay identical. Prompt rot is a fourth entry, and it is the one with the most engineering character: your prompt is a frozen encoding of product facts — tier names, policy thresholds, tool descriptions, the taxonomy you route into — sitting in a repository while the product it describes carries on changing. Nothing degrades. The world moves and the string does not.

Replay in a fixed order and take the residual#

The separation is a replay with a specific ordering, and the ordering matters because each step removes a candidate before the next step's evidence would be ambiguous.

The frozen replay goes first because it is the only step that isolates the model cleanly. Inputs are fixed by construction, the prompt is whatever version you pin for the replay, and the sole difference between arms is the identifier you sent. For the summariser, that arm split immediately: median output 186 tokens on the pinned snapshot against 254 on the alias, refusals up from one case in 240 to nine, exactly the pattern from the first chapter. No amount of staring at input statistics would have produced that, because the inputs were a file on disk that had not been touched since March.

When the arms agree, the model is exonerated for this window and the question becomes which of your inputs are no longer like the ones you calibrated on. Slice statistics are cheap and the useful ones are structural rather than semantic.

# ops/slices.py — compare last week's production inputs against the baseline window
DIMS = ["customer_id", "locale", "producer", "page_count_bucket", "has_text_layer",
        "input_token_bucket", "attachment_mime"]

def slice_report(baseline, current, dims=DIMS):
    for d in dims:
        b = share_by(baseline, d)          # {value: fraction of requests}
        c = share_by(current, d)
        moved = {k: (b.get(k, 0), c.get(k, 0)) for k in set(b) | set(c)
                 if abs(c.get(k, 0) - b.get(k, 0)) > 0.03}
        if moved:
            yield d, moved

On the invoice incident that report answered the question in a single line. Customer 8812, 22% of the month's volume, had gone from producer=QuickBooks to producer=NetSuite, and with it has_text_layer fell from 1.00 to 0.12 and the median page count went from 1 to 3. Their new billing system renders invoices as flattened images, so the extractor was reading an OCR pass — optical character recognition, a guess at the characters in a picture of a page — rather than an embedded text layer that carries the characters exactly. Scored per customer, the aggregate F1 of 0.81 decomposed into 0.93 across everyone else and 0.42 on 8812 — an average that described no customer's experience. That is the signature of data drift generally: the drop is concentrated in a slice, and it appeared the day that slice's composition changed.

The classifier survived both steps. Its frozen set scored identically on the pinned snapshot and the alias, and its input slices were flat on every dimension: same customers, same languages, same lengths, same ticket sources. What had changed was outside the system entirely. Marketing had renamed two tiers on the 14th, Growth to Team and Scale to Business, and the prompt still enumerated the old five. Tickets containing the new names matched nothing in the enumeration and fell through to Other, which is why the affected tickets looked unremarkable: they were, apart from a word the prompt had never heard of. Sliced by the entity that changed, 31% of tickets mentioning "Team" or "Business" landed in the catch-all against a base rate of 4%.

Prompt rot as the residual is a sound inference and a weak one, and it is worth knowing which. It is sound because the two preceding steps are genuinely discriminating, and weak because it relies on there being exactly one cause. The confirming move takes fifteen minutes: find the prompt's last commit date, list the product changes since — tier renames, new tools, changed thresholds, deprecated flows — and check whether failures cluster on the entities involved. A prompt whose last edit predates a rename, with failures concentrated on the renamed things, is not a residual any more.

The whole ordering is cheap enough that there is no good reason to skip steps. Two arms of 240 cases is about 20 minutes of wall time at reasonable concurrency and roughly £38 of inference; the slice report is a query over data you already have and returns in seconds; the prompt diff is a git log and a conversation with whoever owns the product surface. The expensive part of a drift investigation is never the compute, it is the two days that pass while three people argue from intuition about which explanation is most likely, and the ordering exists to replace that argument with £38 of evidence.

Each cause has a different repair#

Attribution matters because the remedies share nothing. For the invoice case the fix was in the ingestion path rather than anywhere near the model: detect the missing text layer, route those documents through a higher-quality OCR pass, and — critically — add fifteen of 8812's new invoices to the evaluation set, because the set's composition had lagged the customer base and would have lagged it again on the next such change. Data drift repairs tend to look like this: a preprocessing branch, plus a correction to what you evaluate on.

The summariser's repair was configuration and a vendor ticket. Pinning back to the earlier snapshot restored the previous behaviour within a deploy, and the paired outputs went to the provider as a report. Nothing in the prompt or the pipeline was wrong, so changing either of them would have been the expensive kind of mistake — a team that concludes "the model got hedgier, we should add an instruction telling it not to hedge" ends up with a prompt carrying a workaround for a change that may be reverted next month, and no record of why the instruction is there.

The classifier's repair was structural. Re-editing the prompt with the new tier names would have worked until the next rename, so the enumeration was generated at build time from the product catalogue that marketing already maintains, with a test asserting that the prompt's tier list matches that source. Prompt rot is usually a duplication problem in disguise: a fact that lives in two places will diverge, and the durable fix is to make the prompt derive from the source of truth rather than restate it.

Where the nightly set is structurally blind#

A fixed evaluation set cannot detect data drift, and this is not a matter of the set being too small or too old. Its inputs are frozen, which is the property that makes it a measuring instrument, and a change in the population of inputs arriving from real users is by definition invisible to a set that never sees them. The invoice incident would have shown 0.94 on every nightly run for as long as it lasted, correctly, while 22% of volume was being extracted at 0.42. Detecting that requires scoring production-sampled cases or watching input slices; those are different mechanisms with different costs, and the pairing is deliberate rather than redundant.

The blindness to prompt rot is subtler and more embarrassing when it happens. A fixed set catches rot only if its expected outputs reflect the current product, and golden cases are usually written by the same person, in the same week, from the same understanding as the prompt. When the tiers were renamed, the classifier's evaluation set still expected Growth and Scale, so the stale prompt scored 0.96 against stale labels while production misrouted a third of the affected tickets. The set and the prompt aged together and agreed with each other all the way down.

Concept drift is the case where the set becomes actively wrong rather than merely blind, because the labels themselves no longer encode the right answer. It gets folded into "data drift" in casual usage and it needs a different remedy: re-adjudicating a sample of cases with a human, on a cadence, and treating disagreement between fresh and archived labels as the signal. If your evaluation numbers ever look suspiciously stable through a period of real product change, that is worth checking before anything else.

There is a fifth cause that the taxonomy does not name because it is not drift at all, and it accounts for a surprising share of no-deploy incidents: your own infrastructure changed without a deploy to the service under investigation. A retrieval index rebuilt with a new embedding model, a chunking parameter altered in a different repository, a feature flag flipped by another team, a rate limiter that started shedding tool calls, a document store that silently stopped returning a category. These produce the same shape as data drift — production scores fall, the frozen replay holds — and the slice report will not find them, because the inputs really are unchanged. Keeping index versions and flag states in the request log, as the compliance-style records later in this topic do, is what makes them findable, and the cheapest habit is to check the change history of every system in the request path rather than only the service that owns the endpoint.

None of the four is exclusive, and the fortnight described here is unusually clean in giving one cause per incident. A provider update landing in the same week a large customer changes their document pipeline produces a frozen replay that splits and slices that moved, and the honest reading is that both are true and you now have two pieces of work. The ordering still helps, because you attribute what the replay explains and treat the remainder as the second problem.

Every one of these investigations began with a number that moved and a decision that it was worth chasing. The invoice case started at 0.94 falling to 0.81 on several hundred sampled cases, which is not a close call. The next chapter is about the ones that are, because a great many alerts fire on segments where a six-point move is what a quiet Tuesday looks like.