A point release can change refusals and tool calls with flat benchmarks

The change was one line in a config file: the pinned model identifier moved from the February checkpoint of a family to the May checkpoint of the same family. The vendor's release note described it as an incremental improvement, listed benchmark results that were within a point of the previous version on every published suite, and said nothing about behaviour changes. Three engineers spent an afternoon trying it — twenty prompts between them, drawn from tickets they remembered — and everything looked fine. It shipped on a Thursday with no eval run, because the eval suite took forty minutes and this was a version bump within a family, not a migration.

The finance dashboard caught it eleven days later. Model spend for the research agent was up 46% on flat request volume. Pulling the traces showed the cause immediately: mean tool calls per run had gone from 3.4 to 5.1, because the new checkpoint was more willing to look something up a second time to confirm it. Nothing was broken. The p95 end-to-end latency, the figure that 95 out of every 100 requests come in under, had gone from 8.2 seconds to 13.6 seconds, and the support team had logged four complaints about the research pane being slow, none of which had been connected to the model bump.

That was the visible one. Three other things had also changed.

Four kinds of drift, none of them capability#

Pulling two weeks of production metrics from either side of the deploy, with request mix held roughly constant, gave the following.

The first two rows are the reason nobody looked. The next two are the reason the afternoon of manual testing came back clean: on the tasks the engineers actually tried, the new checkpoint was slightly better, and slightly better is what everyone expected to find, so they stopped.

The bolded rows are policy and format rather than capability, and they are what post-training moves between checkpoints of the same family. Post-training is everything a vendor does to a model after the expensive pre-training run that taught it language: the instruction tuning, the preference training against human and model judgements, and the safety policy work, all of which are revised on a much faster cycle than the base model underneath them. Refusal behaviour is a product decision encoded in preference data and a system-level policy, both of which get revised release to release. Tool-call propensity is a consequence of how the model was trained to use tools, and a checkpoint tuned to be more thorough on agentic benchmarks will call more tools on your traffic too. Verbosity moves with instruction-following training. Structural reliability moves with everything.

The parse failures had a specific and instructive cause. The extract-invoice binding used a forced tool call, so structurally the output was always valid JSON. The 0.91% came from a different route: when the document was a scanned image whose OCR pass — the optical character recognition step that turns a picture of a page into text before the model ever sees it — had produced a garbled transcription, the May checkpoint had a stronger tendency to decline to fill required fields and to emit an empty string where the February checkpoint had guessed. The tool schema declared total_ex_tax as a decimal string with a pattern, and empty strings failed the pattern at validation time in the application. The model was arguably behaving better; the pipeline treated it as a parse failure and retried three times before giving up.

Why an afternoon of trying it cannot find this#

The refusal shift is the clearest case for why manual inspection is the wrong instrument. moderate-content handles about 190,000 items a month, and the refusals concentrate in content that touches self-harm, medication and legal advice, which is roughly 3% of traffic. A shift from 0.4% to 2.9% overall is, within that 3% slice, a shift from about 13% to about 95% — near-total refusal on a class of input the product depends on being handled.

To detect a 0.4%-to-2.9% shift by sampling uniformly, with any confidence, you need on the order of five hundred observations. Three engineers with twenty prompts have sixty, and their sixty are not uniform: they are prompts the engineers could recall, which means prompts from tickets that were interesting, which means the happy path plus a few famous failures. The classes where policy changes concentrate are exactly the classes nobody keeps in their head, because they are handled correctly and quietly by the incumbent and therefore generate no memory.

This is the general shape of the problem. Manual inspection samples from the part of the input distribution that is salient to the person doing the sampling, and behaviour drift concentrates in the part that is not salient to anyone. Adding more engineers to the afternoon does not fix it, since they share the same salience.

Why these four things move and benchmarks do not#

The drift is not random and it is not a defect. A checkpoint within a family is the same pre-trained base with revised post-training, and post-training is where a vendor encodes the decisions that differ most from release to release: what the model should decline to do, how eagerly it should reach for a tool, how much it should explain itself, how strictly it should follow a format instruction against a competing instinct. Those are product decisions made against the vendor's own evaluation set and their own aggregate user feedback, and your traffic is a small and unrepresentative sample of what they were optimising.

Published benchmarks are largely insensitive to all four. A knowledge or reasoning suite scores a final answer against a reference and does not care whether the model took three tool calls or five to get there, whether it wrote 90 words or 210, or whether it declined the eleven items in the suite that touch medication, since those items are not in the suite. Agentic benchmarks come closer on tool use and still measure task completion rather than call count, so a checkpoint that completes the same fraction of tasks with 50% more calls scores identically and costs you 46% more. The published number is a measurement of capability, and every metric that moved in the table above is a measurement of disposition.

This also explains why the drift is not consistently in one direction across your tasks. A checkpoint tuned toward thoroughness improves extract-invoice, where a second look at an ambiguous field is worth the tokens, and damages research-agent, where a second look is a duplicate search. There is no single verdict on the release that would be correct for both, which is the argument for the per-task gates below rather than a global judgement about whether the new version is good.

Re-qualification is the existing suite plus operational gates#

The instrument that works is the task-level suite from chapter 2, run against the new identifier, with two categories of gate that accuracy-focused suites usually lack.

# tasks/research_agent/eval.py
GATES = [
    Gate("answer_quality",     rubric_score,          threshold=0.86),
    # structural
    Gate("schema_valid",       lambda r: r.parse_ok,  threshold=1.000),
    Gate("required_fields",    required_field_rate,   threshold=0.995),
    Gate("refusal_rate",       refusal_rate,          threshold=0.010, lower_is_better=True),
    # operational
    Gate("tool_calls_p50",     tool_calls_p50,        threshold=4.0,   lower_is_better=True),
    Gate("tool_calls_p95",     tool_calls_p95,        threshold=8.0,   lower_is_better=True),
    Gate("output_tokens_p95",  output_tokens_p95,     threshold=900,   lower_is_better=True),
    Gate("latency_p95_ms",     latency_p95,           threshold=11000, lower_is_better=True),
    Gate("cost_per_1k_usd",    cost_per_thousand,     threshold=48.00, lower_is_better=True),
]

Every bolded row in the drift table has a gate here that would have failed before the deploy. tool_calls_p50 at 5.1 against a 4.0 threshold fails; latency_p95_ms at 13,600 against 11,000 fails; cost_per_1k_usd at $60.30 against $48.00 fails. The refusal gate needs a case set with the sensitive slice in it, which this team did not have on the day and built afterwards from 300 items sampled from the 3% region — that slice is now the most valuable part of the suite, because it is the part that manual testing structurally cannot reach.

Thresholds set as absolute numbers work for gates like schema_valid. For the operational ones it is usually better to express them relative to the incumbent's measured value on the same cases in the same run, so that the suite reports "tool calls +50% versus serving binding" rather than comparing against a constant somebody wrote down a year ago and never revisited.

A model identifier is a deployable artefact#

The organisational half of this is that the config change shipped without an eval run because it did not look like a release. Version pinning, from topic 2.22, gives you the property that the identifier only changes when someone changes it; what this chapter adds is that the change is a deployment of new behaviour and needs the same gate as any other.

# .github/workflows/model-change.yml
on:
  pull_request:
    paths: ["config/models/**"]

jobs:
  requalify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Identify affected tasks
        run: python -m migration.affected --diff origin/main > affected.json
      - name: Run task suites against candidate identifier
        run: python -m eval.run --tasks @affected.json --binding candidate --compare serving
      - name: Post per-gate delta table to PR
        run: python -m eval.report --format pr-comment

The rule the team adopted afterwards was that a pull request touching config/models/** requires the affected suites green and a named approver from the owning team, the same as a schema migration. It also requires a canary: the new identifier serves 5% of traffic for 72 hours with the drift metrics on a dashboard, because a suite of 240 cases cannot contain a failure mode nobody has imagined yet, and the sensitive-slice gate exists only because a previous incident taught them that slice existed.

The canary is worth specifying rather than leaving to judgement, because the metrics that move are ratios over classes of traffic and a canary sized by request count can still miss them. Five per cent of moderate-content traffic over 72 hours is about 950 items, of which roughly 29 fall in the sensitive slice — enough to see a shift from 13% to 95% refusal within that slice, and nowhere near enough to see a shift from 0.4% to 0.6% overall. Sizing the canary by the rarest class you care about, rather than by total volume, is what makes the difference, and for classes rarer than about 1 in 500 the honest answer is that the canary cannot adjudicate them and the offline slice has to.

What re-qualification does not settle#

It does not establish that the new checkpoint is worse. On this evidence it was better at the two things the suites had always measured, and the accuracy gains were real and worth having. Six of the nine tasks moved to the May checkpoint within a fortnight with no changes at all. What the drift table changes is that the move became a decision with numbers rather than a config bump, and three tasks got explicit work: research-agent had its system prompt amended to discourage confirmatory lookups, which brought tool calls to 3.9; draft-reply got a length instruction; extract-invoice had its validation relaxed to accept null for fields the model declines to guess, which is a better pipeline than the one it had before.

It also does not remove the need for production monitoring. The suite tests what you have thought of, and every incident in this chapter began as something nobody had thought of. The drift metrics — refusal rate, tool calls per run, output length percentiles, parse failure rate — belong on a dashboard with alerts, sliced by task, permanently, because they are the signals that move when behaviour changes for any reason, including reasons on the vendor's side that never appear in a release note.

And the whole exercise assumes the move is optional, which is the situation this chapter has been in: the team chose to upgrade, could have chosen not to, and could roll back to the February checkpoint at any point in the eleven days. That freedom has an expiry date attached to it, published in advance, and the next chapter is about what you want standing before the notice with that date on it arrives.