When engineers disagree with your suite, the suite is the suspect

Three engineers had been using the candidate model in their own tooling for a week — one on a code-review assistant, two on the internal ops agent — and all three said the same thing in the same standup: it is obviously better. The suite said the weighted pass rate was 0.859 against the incumbent's 0.871, and two of the fourteen incident cases failed, so the mechanical verdict from the previous chapter was a clean no with a number attached.

Both of those are measurements. Neither is an opinion, which is what makes the standard resolutions wrong. Deferring to the suite because it is the objective one treats a 90-case sample as though it were the population, and deferring to the engineers because they have hands-on experience throws away the only reproducible measurement in the building. The productive move is to notice that two instruments pointed at the same system disagree, which means at least one of them is sampling something the other is not, and to find out which.

Two samples, drawn very differently#

Writing down what each measurement actually covers usually locates the disagreement before any new work is done. The third column is the tie-breaker built later in this chapter: a blind pairwise test, in which raters are shown two answers to the same real prompt with the model names stripped off and have to choose one, which removes the part of a preference that comes from knowing which model produced what.

The row that mattered here is turn structure. Every one of the 90 cases was single-turn, because the suite grew out of a summarisation feature where a single-turn case is the whole task, and the ops agent that two of the three engineers spend their day in runs conversations of twelve to twenty turns with live tool calls that sometimes fail. The suite had no coverage of instruction retention across turns, no coverage of recovery after a wrong tool call, and no coverage of what happens when a user changes the required output format halfway through. Whatever the candidate does in that region, the suite was silent about it, so the two measurements were not in conflict; they were describing different parts of the model.

Filing the bug against the instrument#

The collection step is deliberately narrow. Asking three people what they think produces adjectives; asking each of them for three specific sessions with a deadline produces artefacts. The request that works is specific about the format — a transcript or a session ID, the moment in it where the difference showed, and one sentence on what the incumbent would have done instead — and about the deadline, because a week of use is still fresh at 24 hours and reconstructed at 24 days. Two of the nine sessions that came back did not survive being read: one showed a difference that came from a different system prompt the engineer had been iterating on, and one showed the candidate doing something the incumbent also does when you rerun it. Losing two of nine to that is normal and is itself informative, since it calibrates how much of the original impression was about the model at all.

Reading the remaining seven yielded five distinct properties, three of which the suite had no case for:

• retaining a constraint stated at turn 2 through to turn 14, when the intervening turns are about something else
• recovering after a tool returns an error rather than repeating the same call with the same arguments
• declining to fabricate an order ID when the retrieval step returns nothing
• switching output format mid-conversation when asked
• brevity under a stated word budget

The third and fifth were already covered, and one of them exposed a different defect. On the brevity property the suite had two cases, both of which the candidate failed, and reading the transcripts showed why: the case checks len(out.text.split()) <= 60, and the candidate had produced a 54-word answer followed by a two-line markdown table summarising it, which pushed the token-ish word count to 71 while making the answer better by any reading. The check was a proxy for do not ramble and it had been silently measuring do not use structure. Fixing it meant rewriting the expectation to count prose words outside fenced or tabular blocks, then re-running every system on the corrected case, including the historical ones, because a scorer change is a ruler change.

That is a third explanation for a disagreement, alongside a coverage gap and novelty bias, and it is the one people forget to look for. A deterministic check is a compression of a requirement into something a computer can decide, and every such compression loses something. When engineers say a model is obviously better and the suite says it is worse, the failing cases should be read individually before any conclusion is drawn, because a check that was a good proxy for the old model's failure modes can be a poor proxy for the new one's successes. The other three became new cases, and the way they were written is the part that determines whether the exercise is science or theatre. The expected outcome for each was derived from the product requirement — the ops runbook says an agent must not invent identifiers, the support policy says a constraint stated by the user holds until they retract it — and was written and reviewed before anyone ran either model against it. Writing expectations after looking at the candidate's transcript produces cases that encode what the candidate does, which guarantees the candidate wins and destroys the suite's value as an instrument, and the temptation to do it is strong precisely because the candidate's transcript is right there and looks like a good answer.

The failure mode is easy to demonstrate. One draft of the tool-recovery case specified that after an error the agent should acknowledge the failure, re-read the tool's schema, and retry with corrected arguments — which is precisely the three-step behaviour the candidate had exhibited in the session, described at a level of detail nobody would arrive at from the requirement alone. The requirement says the agent must not repeat a failing call with identical arguments and must surface the failure to the user if it cannot recover within two attempts. Written that way, the case is passable by any model that behaves sensibly, including ones that recover in some other order, and the incumbent turned out to pass it 60% of the time rather than 0%. The over-specified version would have reported a 100-point gap where the real one was 40.

Blinding the humans#

Coverage gaps are one explanation for the disagreement. The other is novelty bias, and it is not a slur on the engineers; a person trying a new model attends more closely, writes better prompts, retries more patiently, and remembers the moments where it surprised them. Separating the two costs ninety minutes.

# blind_pairwise.py — 40 prompts drawn from last week's real sessions
pairs = []
for prompt in sample_recent_prompts(n=40, source="ops_agent", days=7):
    a, b = incumbent.run(prompt), candidate.run(prompt)
    left, right = shuffle_and_record(a, b)      # mapping written to a sealed file
    pairs.append({"prompt": prompt, "left": left.text, "right": right.text})

# three raters, no model labels, forced choice, no ties
# results after unsealing:
#   overall            candidate 27 / 40   (67.5%, p = 0.019)
#   multi-turn prompts candidate 18 / 20   (90.0%, p = 0.0004)
#   single-turn        candidate  9 / 20   (45.0%, p = 0.82)

The split is the answer, and the p-values attached to it are the probability of seeing a division at least that lopsided if the raters had been choosing at random, so 0.019 describes a preference worth believing and 0.82 describes a coin toss. On multi-turn prompts the engineers' preference is overwhelming and survives blinding, so the candidate genuinely is better in the region the suite does not cover. On single-turn prompts, where the suite does have coverage and says the candidate is slightly worse, blinded raters could not tell the two apart, which means the week of enthusiasm was generalising a real multi-turn advantage across the whole model. Both instruments were reporting correctly about their own sample, and both readings of the situation — the suite's and the engineers' — were wrong as accounts of the model overall.

Running the same blind test with the labels shown, as a control, is worth doing once in the life of a team. When it was tried here on a subset of ten pairs, the same raters preferred the output labelled as the new model 8 times out of 10, including on three pairs where the labels had been swapped, and after that nobody argued about whether blinding was necessary.

What this procedure does not settle#

It does not make the humans the authority. The blind test was decisive here because it was blind, forced-choice, and drawn from real prompts rather than from remembered highlights; unblinded impressions had a demonstrated error rate on the same material. When a disagreement is about tone or helpfulness, blinded human preference is often the only instrument available and should be treated as the measurement of record for those properties, with the sample size stated. When it is about whether an order ID is real, human preference is irrelevant and a deterministic check is correct.

It also does not override the vetoes. After the three new properties were encoded as 22 multi-turn cases and the suite went to v10, both systems were re-run: the incumbent scored 0.793 and the candidate 0.864, a 7.1-point gap that clears the +4.0 gate and sits well outside the noise band. The two failing incident cases still failed, and they are a veto, so the mechanical verdict was still no. What changed is that the no now had a specific, cheap remedy attached — those two cases describe a summarisation behaviour that a prompt change might fix — rather than being a verdict on the model. Three weeks later, with a prompt adjustment, 14 of 14 passed and the migration went ahead on the original gates, unmodified.

The generalisable part is not the outcome but the default. When a suite and experienced users disagree, the suite has a specific weakness that daily use does not: it can only ever measure what somebody thought to encode, and it was encoded in the past. Daily use has the opposite weakness of sampling unsystematically and remembering selectively. Pointing the investigation at the suite first is right because a coverage gap is both more likely and more valuable to find, since finding it improves every future comparison, whereas concluding that the engineers were dazzled improves nothing.

This time the gap was found because three people were annoyed enough to say so in a standup. That is not a mechanism you can rely on, because the failure modes that arrive quietly — the ones nobody uses the new model long enough to notice — produce no standup complaint at all, and a suite whose cases were all written eleven months ago will keep returning confident, stable, meaningless numbers while the product moves underneath it.