Evaluating a constraint change

An eval harness answers a question about a model by running it over a set of inputs and scoring what comes back. That works while every call produces something scoreable. A constrained-decoding rollout breaks the assumption, because the whole point of the change is that responses which used to be unscoreable now arrive as valid documents, and any harness with a continue in its exception handler will compare a metric over one denominator against a metric over another.

The size of that artefact is not small and it is not obvious. On a forty-category classifier it accounted for roughly three quarters of an apparent regression, which is enough to reverse a decision. What follows is where the distortion comes from, how to build a harness in which it cannot occur, and how much of the difference that remains is worth believing.

One line decides the comparison#

Here is the shape of the harness that produces it:

for row in eval_set:
    raw = call_model(row.text)
    try:
        pred = Classification.model_validate_json(raw)
    except ValidationError:
        continue          # <-- here
    results.append((row.gold, pred.category))

That continue is the whole defect. In a prompt-only arm, 61 of 2,000 tickets fail validation and are skipped, so a reported macro-F1 of 0.712 is computed over 1,939 rows. In a strict arm nothing fails, so 0.681 is computed over 2,000. Two metrics over two different populations, printed one under the other, and read as a 3.1-point regression.

Two subtractions produce that reading and only one of them appears anywhere in the output. The first is 2,000 minus 61, which is the 1,939 rows arm A's figure is an average over. The second is 0.712 minus 0.681, which is the 3.1 points. The report prints the second and never mentions the first. A metric published without its denominator is a fraction with the bottom torn off, and the whole of this chapter's finding is that the two arms have different bottoms.

The 61 skipped rows are not a random sample of the eval set. Whatever made the model produce an unparseable or unvalidatable answer — a long rambling ticket, a mixed-language body, a case sitting between two categories — is correlated with getting the label wrong. Dropping them from the denominator removes the hardest inputs from one arm and not the other, and the arm that drops them looks better for a reason that has nothing to do with its predictions.

A funnel that narrows on one arm and not the other is the general shape of the problem; the continue is only this harness's version of it. Timeouts swallowed by a bare except, a repair loop that silently succeeds on one arm, and a cache that serves one arm's easy cases from disk all produce the same asymmetry with different code.

The arithmetic, on the same 2,000 tickets#

Run both arms over the identical fixed input set and score them twice: once the way the harness did it, and once with a validation failure counted as a wrong answer rather than a missing row.

The two accuracy rows are worth reconstructing, because they are the same numerator over two different denominators and nothing else. Arm A got 1,536 predictions right. Divided by the 1,939 rows the old harness scored, that is the 79.2% it reported; divided by the 2,000 inputs it was actually handed, it is 76.8%. Arm B got 1,571 right out of 2,000 either way, because it had nothing to drop, so its two rows are the same number twice. The difference between arm A's pair is not a second measurement of arm A. It is the same measurement made to account for the 61 tickets it declined to answer.

Which denominator is correct is not a statistical question, and treating it as one is how the argument stalls. It is a question about what the service does with a response that fails validation. If those tickets end up unclassified, the model got them wrong in every sense the business recognises, and the only defensible denominator is the number of tickets that arrived. The old harness was quietly answering a different question — how good are the predictions, conditional on a prediction having been produced — and that question has no consumer, because nothing downstream is permitted to skip a ticket.

The accuracy row changes sign. Reported the old way, A beats B by 0.6 points; over the same denominator, B beats A by 1.8. Nothing about either model's behaviour changed between those two rows — the only difference is whether the 61 failures counted as errors or as absences, and in production they are errors, because a ticket that comes back unparseable does not get classified.

Macro-F1 does not change sign, and the residue is the interesting part. Over the fixed denominator the gap narrows from 3.1 points to 0.8, so roughly three quarters of the apparent regression was the population moving. The remaining 0.8 is real, and it is the two adjacent billing categories where an enum forces a boundary call that the label strings do not define. Macro-F1 averages over classes rather than over rows, so a loss confined to two of forty classes survives a correction that mostly washes out of micro accuracy — which is exactly why both numbers belong in the table.

Scoring a failure as wrong needs one convention decided explicitly. The clean version assigns every failed response a reserved label, __invalid__, that is never any row's gold label. It becomes a false negative for the true class and never a false positive for a real class, which keeps per-class precision honest and pushes the whole cost onto recall where it belongs. The alternative some teams use — assigning failures to the majority class, or to other — quietly credits the model for guesses it did not make and distorts one class's precision.

Medicine has a name for this distinction. An intention-to-treat analysis scores every patient who was randomised, including the ones who dropped out, precisely because dropping out is not independent of outcome; a per-protocol analysis scores only the completers and reliably looks better. A harness with a continue in it is running per-protocol on one arm and intention-to-treat on the other.

A harness that cannot make this mistake#

The structural fix is to make the failure a value rather than a control-flow escape, so that no downstream code has to remember it happened.

@dataclass
class Outcome:
    input_id: str
    arm: str
    gold: str
    pred: str                 # a real category or "__invalid__"
    failure: str | None       # "json_decode" | "schema" | "truncated" | None
    latency_ms: float
    output_tokens: int

def run_arm(arm: str, call, eval_set) -> list[Outcome]:
    out = []
    for row in eval_set:
        t0 = time.perf_counter()
        resp = call(row.text)
        dt = (time.perf_counter() - t0) * 1000
        pred, failure = "__invalid__", None
        if resp.finish_reason == "length":
            failure = "truncated"
        else:
            try:
                pred = Classification.model_validate_json(resp.text).category
            except json.JSONDecodeError:
                failure = "json_decode"
            except ValidationError:
                failure = "schema"
        out.append(Outcome(row.id, arm, row.gold, pred, failure,
                           dt, resp.usage.completion_tokens))
    return out

Every row of the eval set produces exactly one Outcome in every arm, which makes the comparison paired by construction: for any input you can ask what each arm did with it, and the denominators cannot drift apart. Carrying latency_ms and output_tokens on the same record costs nothing and means the cost and latency comparison comes out of the same run as the quality comparison, on the same inputs, instead of from a separate benchmark against a different traffic mix.

Keeping the failure reason rather than a boolean matters more than it looks. A run where all 61 failures are schema says the prompt-only arm produces well-formed JSON that disagrees with the contract. A run where 40 of them are truncated says the ceiling is too low and both arms are affected, so the comparison is measuring something other than the constraint.

Reading a difference that might be noise#

With paired outcomes, the right test is over the disagreements rather than over the two accuracy figures. On the 2,000 tickets, B was correct where A was wrong on 148 inputs, and A was correct where B was wrong on 113. The 1,739 inputs where both arms agreed carry no information about which is better, and averaging them into two percentages is what makes small differences look solid.

McNemar's test — the standard check for whether a paired win-and-loss split is larger than chance would produce — run over those discordant pairs, the inputs on which exactly one arm was right, gives χ² = (|148 − 113| − 1)² / (148 + 113) = 34² / 261 = 4.43, p ≈ 0.035.

Every part of that expression is doing a nameable job. The concordant inputs are excluded because a ticket both arms got right, or both got wrong, says nothing about which arm is better — it would look identical whichever arm were superior. What remains is 148 wins for B and 113 for A, and the question is whether a split that lopsided could come from a fair coin. Under the assumption that neither arm is better, each discordant pair is a coin flip, so the expected split is even and the observed imbalance is the gap between the two counts, which is 35. Subtracting one from that gap is a continuity correction: the statistic is being read against a continuous curve while the counts feeding it are whole numbers, and the subtraction stops the test being slightly too eager to declare a win. Squaring the corrected gap and dividing by the total number of discordant pairs puts it on the χ² scale, where a value above about four is the conventional bar at one degree of freedom — 34 squared is a little over eleven hundred, and that over 261 is the 4.43 above, which clears it.

The denominator in that fraction is why the size of the eval set matters more than the win rate does. Adding discordant pairs grows the bottom of the fraction in proportion to their number, while a real effect grows the top as the square of the imbalance, so a genuine difference gets easier to detect as the set grows and a spurious one does not. That is the arithmetic behind the warning below about small evals: with few discordant pairs only an enormous imbalance clears the bar, and an enormous imbalance is not what a one-point quality change looks like.

The net gain of 35 rows out of 2,000 is the same 1.75 points the fixed-denominator accuracy row showed, and it is unlikely to be noise at this sample size. Run the same test on a 200-row eval set and the equivalent effect would need roughly 15 net discordant rows to clear the same bar, which most 200-row evals cannot supply — a useful thing to know before someone concludes anything from a comparison over a hundred hand-picked cases.

The per-class picture needs a different instrument, because a two-class regression inside a forty-class average is not visible in any aggregate. Print per-class precision, recall and support — support being the number of eval rows whose gold label is that class — for both arms, sorted by the change in F1:

Two classes moving by 0.18 and 0.14 against thirty-eight sitting near zero accounts for the whole residual: 0.32 spread over 40 classes is 0.008 of macro-F1, which is the 0.8 points the fixed-denominator row showed.

The division by 40 there is the definition of the metric doing its work rather than a convenient fudge. Macro-F1 is the plain average of the per-class F1 scores — their sum divided by the number of classes, with every class weighted the same whatever its support — so a change in the sum moves the average by that change divided by the same count. The two billing classes lost 0.18 and 0.14, so the sum fell by 0.32; over forty classes that is 0.008 on the average; and 0.008 of F1 is what "0.8 points" means under the convention of quoting F1 in hundredths. The thirty-eight other classes barely move, which is the fact that lets a two-class calculation account for the whole residual on its own. It is also why the same damage would be nearly invisible in micro-accuracy, where each class contributes in proportion to its 48 or 41 rows out of 2,000 rather than as one fortieth of the total.

That sorted table is what converts "constrained decoding hurt us" into "these two labels are ambiguous", and the second statement has a fix. The precision and recall columns say more than the F1 column does. Both billing classes lost precision and recall together, which is the signature of two labels being swapped for each other rather than of either being abandoned, and a confusion matrix restricted to those two rows confirmed it: 31 of the 37 newly-wrong predictions were one billing label given where the other was gold.

How much of that 0.8 to believe is a separate question from how it arose, and the paired records answer it directly through a bootstrap, which estimates how uncertain a statistic is by recomputing it many times over resamples of the data already collected. Resample the 2,000 inputs with replacement, recompute both arms' macro-F1 on each resample using the predictions already recorded, and take the percentiles of the difference:

import random, statistics

def delta(sample: list[tuple[str, str, str]]) -> float:   # (gold, pred_a, pred_b)
    return macro_f1([(g, b) for g, _, b in sample]) - macro_f1([(g, a) for g, a, _ in sample])

boot = [delta(random.choices(paired, k=len(paired))) for _ in range(2000)]
boot.sort()
print(statistics.mean(boot), boot[50], boot[1950])    # -0.008 -0.015 -0.002

The interval runs from −0.015 to −0.002 and excludes zero, so the residual loss is real at this sample size and it is somewhere between two tenths of a point and a point and a half — a claim of "3.1 points" was wrong by roughly a factor of four in the middle of that range. Bootstrapping the two billing classes on their own is a different matter: with 48 and 41 supporting rows the per-class intervals are wide enough to overlap zero, so the honest reading is that those classes point at the vocabulary while the eval set is too small to price the effect precisely.

What makes a bootstrap a legitimate estimate is worth a paragraph, because resampling data you already hold looks like getting something from nothing. The question being asked is how much the 0.8 would move if a different 2,000 tickets had been drawn from the same traffic. Those tickets cannot be drawn — they were never labelled — but the 2,000 in hand are themselves one draw from that traffic, so drawing from them with replacement, letting one input appear twice while another appears not at all, imitates the variation a fresh draw would have shown. Recomputing the difference on each imitation produces a spread, and the percentiles of that spread are the interval. No new model calls are made, because the predictions were recorded per input and the only thing changing between resamples is which inputs are counted.

The same logic is why the per-class intervals are uninformative here, and saying so is part of the result rather than a hedge. A resample of 48 rows varies a great deal from one draw to the next, since dropping or duplicating a handful of tickets moves a small class's F1 substantially, and an interval computed over such resamples is wide by construction. Wide enough to include zero is not evidence that the billing effect is absent. It is evidence that 48 and 41 rows cannot price it, which is a statement about the eval set rather than about the classes.

Two places the population moves that a fixed denominator does not catch#

There is one more place the population moves, and it happens before the harness runs. The 2,000 tickets were assembled by sampling rows from the extractions table, which is a perfectly natural thing to do and quietly restricts the eval set to inputs that already succeeded in production, since a response that failed validation never produced a row there. The hardest 3% of real traffic was therefore not eligible to become an eval input in either arm.

The symptom is a fixture set on which the prompt-only arm reports a 0.6% validation failure rate while production reports 3.0% for the same code against the same model, and the usual reading of that gap is that the eval set is "clean" rather than that it is filtered.

The belief that gets this past review is that a fixture set drawn from production is the most realistic thing available, and it is nearly right: the set is realistic about everything except the one variable under test. The chain is short enough to check in a schema browser. A response that fails validation is never written to extractions. An input whose response failed therefore has no row there. Sampling rows samples inputs whose responses succeeded. So the eval set is conditioned on the very outcome the eval exists to measure, and it excludes the hardest inputs from both arms through the same filter — which flatters arm A in particular, because arm A's own failures are what did the excluding.

Rebuilding it by sampling the request log instead — every input the service was asked to handle, including the ones whose responses went to the dead-letter queue — moved arm A's failure rate on the fixture set from 0.6% to 3.1% and took its fixed-denominator accuracy down a further 1.4 points, while arm B moved by 0.2. Keeping the dead-letter inputs as a named slice with its own reported number is worth the extra column, because it is the slice where any change to the constraint will show up first.

The second is that counting a validation failure as a wrong answer models a service that drops failures. If yours retries, the honest comparison scores what production actually returns, which means running the repair path inside the arm and recording the outcome after repair, along with the extra calls and latency it consumed. On this service the prompt-only arm had a one-shot retry that recovered 44 of the 61 failures at a cost of one additional call each; scored after repair, arm A lands at 78.9% accuracy against B's 78.6%, and the two arms are statistically indistinguishable on quality while A spends 3% more calls and roughly 700 ms of extra p99 latency on the tickets that need repairing. That is a genuinely different conclusion from either of the first two, and it is the one that matches what users experience.

Which convention to use is therefore a question about your service rather than about evaluation in the abstract. Score failures as wrong when they are dropped or surfaced as errors; score after repair when there is a repair loop, and carry its cost in the comparison. What you cannot do is exclude them, because excluding them is the one convention that corresponds to no production behaviour at all.

The other limit worth stating is that none of this tells you whether the constraint should be on. It tells you what the constraint did to quality on a fixed set of inputs, with the population artefact removed and the uncertainty quantified. Turning that into a decision requires the two columns the harness collected but this chapter did not spend: what the constraint did to latency and to token spend, and who in the organisation reads each of those numbers.

There is also an assumption running underneath the entire comparison. Arm B was labelled "strict schema" because the request carried a schema and the provider returned 200, and every number in this chapter depends on the constraint having actually been applied to every one of those 2,000 calls. That is not something a status code can tell you.