Repairing an inherited harness

A harness that violates every design rule at once is the common case, not the pathological one. Eight criteria scored on a 1-to-5 scale in a single call, the score fields declared ahead of the rationale, the judge the same model as the generator, and no record of what any of it costs: each of those was a reasonable default at the moment somebody wrote it, and together they gate a release train.

The tempting plan is to fix all of it in one branch and ship a new harness. A fortnight is enough time to do that. It is not enough time to recover from it, because a harness whose eight changes landed together produces one new number and no way to tell which change produced which part of the difference. What follows is the ordering that keeps each change attributable, the baseline that has to exist before the first one, and the ledger that makes the whole exercise defensible to a room that is about to see its quality number fall by thirty points.

The first thing to build is not a fix#

Before anything is changed, freeze a baseline that survives the fortnight. Pull 200 outputs with their contexts from the last month of production, store them as files, and never regenerate them. Have two people label 60 of those 200 against the eight criteria individually — not an overall acceptability judgement, since a per-criterion label is the only kind that can be compared with what the harness will produce once the criteria are binary. Record the current harness's verdicts on all 200.

That is a day and a half of work that changes nothing about the system and makes every subsequent change measurable. Without it, each fix produces a number that can only be compared with the previous number, and by the fourth fix you have a chain of deltas with no external reference. With it, every fix is scored against a fixed human standard, and Cohen's kappa against those 60 labels becomes the metric that says whether the harness is getting better — which the pass rate cannot say, since the pass rate is what the harness does rather than how right it is.

That last clause is the load-bearing one, and it repays being spelled out, because a room that has watched a pass rate for a year will reach for it by reflex. A pass rate is computed from the harness's own outputs alone: count the verdicts, divide by the examples. Nothing outside the harness enters the calculation, so no value the harness produces can be wrong in the sense of disagreeing with anything — a harness that passed every answer would report a perfect score and be internally consistent. Agreement with hand labels is computed from two things, the harness's verdicts and somebody else's, so it can disagree, and disagreement is the only signal in the exercise. Moving the pass rate is something any change to the prompt will do. Moving the agreement means the harness and a careful reader now reach the same verdict more often, which is the only sense in which a judge can improve.

Cohen's kappa is that agreement with the free part subtracted out — the share of matches two graders would have hit by chance given how often each of them passes things — so 0 means no better than coincidence and 1 means perfect. An inherited harness of the kind described above scores kappa 0.42 against frozen labels, which is the middling range where roughly half the agreement that was available beyond chance has been achieved. That number, not its 92% pass rate, is the one a repair is trying to move.

Two properties of the 60 labels are what let them play that role, and both are choices rather than defaults. They are per-criterion rather than overall, because the repaired harness will emit a verdict per criterion and a single acceptability label has nothing to compare against. And they are produced by two people rather than one, because a standard that has not been shown to agree with itself cannot be used to convict an instrument of disagreeing with it. Skip either and the kappa column later in the fortnight is unattributable — it moves, and nobody can say whether the harness got closer to the standard or the standard was never stable enough to be approached.

Ordering by effect per unit of work#

Step 3 goes early because it is a two-hour schema change with no dependency on anything else, and because its effect is measurable in isolation on the frozen 200. Step 4 is the largest piece of work and the largest effect, and it has to follow step 3 rather than precede it, since rewriting criteria while the judge is still anchoring means you cannot tell whether a criterion is bad or whether the harness never gave it a chance. Step 5 has to follow step 4 because splitting an ordinal composite into eight calls produces eight ordinal scores, which is eight times the cost of the same problem.

Those two dependencies are worth separating, because only one of them is about sequencing and the other is about waste. The first is an attribution argument. A criterion rewritten under a verdict-first schema is being tested by a judge that commits before it reads, so a rewrite that ought to have worked will look inert, and the natural response is to rewrite it again. Fixing the schema first means a criterion that still fails to move the kappa has genuinely failed. The second is an economics argument, and it does not depend on the first. Splitting a 1-to-5 composite into eight separate calls buys eight separate scores that each have the middle-of-the-scale problem the first chapter of this topic measured, at eight times the inference bill. The split is worth paying for once the verdicts are binary and worth nothing before that, so doing it early spends the fortnight's budget on multiplying a defect.

Diagram

Each of steps 3, 4 and 5 ends with a re-run over the frozen 200 and a row appended to a ledger. The ledger is the deliverable that makes the fortnight defensible:

Read down the middle column and the harness fell 31 points. Read down the right-hand column and it doubled its agreement with careful human reading. Those are the same six runs.

The ledger also shows where the fortnight's value came from, which is not evenly spread. The two-hour schema reorder in step 3 moved kappa by as much as the three-day criteria rewrite in step 4, and both of them moved it more than everything after them put together. That ordering — biggest effect per unit of work first — is what makes the fortnight defensibly a fortnight rather than a quarter: if the calendar had run out after step 4, the harness would already be carrying most of the improvement and every row up to that point would still say what it bought. A plan whose value arrives at the end is a plan that has to be finished to be worth anything.

Reading the two columns together is also the only way to see that the pass rate fell for a good reason. Each row's definition is written in the last column, and the definitions change as the harness changes, so the middle column is a series of numbers about five different things. The right-hand column is a series about one thing throughout: how often this harness and two careful readers reach the same verdict on the same 60 outputs. That is why the right-hand column is the one a reviewer should be shown first, and why plotting the middle column as a trend would be a mistake even though it is the column everyone recognises.

Two rules the calendar does not support#

Two of the design rules do not get fully adopted in a fortnight, and both omissions can be defended in writing.

The cross-family judge does not become the production judge. Procurement at many companies takes six weeks, so within two you can get an evaluation key for a 60-example slice and measure the offset, which came out at nine points on this harness. What ships is the same-family judge with a documented +9 point leniency published beside the number, and a procurement request with a month of offset data attached. That is a smaller claim than the rule makes and it is the one the calendar supports.

The call split does not apply to every criterion. Three of the eight are mechanical — a format check, a required-disclaimer check, and a citation-span presence check — and those became code rather than judge calls, which is both cheaper and strictly more reliable than any split. Five criteria go to five calls. Describing this as skipping the rule undersells it; the rule exists to stop verdicts contaminating each other, and a regular expression cannot be contaminated.

What is not skipped is the binary conversion, because everything else in the ledger depends on it. Two of the eight criteria resisted rewriting — "the answer is appropriately concise" and "the tone matches our brand" — and both were cut rather than kept as scales. Neither had ever caused a release to be blocked, both were unstable across re-runs, and a criterion that fails neither of those tests is not carrying weight.

Golden sets age and criteria drift#

The repaired harness measures the eight properties someone wrote down, on 400 examples someone chose, judged by a model that will be deprecated within eighteen months. None of those is stable. Golden sets age as the product changes, and a suite that has not had examples added in six months is measuring last year's product. Criteria drift as the team's standards move, and the harness will keep reporting confidently against the old text. Judge model upgrades shift every number by a few points, which is why the frozen labelled slice has to be re-run at every model change rather than kept as a one-time exercise.

The frozen 200 has the same problem in a sharper form. Its value comes entirely from being unchanged, and the day the product changes enough that those outputs no longer resemble what the system produces, the kappa computed against them stops describing the current harness. Re-labelling a fresh 60 is the maintenance cost of the whole arrangement, and it is the line item most likely to be dropped once the fortnight's crisis is over.

Repairing a harness, then, is a sequencing problem rather than an engineering one. Every individual change here is small — a schema reorder is two hours, the criteria rewrite is three days — and the thing that makes them worth doing is that each one lands alone against a fixed human standard, so the ledger can say which change bought which points. A branch containing all of them produces one number and an argument.

Running this is a weekly habit rather than a project: the nightly suite reports, someone reads the failures grouped by criterion, the criteria that generate arguments get rewritten, and new examples go into the golden set every time production produces a failure the suite did not catch. The rules in this topic determine whether the numbers that habit produces mean anything; what they mean is decided by what you do on the mornings the number moves.