Counting reverts and lost incident hours on the same ledger

The useful question to put to any programme's report is what outcome would have caused us to stop. If there is no such outcome, the number at the top is decoration. A prompt search programme is unusually prone to this because its benefit arrives in a currency the programme itself invented — points on a holdout — while its costs arrive in engineer-hours, in reverts, in incident time and in the depletion of an asset nobody prices.

Building the comparison is two pieces of work. The benefit figure has to be audited, because the natural way of constructing it is wrong in two specific ways. Then both sides go on one ledger in a currency somebody actually spends, which is engineer-hours, with cash beside it where the cash is real.

The benefit number is smaller than it looks#

A cumulative total built by adding up monthly margins is wrong twice over. Reverted contributions are still in it: a prompt that was live for eleven weeks and then replaced by the baseline is not contributing to today's quality, so its margin has no business in a running total. And summing margins compounds selection, because each month's margin was measured against a holdout that had by then been scored several times, and each month's baseline was the previous month's winner, so the errors accumulate in a consistent direction.

The second of those is worth following one link at a time, because "the errors accumulate" is exactly the sort of sentence that gets nodded at. Month one measures a winner against a hand-written baseline, and its margin is a little optimistic for the reason every selected margin is: the winner was kept partly because those particular holdout cases happened to like it. Month two then measures its own winner against month one's winner. So the baseline for month two already has month one's optimism inside it, and month two's margin is added on top rather than netted against it. By month six the running total is a sum of six margins, each measured from a starting point that was itself flattered, with every error pointing the same way because selection only ever pushes upward. Nothing in the total is arithmetically wrong. It is simply not a measurement of anything.

The only honest cumulative figure comes from one run: today's live prompt against the original baseline, both scored on the current holdout, in the same run, with the current model.

That single comparison closes both leaks at once, which is why it is worth preferring to any reconstruction from the monthly reports. Reverted contributions drop out because the left-hand side is whatever is serving traffic today, and a prompt that was reverted is not that. The compounding drops out because there is no chain of baselines left to compound: the right-hand side is the prompt from before the programme started, and nothing ever selected it, so it carries no optimism for a later margin to inherit. One measurement, one comparison, both prompts under the model that is in use now.

res = evaluate_both(
    a=load_prompt("drafting@v19-live"),
    b=load_prompt("drafting@v08-may-baseline"),
    cases=holdout,           # current holdout, scored once for this question
    judge=JUDGE_CONFIG,
)
print(res.margin, res.paired_ci)   # 0.018, (-0.004, 0.041)

That is +1.8 points with a paired interval that includes zero: an improvement that is probably real, probably smaller than the running total claimed, and not distinguishable from nothing at the holdout's resolution. That is the figure that goes on the ledger.

An interval that includes zero invites two readings and both of them are wrong. It does not establish that the programme achieved nothing: the measured margin is positive, and the same interval covers gains larger than the one observed. It does not establish a gain either, because the data is equally consistent with no difference at all. What it establishes is that this holdout cannot settle the question, which is a statement about the instrument rather than about the prompt. The honest response is to carry the interval onto the ledger beside the point estimate, rather than picking whichever end of it supports the argument somebody was going to make anyway.

There is a second question the benefit side needs and often cannot answer: what is a point worth. A rubric score is a proxy, and unless someone has connected it to a business quantity, the comparison is between an unpriced benefit and a priced cost. On one system the link existed and was weak — a matched comparison over 6,000 conversations, pairing each conversation served by one prompt with a conversation of the same topic, length and customer segment served by the other, so the two are compared on like with like — putting a 1-point rubric gain at roughly 0.4 percentage points of reduction in human escalation. Escalations cost about £4.10 each in support time, so 1.8 points across 400,000 monthly conversations is around £11,800 a year. Where no such link exists, say so plainly rather than implying one; an unpriced benefit against a measured cost is a weaker conclusion but an honest one.

The chain of operations there is the part worth carrying away, because most teams have to build it themselves and the order is what makes it auditable. A rubric point is first converted into a rate — how much of a percentage point of escalation one point buys — and that is the step requiring the matched comparison, the step nobody can do from a dashboard. The rate is multiplied by the measured gain to give a change in escalation rate. The change in rate is multiplied by conversation volume to give a count of escalations avoided. The count is multiplied by what an escalation costs to give money. Four inputs, three multiplications, each separately disputable — which is the reason to write them out rather than quote the total, since a reviewer who disagrees can then name the factor they disagree with. Check the time units at every step. The volume figure and the money figure have to cover the same period, and a rate established per month carried into an annual total is the commonest way this particular arithmetic goes wrong by a factor of twelve.

Both sides, one ledger#

Engineer-hours are the better primary currency, because they are what the team actually spends and what the alternative uses would consume. Dollars go alongside where they are real.

That table is meant to be added up rather than skimmed, and two of its cells repay reconstruction. The hours column runs 22 for the initial setup, 9 across six monthly runs, 8 for two reverts, 9 for the October incident and 4 for holdout depletion, and those five entries are what make the 52 in the total row. The cash column carries one entry, and it is the same six runs: at $47 of provider spend apiece they come to the $282. That two reverts cost four hours each and an incident cost nine is the kind of thing a team knows and never writes down, and writing it down is the entire exercise, because an unrecorded cost is indistinguishable in any report from a cost of zero.

Fifty-two engineer-hours over six months is about 1.7% of one engineer, which is not a large number, and £11,800 a year against $282 of spend is a favourable ratio if the estimate holds. Reasonable people can now disagree about whether to continue, and that is the entire objective of the exercise. A points-and-nothing-else report admits no disagreement at all.

Two lines deserve defending because they are the ones people push back on. The incident day belongs on the ledger when it is attributable: a prompt merged in August without the block-by-block annotation a rollback rule requires, an on-call engineer at 02:30 who cannot tell which of its paragraphs is producing the behaviour a customer is complaining about, and a change that would have taken twenty minutes on a hand-written prompt waiting for the author to be available. If the same night had happened to a hand-written prompt it would not be on this ledger. The holdout line is there because a holdout is capital, it is depleted by use, and a programme that spends it should carry the amortised replacement cost the way any process that consumes a shared asset does.

What is deliberately absent is compute measured as wall-clock. The runs happen overnight on infrastructure that is otherwise idle, so six hours of search time costs the provider spend and nothing else, and inflating it into a scarier number would be dishonest in the other direction.

Diagnostic findings, counted carefully#

One benefit is real and does not appear as points. Across six runs a search can surface defects nobody was looking for: eleven candidates independently attempting to suppress hedging, which exposed a judge that rewarded confidence and led to fourteen refusal cases being added to the eval set; a chunker that split tables, discovered because four candidates in one run added instructions about reconstructing them; and the finding that the objective was flat over a wide region, which ended eight months of arguing about prompt wording in code review.

That value is genuine and it is also the perfect escape hatch for a programme that is failing, because "we learned a lot" can be claimed for any outcome. The discipline that keeps it honest is to count only diagnostic findings that turned into a merged fix with a ticket number, and to write them on the ledger as the hours they saved or the defect they closed. Two of the three above qualify; the third is an insight the team acted on but cannot price, so it goes in the write-up as prose and not in the table.

What a ledger cannot settle#

Attribution is arguable on every line and the argument is worth having in advance rather than in the review. The incident day above is on the ledger because a specific, checkable thing was missing — an annotation the rule required — and someone can dispute that the annotation would have helped. What should not happen is the line being dropped because it is disputable, since the alternative is a cost of zero, which is certainly wrong. The rule that works is that a cost goes on the ledger with its attribution stated, and the dispute is about the number rather than about whether the row exists.

The benefit side has the harder gap. Where no link between rubric points and a business quantity exists, the ledger compares a measured cost against a benefit in units nobody outside the team uses, and no amount of care in the accounting fixes that. A weak link built from a matched comparison is worth far more than none, and it is worth building once even if it is only re-derived annually, because it is what allows the programme to be compared against the other things the same hours could buy.

And a ledger says nothing about counterfactuals. Fifty-two hours spent on prompt search is fifty-two hours not spent reading failures, and the comparison earlier in this topic suggests error analysis returns more per engineer-hour on a system with structural defects left in it. That comparison belongs beside the ledger and cannot be derived from it, because the ledger only records what was done.

The first run, from nothing#

If none of this exists yet, the first run is small and the order matters. Read thirty recent failures and cluster them by cause; fix anything structural you find, because a search over a broken pipeline produces a workaround that dies with the fix. Split the eval set by entity with a stable hash, and put the holdout somewhere the optimiser has no handle on. Write the constraint filters before the first run — a token ceiling, a latency ceiling, and a policy judge over the candidate text — so the first thing you see is not a candidate you will be tempted by. Run three seeds, report the range rather than the maximum, diff the winners for instructions that recur, and consider hand-writing those instead of merging the machine's text. Write the rollback rule and its thresholds before you look at the scores. Then score the holdout once, and record that you have spent one of its scorings.

A search will reliably find something that moves your metric. Whether that something is worth having is a question the search cannot answer, and the ledger is how a team answers it: benefit audited down to one honest measurement, cost counted in the currency the team actually spends, and a stopping condition committed to before anyone knows which way it will cut.

A programme with a written stopping condition is one that can be shown to have failed, which is the only kind whose success means anything.