Disclosing a regression while it is still yours to disclose

The Monday job finishes at 08:14 and the four-week rolling rate on the policy assistant reads 88.2%, against 92.4% the week before. The provider pushed a point release of the model on Thursday and the deployment picked it up automatically, which is the obvious suspect. Nobody has complained. Support has logged four flags all week, which is inside the usual range. There is a branch open on your machine that pins the previous model version, forcing the deployment onto a named older build instead of whatever the provider is currently serving, and it would take about forty minutes to test and ship, and the Monday number would be back above 92% next week with nobody outside the team having heard about any of it.

Finding a regression before a customer does is the best position available, and the temptation to spend it on a quiet fix is strong precisely because the fix is easy. What the quiet fix buys is one uncomfortable conversation avoided; what it costs is the ability to be believed the next time you say quality is fine, which is a thing you will need repeatedly and cannot rebuild quickly.

First establish that four points is four points#

Before anything is disclosed it has to be measured properly, and the weekly live sample is the wrong instrument for this comparison. The 150 answers sampled last week and the 150 sampled this week are different questions asked by different people, so the difference between 92.4% and 88.2% carries the sampling error of both samples. Working that through gives a standard error — the typical distance between a figure measured on a sample and the true figure it estimates — of about 3.4 points on the difference, so the 95% interval runs from roughly minus 10.9 points to plus 2.5 points and comfortably includes zero. On that evidence alone there is no established regression at all.

The frozen 400-item set exists for exactly this. Re-running it under both model versions means every item is graded twice, and the comparison becomes paired, so the items that both versions get right and the items both get wrong cancel out and only the disagreements carry information. Suppose 22 items went from correct to wrong and 6 went from wrong to correct.

import math
from scipy.stats import binomtest

n = 400
b, c = 22, 6                                        # correct -> wrong, wrong -> correct

diff = (c - b) / n                                  # -0.040
se = math.sqrt(b + c - (b - c) ** 2 / n) / n        # 0.0131
lo, hi = diff - 1.96 * se, diff + 1.96 * se

print(f"paired change {diff:+.1%}  95% CI [{lo:+.1%}, {hi:+.1%}]")
print(f"exact McNemar p = {binomtest(b, b + c, 0.5).pvalue:.4f}")

#=> paired change -4.0%  95% CI [-6.6%, -1.4%]
#=> exact McNemar p = 0.0037

The final line of that snippet runs an exact McNemar test, which asks one narrow question: is the split between the 22 items that got worse and the 6 that got better more lopsided than a fair coin tossed 28 times would produce? Its p-value of 0.0037 is the probability of seeing a split at least that uneven if the two model versions were genuinely equally good, and at that value the imbalance is not something to put down to luck. The same underlying effect that was invisible at ±6.7 points unpaired is a four-point drop with a ±2.6 point interval when measured on the same items, and the interval excludes zero. That is a real regression and you can now say so without hedging. The pairing is the whole difference, and it is worth keeping the frozen set alive for no other reason.

Two checks belong in the same two hours. Re-run the previous model version today, rather than comparing against the stored number from three weeks ago, because grader drift — the same raters growing steadily more or less lenient over the weeks between runs — along with a retrieval index rebuild or a prompt template change, can move the baseline on its own and you do not want to attribute someone else's change to the vendor. And look at which 22 items regressed, because a drop concentrated in two policy areas is a different product problem from one spread evenly, and the concentrated version is the one a single customer will experience as the feature falling over.

Then convert it into the units from the calibration exercise before writing anything. Four points at 3,000 answers a day is about 120 additional wrong answers a day, and at four minutes a correction that is eight person-hours a day landing on a team that was staffed for sixteen. That sentence will do more work in the disclosure than the interval will.

Three ways this can go#

The middle column is the one people forget exists, and it is the reason the quiet fix feels like the only alternative to a fire drill. Walking into a product owner's office at 08:20 with "quality has dropped and I don't know why yet" produces alarm without giving anyone anything to do with it, and after two or three of those, your reports get filtered. Two hours of measurement between detection and disclosure is not concealment; it is the difference between a report and a rumour, and the threshold for how long that gap may be is something to agree in advance rather than to judge in the moment.

The first column's cost is easy to underestimate because it is conditional. The quiet fix only hurts if somebody learns the sequence, and people mostly do, because the sequence is written down in places you do not control: the eval history, the commit that pins the version, the vendor's changelog with its date, the Slack thread where you asked a colleague about McNemar tests. The exposure is not that someone catches you lying, since you did not lie. It is that a colleague reconstructs a timeline in which you knew on Monday, fixed on Tuesday and mentioned it never, and from then on the question in their head when you present a number is what you decided not to include.

The decision is not yours to make#

The argument for disclosure that engineers find most persuasive is usually the one about trust, and it is real, but there is a more concrete reason that settles the question faster. Pinning the previous model version has a cost that lands outside your team. The newer version brought the larger context window that the product owner has already promised to a customer for November, and reverting it pushes that commitment. Absorbing the regression instead costs the support lead eight person-hours a day for however long the fix takes, on a team sized for sixteen. Neither of those trade-offs is yours to resolve, because you do not know what the November commitment is worth or whether the support team has two weeks of slack.

The quiet fix is a decision on both of those questions made unilaterally and silently, which is what makes it a governance problem rather than a communication one. Framed that way, the disclosure stops being a confession and becomes a routine escalation of a decision to the person whose decision it is, and that framing also changes how the note is written: it opens with the numbers and closes with a request for a call by a stated time, rather than opening with an apology.

The instinct to fix first is worth understanding rather than just resisting, because it is not usually cowardice. It comes from a reasonable model of attention, in which every problem raised consumes a budget of stakeholder confidence, and a problem that will be gone by Thursday is not worth spending that budget on. That model is right about deterministic bugs and wrong here for two reasons. The first is that you do not yet know the fix will hold, since the cause is unconfirmed and pinning a version is a workaround rather than an explanation. The second is that a probabilistic feature's quality is going to move again, repeatedly, for the life of the product, so the relationship you are managing is not one disclosure but a long series of them, and the value of the series depends entirely on whether the recipients believe they are seeing all of it.

There is a compounding effect that is easy to miss while it is accruing. A team that has reported three regressions with numbers and mitigations, two of which turned out to be smaller than first measured, gets a specific privilege: when they say a movement is noise, people accept it and go back to work. A team that has never reported a regression in eighteen months of running a probabilistic feature is not a team with a stable feature; it is a team whose reports carry no information, and whoever notices that first will start asking for the raw evaluation data.

Who hears it, when#

The rule for the recipient list is that anybody who would be embarrassed to learn this from a customer belongs on it. In practice that is the product owner named as escalation owner in the launch record, the support lead whose team absorbs the extra corrections, and — if the regression is concentrated somewhere users will notice — the people who talk to those users. It does not automatically include the whole company, and it does not include a customer-facing announcement for a drop that is invisible at the individual-user level, because a notice about a change nobody experienced creates the alarm you were trying to avoid.

The note itself is short and has a fixed shape, which removes most of the writing anxiety.

**Policy assistant: 4-point quality regression, model update 2025-10-14**

Metric: answer judged correct by 2 of 3 raters, frozen 400-question set.
Before (v2025-09-02): 92.5%. After (v2025-10-14): 88.5%.
Paired change -4.0 points, 95% CI [-6.6, -1.4]. Both runs today, same graders.
Concentrated in leave carry-over and contractor status; 14 of the 22 regressions
are in those two areas.

Impact at current volume: ~120 additional wrong answers per day, roughly
8 extra person-hours of correction against a team staffed for 16.

Cause: not confirmed. The timing matches the provider's point release; we have
not reproduced it against a controlled prompt yet.

Options:
1. Pin v2025-09-02. 40 min to ship, restores the number, blocks the new
   context-window improvements we wanted next month.
2. Keep the new version, add a review step on the two affected areas.
   ~3 person-hours/day of review, in place by Thursday.
3. Accept, monitor weekly. Not recommended while the cause is unknown.

Recommendation: option 1 now, option 2 as the fallback if pinning is blocked.
Need a decision by Wednesday 12:00 so we can measure a full week either way.

Naming the cause as unconfirmed is better than offering a plausible one you may have to withdraw. Model providers change serving stacks, batching behaviour and quantisation — the numeric precision at which the model's weights are stored and multiplied, lowered to buy memory and speed at some cost in fidelity — without any of it appearing in a version string, and an engineer who confidently blames the vendor on Monday and discovers on Thursday that a colleague changed the retrieval chunk size has spent credibility on a guess. "Timing matches the provider release, not confirmed" costs nothing and survives being wrong.

What this is not a licence for#

It is not a standing instruction to broadcast every movement in the number. A weekly rate that moves from 92.4% to 91.6% with an interval of ±2.5 points has not moved, and reporting it as a regression trains people to treat your reports as weather. The distinction between a line in the weekly report and a note that pings three people should be written into the launch record as a threshold — an effect that survives a paired re-measurement, exceeds some stated size, or lands on a published slice — so that the decision to escalate is made by a rule you wrote in a calm week rather than by you at 08:20 with your own interests in the room.

It also does not substitute for fixing the thing. A disclosure with a mitigation attached is a decision request, and if the decision is taken and then nothing changes for three weeks, the second disclosure lands on an audience that has already learned what the first one was worth. And it does not always work, because some regressions cannot be attributed, cannot be mitigated cheaply, and cannot be pinned away when the previous model version has been retired by the provider. In that case the honest note says the number is lower, the cause is external, the options are worse than they were, and here is the narrower scope we can still stand behind.

That last sentence is a different kind of conversation from a regression report. It is the case where the answer to a request is that the thing cannot be delivered at a rate anybody would accept, and where the useful move is naming the adjacent thing you can measure instead.