A launch criterion without a population is not measurable

Six weeks after launch, in a room with the head of customer operations and the product owner, two people are disagreeing about whether the feature met its launch criteria and neither of them can be shown to be wrong. One has three screenshots of the assistant giving a bad answer about parental leave carry-over. The other has been using it daily and has not seen a bad answer in a fortnight. The criterion in the document says the assistant answers policy questions correctly. Three screenshots do not contradict it and a fortnight of good experience does not confirm it, so the meeting spends forty minutes converting anecdotes into volume and ends with someone saying they will "keep an eye on it".

That criterion cannot be passed and cannot be failed, which means it was never a criterion. Four things are missing from it and each one is load-bearing.

The four things that are missing#

The population is the one that matters most, because without it the other three cannot be defined at all. "Policy questions" is not a set anybody can draw from. The set of questions employees actually type into an HR assistant is dominated by a dozen topics, contains a long tail of oddities, and shifts sharply in January and at the end of the leave year. A rate over that population is a different number from a rate over the fifty questions the policy team thinks are important, and a criterion that does not say which one it means will be interpreted as whichever one is convenient by whoever is under pressure.

Here is the same criterion written so it can be lost:

On a stratified sample of 400 questions drawn from Q3 helpdesk tickets tagged policy, sampled proportionally across the eleven policy areas with a floor of 30 questions per area, at least 92% of the assistant's answers are judged correct by two of three trained raters against the published rubric, with the exact binomial interval reported alongside. Re-measured every Monday on a fresh sample of 150 answers taken from the previous week's live traffic, with the four-week rolling rate published on the quality page.

That is long, and the length is the point: everything in it is a decision somebody was going to have to make eventually, made now, in writing, while nobody is angry. It also has a property the original lacks, which is that it can be failed. If the Monday measurement comes back at 88% with an interval that excludes 92%, the feature has missed its criterion and there is nothing to argue about.

Why the original sentence looked adequate#

Acceptance criteria in the form the industry has used for twenty years are written as single statements of behaviour, and they work because the systems they describe are reproducible. Given a customer with an expired card, when they submit the payment form, then they see the card-expired message is a complete specification: it names one input, one output, and an implicit rate of 100%, and the implicit rate is fine because a correctly implemented branch takes itself every time. Nobody writes "then they see the card-expired message 99.9% of the time", and if somebody did, the reviewer would ask what was wrong with the code.

The stakeholder writing the assistant answers policy questions correctly is following that form exactly. It has a subject, a behaviour, and an implied always, and it looks like every other line in the document. The form breaks here because two of its assumptions have gone: a single input no longer determines a single output, and the space of inputs is no longer enumerable by the person writing the criterion. Payment forms have a finite set of states somebody can list; the set of questions an employee might ask about parental leave does not have edges, which is why a criterion has to name a sample of it rather than the whole thing.

What replaces the old form is closer to a service level objective (SLO) than to an acceptance test, and saying that out loud helps, because SLOs are a vocabulary most organisations already have. An availability target is stated as a rate over a population of requests in a window, with an error budget — the quantity of failure the target permits before somebody is obliged to act — and a named owner of the decision to spend it, and nobody finds that strange. The probabilistic quality criterion has the same skeleton with correctness in place of availability, and framing it that way moves the conversation off the ground of "engineering says the feature might be wrong" and onto ground where a rate with a window is a normal way to describe a running service. The one difference worth flagging when you use the analogy is that availability is measured by instruments and correctness is measured by judgement, which is why the method and the rubric have to be named where an SLO would simply point at a probe.

The fix that is not a fix#

The reflex when the vagueness is pointed out is to add a percentage: the assistant answers policy questions correctly 95% of the time. That version is worse in one specific way, because it now looks measurable and still is not. It names no population, so 95% of what remains open. It names no judge, so "correctly" is still whatever the person reading it thinks correct means, and correctness for a policy answer is genuinely contested — an answer that is accurate but omits the exception that applies to part-time staff is correct to one rater and wrong to another. It names no window, so nobody knows whether a bad week is a breach. And the 95% itself, in the case that motivates this, came from twenty attempts at a demo, so it is a number with a twenty-five point interval being written into a contract with the business.

The addition of a percentage also creates a specific failure mode later. When the first measurement arrives at 93% on a well-constructed sample, the conversation is now about a missed target rather than about whether 93% supports the workflow, and the honest answer — that 95% was never measured and the population it referred to never existed — arrives at the worst possible moment, which is when somebody feels let down.

Getting the rewrite accepted#

Rewriting the sentence takes ten minutes. Getting the person who wrote the original to prefer your version takes a specific argument, and the argument that does not work is the one about rigour. Telling a stakeholder that their criterion is not statistically well-formed invites the reply that engineering is overcomplicating things, and they are half right, because so far you have only described a cost to them and no benefit.

The argument that works is that the vague version leaves them with nothing either. It goes as a pair of questions asked in order, and asked genuinely rather than rhetorically. The first: suppose in week three somebody forwards you three wrong answers — under this criterion, has the feature failed? They cannot answer, because three wrong answers is compatible with the system being excellent and compatible with it being unusable. The second: suppose I come to you and say it is working fine — how would you check? They cannot answer that either, which is the more uncomfortable of the two, because it means the criterion they wrote gives them no leverage over you at all. It reads like a demand and functions as a blank cheque.

Once that has landed, the offer is straightforward. The rewrite is a commitment they can hold you to on a schedule, with a number that will sometimes be embarrassing, published somewhere they can read without asking. Most stakeholders take that trade quickly, because the thing they actually wanted was not the word "correctly" but the assurance that somebody was accountable for quality after launch. The population is the part to negotiate rather than dictate: ask them which questions would embarrass the company if they were answered wrong, and put a floor of thirty examples under each of those areas so the criterion cannot be passed by being good at the easy ones. A criterion whose population you chose alone is a criterion they will stop believing the first time it disagrees with their experience.

Publish the failure path with the number#

A rate is half of what should be published before launch. The other half is what happens to the failures, because the expectation that forms in users' heads is set by their first bad experience and by whether anything visible happened afterwards. A user who hits a wrong answer, reports it, and hears nothing concludes the system is unmanaged; the same user who hits a wrong answer and gets a reply naming the category and saying it is on the fix list concludes the system has owners.

The escalation owner is a person with a name, not a team, and the decision rights in that last box need to be written down before the situation arises. Deciding under pressure who is allowed to turn a feature off is how a two-hour problem becomes a two-day one.

All of this fits on one page, and keeping it in a machine-readable form next to the service means it can be linked from the release notes, the runbook and the support macro rather than living in a document nobody can find.

feature: hr-policy-assistant
population:
  source: helpdesk tickets tagged 'policy', rolling 90 days
  sampling: stratified by policy area, min 30 per area
  size: 400
target:
  metric: answer judged correct by 2 of 3 raters
  rubric: /quality/rubrics/policy-answer-v3.md
  rate: 0.92
  report: exact binomial 95% interval alongside the point estimate
window:
  baseline: frozen 400-item set, re-run on every model or prompt change
  live: 150 sampled answers per week, four-week rolling rate published
failure_path:
  user_control: "Report this answer" on every response
  triage_sla: 1 business day
  owner: support lead (rota published on the quality page)
escalation:
  triggers: [rolling rate < 0.92, single category > 20 cases/week]
  owner: Dana Whitfield, product owner
  rights: may narrow scope or disable the feature without further approval

If the organisation has a risk function that will want this in its own vocabulary, the AI Governance Framework published by FINOS, the Fintech Open Source Foundation, has an entry on non-deterministic behaviour catalogued as RI-6, and it supplies wording and a mitigation list that auditors already recognise, which saves inventing risk language from scratch and saves arguing about the invented version.

What a measurable criterion does not do#

It does not make the feature better. A well-specified 92% is still 92%, and the whole of the previous chapters' arithmetic about counts and workload still applies to it. It is also gameable in exactly one direction, which is the choice of population — a sample drawn from the questions the system handles well will pass any threshold you like, which is why the person who will be unhappy has to agree the population before the first measurement rather than after.

The criterion is also a statement about a period that has ended. A frozen 400-item benchmark answers whether the system is as good as it was on the day the benchmark was built, and populations drift: a policy change in April makes a quarter of the Q3 sample obsolete, and a new customer segment sends questions in a register the sample never contained. That is why the rewrite carries both a frozen set and a weekly live sample; the frozen set detects changes in the system and the live sample detects changes in the world, and neither substitutes for the other.

Committing to that weekly measurement has a consequence worth sitting with. Sooner or later the Monday number will be materially worse than the previous Monday's, and because you built the measurement, you will be the first person in the company to know.