A canary can be too small to see the failure it watches for

The rollout plan for the new model tier had one line about validation: canary at 2% for three days, promote if no regression. The failure everyone was worried about was named in the same document — the assistant stating a policy detail confidently and incorrectly, which support had been tracking manually and estimated at roughly 1 in 500 requests on the current configuration. The concern was that the new model might make it worse. The plan's justification for 2% was that it comes to 6,000 requests, and 6,000 of anything sounds like a sample you can trust.

Six thousand is the wrong number to look at. The quantity that determines whether the canary can see anything is how many failures it expects to contain, and at a base rate of 1 in 500, 6,000 requests contain about twelve. If the new model doubles the rate, the canary contains about twenty-four. The question is whether twelve and twenty-four are far enough apart to tell apart, given that both are counts of a rare event and will vary run to run.

Twelve against twenty-four#

Counts of independent rare events follow a Poisson distribution, and the standard deviation of a Poisson count is the square root of its mean, so a control expectation of 12 has a spread of about 3.5 and a doubled rate of 24 has a spread of about 4.9. Deciding on a threshold and computing the power — the probability that the canary raises an alarm in the world where the rate really has doubled — is a few lines:

from math import exp, lgamma, log

def pmf(k: int, lam: float) -> float:
    return exp(-lam + k * log(lam) - lgamma(k + 1))

def sf(k: int, lam: float) -> float:          # P(X >= k)
    upper = int(lam + 40 * lam**0.5) + 200
    return sum(pmf(i, lam) for i in range(k, upper))

def alarm_threshold(lam0: float, alpha: float = 0.05) -> int:
    k = int(lam0)
    while sf(k, lam0) > alpha:
        k += 1
    return k

t = alarm_threshold(12.0)            # 19
print(t, sf(t, 24.0), sf(t, 18.0))   # 19  0.872  0.438

Under perfect counting the picture is less hopeless than the informal version of the argument suggests, and it is worth being precise rather than rhetorical. Nineteen or more failures out of 6,000 happens 3.7% of the time when nothing has changed, so 19 is a reasonable alarm threshold, and a true doubling produces at least 19 about 87% of the time. That is not a canary that cannot see a doubling; it is a canary that misses one doubling in eight, which is a worse guarantee than most people think they are buying but is not nothing.

The same threshold detects a 50% increase — from 1-in-500 to 1-in-333 — 44% of the time. A change that makes your worst failure half again as common passes this canary on a coin flip. And with only twelve expected events, the confidence interval around the observed count is wide enough that "no regression detected" is consistent with the rate having moved anywhere between roughly half and one and three quarters of what it was.

The counting is where it actually falls apart#

The 87% figure assumes you observe every failure and none that are not failures. Nothing in a production canary works that way, because the failure in question — a confidently stated policy detail that is wrong — is not something the service can detect from its own output. It is detected by a judge model — a second model prompted to read the answer and grade it — or by a retrieval-grounded consistency checker, human graders on a sample, or downstream complaints, and each of those changes the arithmetic.

Take a judge with a 2% false-positive rate, which is a good judge on a task like this. Across 6,000 canary requests it produces about 120 spurious flags on top of the 12 real ones, so the observed count is around 132 in the control condition and around 144 under a doubling. Running the same threshold calculation on those numbers gives an alarm at 152 and a power of 26%. The signal is still there and the noise floor has risen underneath it, so three quarters of the time the doubling passes.

Human grading on a sample is worse in a different way. Grading 300 of the 6,000 canary requests is a substantial amount of human work, and it expects 0.6 failures in control against 1.2 under the doubling. There is no threshold on those numbers that means anything; the most sensitive rule available fires 12% of the time when the rate has genuinely doubled.

Complaints are the sparsest signal of all. If one in eight bad answers produces a complaint, and complaints arrive over the following week, then a three-day canary expects about 1.5 complaints in control and 3 under the doubling, mostly arriving after the promotion decision was made.

Budgets that are honest about what they can see#

The table below assumes perfect counting, so every row is an upper bound on what that canary can achieve, and any real detection pipeline does worse. The service is 100,000 requests a day and the base rate is 1 in 500.

Reading the table honestly leads somewhere uncomfortable: catching a modest increase in a 1-in-500 failure requires exposing a large fraction of your traffic for a week, which is not a canary in any meaningful sense. That is the correct conclusion, and the response is to stop asking the canary to do this job. A canary detects gross, fast failures on signals that are dense — provider error rate, timeout rate, p95 latency, cost per request, output-format validity, refusal rate, tool-call failure rate, truncation rate. Those have base rates in the percent range rather than the fifth-of-a-percent range, so a 2% canary carries hundreds of events and detects a 20% shift comfortably. Rare quality failures are the job of the offline evaluation suite from topic 3.16, where you can concentrate 2,000 adversarial cases on precisely the failure mode instead of waiting for it to occur naturally, and of a long soak, meaning weeks of traffic at a large share, once the gross failures have been ruled out.

The cohort is a sample or it is nothing#

The second failure has nothing to do with size. A team running a 5% canary implemented the selection as "the first requests arriving in each minute" — cheap to implement with a per-minute counter, no hashing, no user identity required. Three days passed with no movement on any metric, the change went to 100%, and quality complaints started the same afternoon.

The first requests of a minute are not a random 5% of traffic. They are whatever is scheduled, retried or automated, because those arrive on clock boundaries while human traffic arrives continuously. When the team went back and characterised the cohort, 71% of it came from three integration accounts running nightly syncs, 88% of it was single-turn, and the median input was 340 tokens against a fleet median of 1,900. The regression was in multi-turn conversations where the assistant had to reconcile earlier context, which the cohort barely contained. The canary measured a real thing accurately; it measured a population that was not the one being promoted to.

Selection should be a hash of a stable unit with a per-experiment salt, the same mechanism chapter 1 used for flag targeting:

def in_canary(unit_id: str, salt: str, basis_points: int) -> bool:
    digest = hashlib.sha256(f"{salt}:{unit_id}".encode()).digest()
    return int.from_bytes(digest[:4], "big") % 10_000 < basis_points

Which unit to hash is a real decision. Hashing the request id gives the best-balanced cohort and lets a single user see both configurations across consecutive turns, which is unacceptable for a conversational product. Hashing the conversation id keeps a conversation internally consistent and is usually right for an assistant. Hashing the account id is the coarsest, and on traffic with heavy concentration it is dangerous, because with one account contributing 15% of requests a 5% account-level cohort is one coin flip away from being mostly that account.

Before trusting a cohort, compare its composition against the fleet on the dimensions that plausibly interact with the change: share of traffic from the top five accounts, turn-count distribution, input-token percentiles, locale mix, channel mix, hour-of-day profile, and the proportion of automated versus interactive traffic. A cohort that is 71% from three accounts fails this check in one line. Running the selection for a day with both arms on the old configuration — an A/A period — is the stronger version, because it shows whether the metrics themselves agree before any change is introduced, and disagreement in A/A means every later comparison is measuring cohort composition.

The guardrail that fires without you#

Detection of the rare quality failure is a decision a human makes at the end of the window. Gross failures should abort the canary automatically, which is the job of a guardrail: a rule that watches one metric on a short window and halts the rollout on its own, with no human in the path. Its thresholds have to be expressed against the concurrent control arm rather than as absolute values:

guardrails:
  - metric: provider_error_rate      # 5xx + timeouts
    compare: canary_vs_control
    window: 10m
    min_events: 200                  # do not fire on thin traffic
    abort_if: ratio > 1.5
  - metric: p95_latency_ms
    compare: canary_vs_control
    window: 10m
    abort_if: ratio > 1.3
  - metric: cost_per_request_usd
    compare: canary_vs_control
    window: 60m
    abort_if: ratio > 1.25
on_abort:
  action: set_flag
  flag: assistant_model
  value: last_known_good
  notify: ["#inc-assistant", "oncall-assistant"]

Comparing against the concurrent control is what makes the rule survive contact with reality. An absolute threshold on error rate fires for everybody during a provider incident, including canaries that are behaving perfectly, and gets muted after the second false page. A comparison against last week's baseline drifts with traffic mix, seasonality and unrelated deploys. The control arm is running on the same infrastructure, in the same minutes, against the same traffic mix, and the ratio between the arms isolates the change. The min_events floor is what stops the ratio from firing on two errors against one at three in the morning.

The abort action writes a flag value, which is the runtime path from chapter 3 with a program on the end of it instead of a person, and it is worth noting how little else it could be. An automated guardrail that has to open a pull request is not a guardrail.

A canary and an experiment get confused because they share the mechanism from chapters 1 and 2 — a flag, a hashed cohort, two arms — and they answer different questions. A canary asks whether the candidate is broken, on signals that are dense and fast, with a bias towards aborting; it is sized for safety and it is fine for it to be underpowered on subtle quality differences because it is not trying to measure them. An experiment asks whether the candidate is better, needs enough traffic in each arm to resolve the effect it cares about, and has to run for its planned duration to avoid the peeking problem, where checking the result repeatedly and stopping at the first favourable moment inflates the false-positive rate well past the one the test was designed for. Running one and reporting the other is the common error: a three-day canary at 2% that shows a 4.8-point improvement has shown nothing of the kind, because at that sample the interval around 4.8 points spans zero comfortably.

The ramp, meaning the schedule by which the exposed share is widened over time, is what connects the two. A canary at 2% for a few hours rules out the gross failures, at which point the reason to stay at 2% has expired and staying there only accumulates exposure without accumulating evidence. Going 2% for four hours, 10% for a day, 50% for two days is more informative than 2% for three days at every point after the fourth hour, and it has a smaller total number of affected requests if something goes wrong late, because the guardrails have already been watching at each step. The doubling of a 1-in-500 harm gets caught at the 50% stage or in offline evaluation, and not at 2% under any schedule.

What a canary cannot do at all is tell you how the candidate behaves on inputs the canary cohort never sends, and it cannot show a candidate's output to nobody. Every request in the canary arm is a real answer to a real user, which is the source of its value and the reason it cannot be run on a configuration you have doubts about. The next chapter is about the technique that removes the user from the loop entirely, and about the ways it turns out not to have removed the side effects.