Bypassed gates and stale greens

A quality eval wired to every pull request is the arrangement most teams build first, and it is the strictest-looking option on the table: nothing merges without a measured score. What happens to it next is predictable enough to be worth writing down in advance, because the sequence is always the same. The job is slow and it costs money; it sits between an engineer and their branch; and one afternoon somebody adds a flag that makes it non-blocking, with a one-line commit message and no review comments.

This chapter is about that end state and how to get out of it, including the part that matters more than it sounds: auditing the run history a bypassed job leaves behind, before anybody treats those numbers as a baseline.

Naming the protection precisely#

The question a postmortem should ask is what quality protection the job currently provides, and the answer is none, in a specific sense worth spelling out rather than asserting.

      - name: eval gate
        continue-on-error: true      # added 2026-03-04, "unblocking the team"
        run: uv run evals run evals/order_assist/suite.yaml --sha "$GITHUB_SHA"

A check protects something only if a failure changes what happens next. This one cannot fail in a way that changes anything. continue-on-error means a red result still reports the job as successful to the branch protection rules — the repository setting listing which checks must report success before the merge button is enabled — so the merge proceeds identically whether the eval scored 82 or 41. The output goes to a log tab that the run summary does not surface. There is no alert wired to the result, no record of it outside the continuous integration provider's 90-day log retention, and no downstream job that consults it.

Following the flag through the machinery turns that from an assertion into something checkable. A workflow step ends with an exit code, and a non-zero code is how a program tells whatever started it that it failed. The runner's default is to pass that upward: a failed step fails its job, and the job's outcome is written into a check run, which is the record the provider publishes for that job through its API. Branch protection reads one field on that record, the conclusion. It never reads the exit code, it never opens the log, and it has no access to the eval's score at any point, because the score was only ever printed.

continue-on-error severs exactly one link in that chain: the step's exit code stops setting the job's outcome. Every link after the cut carries on working correctly, on the value it was handed. The check run is published, the conclusion says the job succeeded, branch protection believes it, and the merge button unlocks. That is why the failure is so quiet. Nothing is broken and no component is lying — one input to a chain of correct components has been replaced, and the score that would have contradicted the outcome was never in the chain to begin with.

What remains is worse than nothing in one respect, because the job's presence does active work in the wrong direction. The repository README carries a green badge. An internal compliance document, written when the gate was introduced, states that answer quality is verified on every pull request before merge, and that sentence is now false in a way nobody has noticed. A deleted job would at least have left an honest gap; this one leaves a claim of coverage, and the provider bill is being spent to maintain the claim.

The arithmetic that produced the bypass#

The obvious response is to delete the line, restore branch protection, and ask the team to be disciplined about quality. That reproduces exactly the conditions that produced the bypass, and it will be bypassed faster the second time, because everybody now knows the flag exists and knows that adding it went unchallenged for six weeks.

The arithmetic of why the original arrangement was unsurvivable is what turns a discipline conversation into a design conversation. Take a team where a pull request receives 4.7 pushes on average before it merges — a first push, a self-review fix, two rounds of review feedback, a rebase — and every push retriggers a 38-minute, $9 job.

The commit that added the flag says what it was trading against, in three words: "unblocking the team." Thirty-eight minutes and about nine dollars is the price of one push, and a push is not the unit anybody works in — the unit is a pull request, which takes 4.7 of them. Multiplying through gives the second column: 38 minutes at 4.7 pushes each is a little over three hours of eval time before a branch merges, and $9 at 4.7 pushes is a little over forty dollars. Multiplying again by the thirty pull requests a week gives the third column. Thirty lots of three hours is about ninety hours sitting in a queue, and thirty lots of forty-odd dollars is a little over twelve hundred.

The third column arrives from the other direction too, and doing it both ways is a cheap check that the trigger has been counted correctly. Thirty pull requests at 4.7 pushes each is about a hundred and forty runs a week, and a hundred and forty runs at $9 each is the same twelve hundred and change. The two routes agree because they are the same three numbers multiplied in a different order. Had they disagreed, the error would have been in the count of triggers — a job that also fires on the merge commit, or one that reruns when a label changes — rather than in the arithmetic.

The author pays that cost, and the author also holds the flag. A process putting an unavoidable 38-minute delay in front of every push has already decided its own outcome; the only variable is which engineer writes the bypass and in which week. Treating that as a willpower problem is how a team ends up with a gate that is technically mandatory and practically off, which is strictly worse than a gate that is honestly advisory, because the advisory one does not lie in the compliance document.

The asymmetry underneath that is what makes the outcome structural rather than merely likely. The cost of the gate is concentrated, immediate and personal: one named engineer, waiting, now, on a branch they want to close. The benefit is diffuse, deferred and collective: a regression that might have shipped, to customers who would have noticed some weeks later, avoided on behalf of everybody. Nothing in the repository weighs the two against each other, and adding the flag requires the consent of nobody who holds the second half.

The replacement is the merge-triggered arrangement: deterministic checks blocking every push, the frozen-sample eval running on merge to main with its number recorded against the commit sha, alerts to the owning team on an absolute threshold and on a drop from the trailing mean, and a release job refusing to promote a build whose number is missing or below the line. That moves the wait off the author entirely and cuts weekly spend from about $1,270 to about $180, while the time from a bad merge to a named commit goes from "never, because nobody reads it" to 25 minutes.

Most of that fall in spend is the count of runs. One run per merge instead of one run per push takes about a hundred and forty runs a week down to thirty, and the wait that survives belongs to nobody in particular, because it happens after the branch is closed on a commit that already exists. The gate has not become weaker; the person paying for it has stopped being the person who can switch it off.

Auditing the history before trusting any number in it#

Six weeks of run history exist, and somebody is going to want to use it as a baseline. A non-blocking job degrades silently in ways a blocking job cannot, because the failure modes that would normally announce themselves as a red build now announce themselves as nothing at all. Five things are worth checking.

Whether it ran at all is the first, and path filters are the usual culprit — the paths and paths-ignore keys that tell the provider to skip a workflow entirely when a change touches none of the files they name:

on:
  pull_request:
    paths-ignore:
      - '**/*.md'
      - 'prompts/**'     # added 2026-03-19 "docs and prompt copy don't need evals"

Prompt templates live under prompts/. Every pull request that changed only a prompt — which is most of the pull requests that could plausibly move answer quality — skipped the eval entirely, and a skipped job in most providers reports as neutral or successful rather than as absent. Counting runs against the merge log is how this gets found:

-- merges to main in the window, and whether an eval run exists for each
select
  m.sha,
  m.merged_at,
  m.author,
  r.score,
  r.cases_scored,
  r.cases_errored,
  r.suite_version
from merges m
left join eval_runs r on r.sha = m.sha
where m.merged_at >= now() - interval '6 weeks'
order by m.merged_at desc;
-- 143 merges, 96 with a run, 47 with none  -> 33% of merges unmeasured

A skipped job reporting as a pass rather than as an absence looks like a provider bug and is a deliberate choice, and knowing why it was made is what stops anybody waiting for it to be fixed. A required check has to report something before the merge button will unlock. If a workflow that never ran reported nothing at all, the check would sit pending forever and the pull request could never merge — which would mean that adding a path filter to any required workflow permanently deadlocks every pull request that filter excludes. Providers avoid that by resolving a skipped workflow to a non-failing conclusion. The consequence is that one field now carries two different states under one value: ran and passed, and never ran. Branch protection cannot tell them apart because nothing in the field distinguishes them, and neither can anybody reading the list of green checks on the pull request.

The comment on the last line of that query is the whole finding. The 96 merges with a run and the 47 with none add back to the 143 merges in the window, and 47 of 143 is 33% — a third of everything that reached main in six weeks carries no measurement at all. The left join is what makes that visible. An inner join would have returned the 96 rows that exist, reported nothing about the gap, and produced a result set every bit as convincing as this one. That is the same shape of error as the score aggregation two checks further down, which averages only the cases that came back: both quietly answer a question about the rows that happen to be there.

Whether the flag masked a crash rather than a failing score is the second. continue-on-error swallows a non-zero exit for any reason, so an expired API key, an out-of-memory kill on a large case, or a schema change in the results writer all look identical to a low score from the outside, and all of them look identical to a pass from branch protection. In these run records, 11 of the 96 runs had cases_scored under 50 against a set of 400 cases, which is a crashed run reporting a score computed over whatever finished.

Under 50 against 400 is at most an eighth of the suite, and the number that comes out of an eighth of a suite is written to the same field, under the same name, as a number computed over all of it. Nothing in the score marks the difference. A mean over fifty cases and a mean over four hundred are both plausible-looking values in the same range, and the smaller one is noisier in a way no consumer of the field can see. That is why the case count has to be stored beside the score rather than eyeballed at the time: whoever reads the series in six weeks will not have the run to inspect.

Whether errored cases were dropped or counted is the third, and it quietly inflates a score. The runner's aggregation was mean(score for case in cases if case.score is not None), so a case whose generation call timed out contributed nothing at all rather than a zero. Timeouts correlate with long, complex questions, which are also the questions an assistant handles worst, so dropping them biases the score upward — re-scoring the errored cases as failures moved this suite's trailing mean from 80.6 to 76.9.

The denominator is where that goes wrong, and it is worth writing the two means out before comparing them. A mean over the whole set is the sum of the scores divided by the number of cases in the set. The comprehension above divides by the number of cases that returned a score, because a case filtered out of a generator expression is gone from the count as well as from the sum. Those are two different quantities answering two different questions. The first asks how well the assistant did on this suite. The second asks how well it did on the cases it managed to answer, which is a question nobody asked and which flatters the system by construction whenever failing to answer is itself a failure.

How much it flatters follows from the two trailing means in one step of algebra. Call the sum of the scores that came back S, the number of cases in the set N, and the number that errored k. The reported mean is S divided by N minus k. Re-scoring the errored cases as failures adds k zeros to the sum, which leaves the sum at S, and restores those cases to the count, which puts the denominator back to N — so the corrected mean is S divided by N. Dividing the corrected mean by the reported one cancels S and leaves N minus k over N, which is the fraction of the set that survived. Here the corrected mean is 76.9 and the reported one 80.6, so that fraction is 76.9 divided by 80.6, a little over 0.95. About one case in twenty went missing, and it bought nearly four points of score.

That step assumed the missing cases would all have scored zero, which is the conservative reading rather than a certain one; some of them might have passed. The assumption it does not make is the one that matters, which is that the missing cases resemble the ones that stayed. They do not. Dropping cases at random would leave the estimate centred where it was and only widen the uncertainty around it, because a randomly chosen subset of a set has the same expected mean as the set. Dropping cases that are correlated with the outcome moves the centre instead, and moves it in a direction you can predict before looking: timeouts land on the long, complex questions, those are the questions the assistant handles worst, so what is removed is the low-scoring end and the mean rises.

Whether the instrument changed underneath the series is the fourth. The case file had been edited twice in the window, once to remove 18 cases somebody considered unfair and once to add 30 new ones from a customer complaint, and the judge prompt had a paragraph added about tone. None of those were recorded against the runs, so the series contains at least three incomparable segments, and the apparent 4-point improvement in week four is the day the 18 hardest cases were removed.

Calling that a change in the instrument rather than a change in the score is the point of the check. A pass rate is a statement about one specific set of cases graded by one specific rubric, so two runs are comparable only when the set and the rubric are the same on both sides of the comparison. Removing 18 cases somebody considered unfair does not make the assistant better at anything. It removes 18 rows from the denominator, and it removes the particular rows that were pulling the mean down, because a case tends to get called unfair after the system has failed it several times. The 4-point improvement is arithmetic performed correctly on a different population. Recording the case-file version and the judge-prompt version against every run is what makes that visible, and it costs two fields.

Whether anything was cached is the fifth, and it cuts the other way. A generation cache keyed on the prompt hash makes reruns cheap and also makes the eval blind to a model change, since a provider-side model update produces different outputs a cache will never request. A team with no cache pays the full bill and keeps the sensitivity.

Telling a fresh number from a stale green#

The lesson people draw from a bypassed gate is that it should have been stricter. The more useful lesson is that non-blocking and invisible are different properties which happened to coincide. Routing — who is alerted, on what condition, with what in the message — is a solved part of the design. What the merge-triggered arrangement additionally needs, and what a bypassed job conspicuously lacks, is any way for a consumer of the number to tell a fresh result from a stale one, because a green from a job that stopped running looks exactly like a green from a job that ran and passed.

The release job therefore reads several fields rather than one:

def promotion_check(sha: str, run: EvalRun | None) -> Decision:
    if run is None:
        return Decision.refuse(f"no eval run recorded for {sha}")
    if run.suite_version != CURRENT_SUITE_VERSION:
        return Decision.refuse(
            f"run used suite v{run.suite_version}, current is v{CURRENT_SUITE_VERSION}"
        )
    if run.cases_scored < 0.95 * run.cases_total:
        return Decision.refuse(
            f"partial run: {run.cases_scored}/{run.cases_total} cases scored"
        )
    if run.finished_at < utcnow() - timedelta(hours=36):
        return Decision.refuse(f"stale: scored {run.finished_at:%Y-%m-%d %H:%M}")
    if run.score < BLOCK_RELEASE_BELOW:
        return Decision.refuse(f"score {run.score:.1f} below {BLOCK_RELEASE_BELOW}")
    return Decision.allow()

The first branch in that function is where the design lives. A promotion check that only compared a score against a threshold would still have to do something when there is no score, and there are two available somethings: refuse, or allow. Allowing is the tempting default, because a missing record usually means an infrastructure hiccup rather than a bad build, and refusing on it turns every eval outage into a release outage. It is also the choice that makes the whole arrangement unenforceable. The cheapest way to pass a gate that allows on absence is to stop producing the number, which is exactly what a job that quietly stopped running does, without anybody deciding to do it. Refusing on absence makes the presence of the number part of what is being checked, and that is the difference between a gate that fails closed and one that fails open.

Absence, partiality, staleness and suite drift each refuse promotion with a distinct message, so the failure a bypassed job hides — the job silently not running — becomes the loudest thing in the release log rather than the quietest. A second monitor on the eval infrastructure itself, alerting when no run has been recorded for main in 24 hours, closes the remaining gap where the whole workflow is disabled.

The other ways a gate goes quiet#

An audit that greps for continue-on-error will miss most of the family. A job can be removed from the required-status-check list, which lives in repository settings rather than in the repository, so the workflow file still looks like a gate and the diff history shows nothing. A required check that gets renamed — eval gate to eval-gate — stops matching the protection rule and silently becomes advisory, which is a favourite because it happens during an innocent workflow cleanup. A step guarded by if: github.actor != 'dependabot[bot]' grows a second clause, then a third, until it covers most merges.

Each of those produces the same end state, so the audit worth running is not a search for a string but a comparison of the required-status-check list against the jobs that actually ran on the last fifty merges. That is a short script against the provider's API, and it is the only version of this check that cannot be fooled by where the configuration lives.

Why that comparison is the only reliable one comes down to how the rule is stored. A branch protection rule holds a list of strings, in repository settings, and matches them against the names of the check runs a workflow reports. A name is a matching key rather than an identity, so renaming a job is a deletion and a creation as far as the rule is concerned, and neither of those events is a change to any file. The settings do not live in the repository either, so the required list can be edited without producing a diff, a review or a commit. Every member of this family reduces to the same property: the set of checks that must pass and the set of checks that ran are two lists maintained in different places, and nothing compares them unless somebody writes the thing that does.

A variant of the same problem lives inside the eval itself, where no workflow audit reaches. A judge model returning malformed scores, a rubric file that falls back to a default, a case file truncated by a bad merge: each produces a number that looks like a number. The defences are the ones already in the promotion check — case counts, suite version, freshness — applied to the eval's own output rather than to the job status.

What a bypassed gate teaches, in the end, is that the flag was a symptom. The cause was a slow, expensive check placed where its cost fell on an individual with a branch open, and any arrangement with that property converges on the same place regardless of who is on the team. Moving the check off the critical path removes the incentive; recording its output durably, by commit, with case counts and a suite version, removes the ability of a silent failure to look like a pass; and refusing promotion on absence rather than only on a low score turns the quietest failure into the loudest.

None of it settles what the threshold should be. A number picked because it sat four points below wherever the score happened to be is a number that will either fire on noise or miss real regressions, and which of those it does depends on a number nobody in this story has calculated. That is the next chapter: how small a regression the gate has to be able to see, and what that requirement does to the number of cases.