Merge gates and scheduled suites

The decision this chapter is about fits on one line of repository configuration: which checks a branch protection rule requires before the merge button does anything. Everything else here is plumbing that follows from it. Getting that line wrong in the generous direction — requiring every test the repository has, including the ones that call a provider — produces a pull-request pipeline of twenty minutes with a real failure rate that has nothing to do with the change under review, and a team that has learned to treat red as "re-run".

Getting it wrong in the other direction is the more tempting mistake, because the obvious repair for a slow, expensive, flaky build is to delete the tests causing it. Those tests are usually the only code in the repository that has ever touched a provider, and everything the previous chapters left open — whether the cassettes still match reality, whether retry-after is honoured, whether the adapter's mapping of stop_reason is complete, whether the tool schemas are accepted at all — is answerable only by code that makes a real call. This chapter sorts the tests into four groups by what a failure means and who has to react to it, then makes the sort enforceable rather than conventional.

What a slow gate costs beyond the machine time#

Take a team of six merging roughly forty pull requests a week, each triggering at least one run of a twenty-two-minute build, a sixth of them triggering two. That is around 47 runs a week, and the part that hurts is not the machine time but the position of the wait in a person's day: at 22 minutes with a one-in-six failure rate, nobody waits, so they switch tasks, come back forty minutes later, discover a failure that has nothing to do with their change, re-run, and come back again. The measurable version of this is the time from opening a pull request to merging it, which on teams in this situation typically sits between four and eight hours for changes that took twenty minutes to write.

The run count in that paragraph is worth deriving, because it is the figure the machine-time argument gets made from and it is smaller than the argument needs. Forty pull requests each trigger a run, and a sixth of forty is between six and seven more, so the total is around forty-seven runs in a week. At twenty-two minutes each that is a little over seventeen hours of build time, which on any hosted runner is a bill somebody could pay out of a stationery budget. The machine time is genuinely not the problem, and pointing at it is how these conversations get lost: a team that argues the case on infrastructure cost will be told, correctly, that the cost is trivial.

The second cost is what a one-in-six red rate does to attention. A gate that fails for reasons the author did not cause teaches the author that red means "re-run", and that lesson applies to every check in the run, including the contract test that would have caught a missing required field. A flaky gate degrades the tests around it, which is why the API spend is usually the least of it.

The step that makes that spread inevitable is the shape of the report. A pipeline hands back one verdict for the whole run, and the author reads the verdict before reading anything under it. Distinguishing a flake from a real failure means opening the run, finding the failing test among hundreds, and reading its output — several minutes of work, against a re-run button that has resolved five failures in six. So the cheap action wins, and it wins uniformly, because the button does not know which check failed and neither does the person pressing it at that moment. A contract test that fires correctly is therefore dismissed by the same reflex that correctly dismisses an overloaded endpoint. The reflex is not carelessness; it is the rational response to a signal whose prior is five-to-one against meaning anything, and the only way to change the behaviour is to change the prior.

Four suites, four different jobs#

Sorting by what a failure means, and therefore by who has to react and how fast, produces four groups: the offline gate, the suite that contacts a provider, the evaluation, and a canary — a small share of live production traffic instrumented closely enough that a new failure shows up in its error rate within minutes of a deploy.

Each row of triggers follows from the row above it. A failure caused by your change should block your change; a failure caused by the provider changing has nothing to do with the pull request in front of it, and blocking on it means an engineer who wrote a CSS fix is now debugging a schema change at Anthropic.

Making the split real in configuration#

Marking is the easy part. In pytest, a marker on each provider-touching test and a default deselection in pyproject.toml is enough:

[tool.pytest.ini_options]
markers = ["provider: makes a real call to a model provider"]
addopts = "-m 'not provider' --disable-socket --allow-unix-socket"

--disable-socket, from pytest-socket, is what makes the split hold. Without it, the gate suite's offline property is a convention that erodes the first time somebody forgets a marker, and the erosion is silent because the test passes. With it, any test that opens a TCP connection fails immediately with a message naming the call site, so a missing marker is caught by the mechanism rather than by review.

How it manages that is worth knowing, because it decides what the ban does and does not cover. pytest-socket replaces the socket type in Python's standard library with one whose connect method raises instead of connecting, for the duration of the test session. Anything in the process that eventually reaches the standard library to open a TCP connection therefore raises — the SDK, an HTTP client, a fixture loader, a metrics agent — regardless of which layer it sits at, because they all bottom out in the same place. The Unix-socket exemption exists because that same code path is how a process talks to a local database or a container runtime, and banning those would fail tests that were never online in the first place.

Two things it cannot see are worth stating plainly. A test that reaches the network only on a branch this run did not take is not detected, because nothing attempted a connection; the ban catches what happens, not what could happen. And a test that shells out to a subprocess gets no protection at all, since the replacement lives in this process's standard library and the subprocess has its own. Neither is a reason to skip it. Both are reasons not to describe the gate as provably offline when what it is, is offline for every path anybody has run.

The second enforcement is at the credential level, and it is worth doing even though it overlaps:

# .github/workflows/gate.yml
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e '.[test]'
      - run: pytest              # no secrets in this job's environment at all
---
# .github/workflows/provider.yml
on:
  schedule: [{cron: '0 3 * * *'}]
  workflow_dispatch:
jobs:
  provider:
    runs-on: ubuntu-latest
    env:
      ANTHROPIC_API_KEY: ${{ secrets.CI_TEST_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e '.[test]'
      - run: pytest -m provider --record-mode=rewrite
      - run: python tools/diff_cassettes.py --fail-on-structural-change

The gate job has no key in its environment, so the offline claim is enforced by the absence of a credential rather than by discipline. The key used by the nightly job should be a separate one, scoped to a test project or organisation with a spend limit configured, so that a runaway loop in a test costs a bounded amount and is billed somewhere visible.

What remains in the gate after the extraction is worth a sanity check, because the target is a number rather than a category. Remove twelve minutes of provider calls from a twenty-two-minute build and roughly ten remain, most of it in a Python service being dependency installation, container builds and a linter rather than tests — the 340 offline tests from the retry chapter's example run in about two seconds. Caching the virtual environment and running the test job in parallel with the lint and type-check jobs typically brings the wall clock to three or four minutes, which is inside the window where a person will wait for the result instead of switching tasks. That threshold is the actual design target; everything above it changes how the team works rather than merely costing time.

A suite that cannot block has to be able to shout#

The obvious failure of this design is that a nightly job nobody reads is a nightly job that does not exist. Two months of an unattended red build is worse than no build, because the repository now contains a mechanism people believe is watching.

What makes it stick is giving the failure a destination and an owner. The nightly run posts to a channel that a named person watches on a weekly rota, and a structural cassette diff or a provider-suite failure opens an issue automatically with the diff in the body, assigned to that rota. A convention that an issue from this source is triaged within three working days — fixed, or explicitly downgraded with a reason — is enough to keep it alive, and the honest test of whether it is alive is the age of the oldest open issue from that source. If that number is climbing past a fortnight, the suite has become decoration and should either be given a real owner or removed on purpose rather than by neglect.

The 40-minute, $12 evaluation stays off the critical path for a reason that goes beyond its cost. A score computed over a dataset carries variance, so a run-to-run difference of a point or two is frequently noise, and a gate has to convert that into a boolean. Set the threshold tight and pull requests are blocked by sampling variation; set it loose and a real regression passes. Neither setting produces a check whose red state means something specific. The reporting version avoids the dilemma: run the evaluation after merge, or on demand when a pull request carries a run-eval label, and post the result as a comment showing the score, the previous score, and the cases whose verdict changed. This is also why the rendered-prompt snapshot from chapter 3 is not redundant with the evaluation — the snapshot is fast, deterministic, and reports a cause, while the evaluation is slow, statistical, and reports an outcome. A team keeping only the second can see that quality dropped over a week of merges and cannot say which merge did it.

Detection lag and the meaning of green#

Where the "up to 24 hours" comes from is arithmetic on the schedule rather than an estimate. The job fires once a day, at 03:00, so the lag for any given change is the time between that change landing and the next 03:00. A change merged at 02:50 is checked ten minutes later. A change merged at 03:10 waits until the following night. Averaged over a working day the wait is around half a day, and the number worth planning against is the worst case rather than the average, because the worst case is what a change merged during office hours actually gets.

Moving provider tests to a nightly schedule accepts a detection lag of up to 24 hours for anything only a real call reveals. For a provider deprecating a field with three months' notice, 24 hours is irrelevant. For a change that starts returning malformed tool calls at 09:00, it is the difference between a bad morning and a bad day, and no build schedule fixes that — the detector for live breakage is instrumentation on real traffic, which is why the fourth column exists and why none of the first three replace it.

The other limit is that the gate's green state is a statement about your change and not about your system. Every offline test can pass on a day when your provider is down, your key has expired, and your model alias has been retired. That is the correct behaviour for a merge gate and a bad property to forget, particularly during an incident, when a green build is easy to misread as evidence that the service is fine.

It follows from what the gate is made of rather than from any oversight in building it. Every check in it compares one artifact in the repository against another artifact in the repository: a fake against a port, a rendered string against a stored copy, a schema against a handler's annotations, a delay against a recorded list. None of those comparisons has a term in it that the outside world supplies, so no fact about the outside world can change any of their verdicts. As far as the gate is concerned the provider does not exist, which is exactly why it is fast, free and stable, and exactly why it has nothing to say during an outage. The property and the price are the same property.

None of this needs to arrive at once, and the order that pays fastest is roughly the order of this topic. Introduce the port and the composition root for one call site, which unblocks everything else. Write the two or three fake-driven tests for the branches that have actually broken in production — the truncation branch, the parse failure, the retry policy — since those are the ones with a track record. Store the rendered prompt for the one prompt that gets edited most. Add the contract test for whichever tool schema is duplicated. Then, in a single pull request, mark the provider tests, delete the key from the pull-request job, and turn on the socket ban.

What that arrangement buys is a gate whose red state means one thing: you broke something. Everything else the repository knows how to detect still gets detected, on a schedule, by a job with an owner, and reported to somebody who can act on it without holding up a merge. The 63 tests are not the problem and the deletion is not the fix; their trigger is the problem, and a trigger is one line of configuration.