The new joiner is the only honest packaging test

Marta did all of it. svc-scaffold moved into a repository with a manifest, a lock file, packaged templates, a preflight check that names every environment variable it needs, and a README with seven numbered steps. She posted the link in #eng-tooling and asked people to install it from scratch and report back. Eleven people did over the following week, and eleven reported success — a couple of typo corrections to the README, one person who had to restart their shell, nothing else. She marked the packaging work done.

Aditi joined the team five weeks later. On her first morning she opened the README, ran step one, and stopped at step three. By lunchtime she had a list of nine things that were true of every machine that had reported success and were not true of hers. Four of them were not in the README at all.

What eleven successes were actually evidence of#

The eleven people who installed it had been at the company between eight months and six years. All of them had gone through the same laptop provisioning, which installs a Homebrew prefix, a company certificate bundle, and a shell profile that exports ARTIFACTORY_URL and a read token for the internal Python index. All of them had run the platform onboarding script at some point, which requests registry credentials and writes them to the login keychain. All of them had joined the engineering group in the company's single sign-on directory — SSO, the central system that decides who you are and, through group membership, what you are permitted to reach — which is what grants registry:read. Nine of the eleven had at some point worked on a service that required the VPN split-tunnel profile, and the other two had it because it ships in the base laptop image now.

Every one of those facts is a dependency of svc-scaffold. None of them is stated anywhere in the package. The install succeeded eleven times because the sample was drawn from people for whom the missing pieces were already present, which is the one population structurally incapable of revealing that they are missing. This is the same shape as a test suite that only runs against a database someone left seeded — it passes, and it passes for a reason unrelated to the code being right.

Aditi's laptop was three days old. It had an ARM chip and a preinstalled Python 3.13 rather than the 3.11 that everyone else's Homebrew had settled on years earlier. Her SSO group membership had been requested but not approved, so registry:read was pending. Her VPN profile covered the corporate network but not the 10.40.0.0/16 range where the internal package index lives, because that profile is attached to a group she was not in yet. Her failure list, in the order she hit it:

Two of those four are the package's fault in a way that is fixable by editing files: the Python version floor should have been a ceiling as well, and the index URL belongs in the repository's pip.conf rather than in eleven people's shell profiles. The other two are not fixable by editing files, and they are the more interesting half, because they are facts about access rather than about code. A package cannot grant its own permissions. What it can do is name them, and the README that named SCAFFOLD_TOKEN got Aditi to the right Slack channel in a minute, while the unnamed registry:read scope cost her most of an afternoon and two people's attention to diagnose from a bare 403.

Running the install where nothing is prepared#

The mechanical part of this is cheap, and it is worth doing before it costs a new joiner a morning. A container that starts from the same base image the laptops use, with no company profile, no keychain and no shell rc, is a reasonable stand-in for an unprepared machine — not a perfect one, and the gap is the subject of the next section, but it catches the code-shaped failures for the price of a CI job.

# .github/workflows/clean-install.yml
name: clean-install
on: [push, pull_request, schedule]

jobs:
  install:
    strategy:
      fail-fast: false
      matrix:
        python: ["3.11", "3.12", "3.13"]
        os: [ubuntu-24.04, macos-14]   # macos-14 is arm64
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python }}
      - name: Install exactly as the README says
        run: pip install -r requirements.lock && pip install --no-deps .
      - name: Smoke task
        env:
          SCAFFOLD_TOKEN: ${{ secrets.SCAFFOLD_TOKEN_CI }}
        run: |
          svc-scaffold init billing-api --out /tmp/out
          python tests/assert_scaffold.py /tmp/out

Two details in that job carry most of its value. The install step runs the README's command rather than a convenient equivalent, because a CI job that installs with pip install -e . while the README says something else is testing a procedure nobody follows. And the matrix includes 3.13, which is the version that broke, precisely because nobody on the team was running it — a matrix that covers what your colleagues have is a matrix that repeats the sampling error.

The smoke task is the part that most packaging CI omits. An install that finishes is a weak signal, since three of Marta's four original failures happened after a successful install. The assertion has to be about output that would actually be wrong if the package were subtly broken:

# tests/assert_scaffold.py
import json, sys
from pathlib import Path

out = Path(sys.argv[1])
spec = json.loads((out / "openapi.json").read_text())

assert spec["paths"], "openapi.json has no paths — render-openapi produced nothing"
assert "/healthz" in spec["paths"], "health endpoint missing from generated spec"

dockerfile = (out / "Dockerfile").read_text()
assert "registry.internal/base/python" in dockerfile, \
    "Dockerfile did not use the company base image — registry defaults were not applied"

print("scaffold ok:", len(spec["paths"]), "paths")

Both assertions correspond to a specific failure that already happened. The empty paths object is the bug from the Slack zip, and the base-image check is what would have caught registry_defaults silently falling back to public values when the token was absent or unscoped. Regression assertions written from real incidents beat assertions written from imagination, because the incident tells you which failure is silent, and silent failures are the ones a smoke test exists to convert into loud ones.

What the container will not tell you#

A green matrix and a clean container are worth having, and they will not answer the question this chapter is about. Their limits are structural rather than a matter of effort.

The last three rows are where the new joiner earns the title of the honest test. CI runs as svc-scaffold-ci, a service account with registry:read granted at creation and never revoked, so the 403 Aditi hit is invisible to it by construction. CI cannot experience the fact that requesting the SSO group takes a business day and goes to a rota that has one person on holiday. And CI cannot be confused by step four of the README, which said "point the tool at your team's template overrides" without saying where those live or what happens if your team has none — a sentence eleven people read as obviously meaning the thing they already had configured.

That last category is the one that keeps growing after the code-shaped problems are fixed. Documentation degrades toward the author's context, because each edit is made by someone who already knows the answer, and there is no compiler for a paragraph. The observation that a colleague completing a known task unaided is the strongest available evidence of packaging is not a claim that the colleague is a better tester than a machine. It is a claim about what the two can observe: the machine tests the artefact against a prepared environment it was given, and the person tests the artefact plus the instructions plus the access process against an environment nobody prepared.

The test you can only run once#

The awkward property of this test is that it destroys its own subject. Aditi can be a new joiner exactly once; after her first morning she has a keychain entry, an SSO group and a mental model, and she joins the population that cannot detect the next undeclared assumption. Teams that take this seriously handle it in a few ways, none of them elegant. The most common is to treat every actual onboarding as an audit: the new person keeps a running list of every deviation from the README, and fixing that list is somebody's task in that sprint rather than a nice-to-have. A second is to rotate a first-run duty, where an engineer from a different team — one that does not share your credential set — installs the package cold each quarter and reports where they stopped. A third is to make the preflight check itself carry the diagnosis, so that a missing permission surfaces as a named scope and a request link rather than as an HTTP status code. The preflight from the previous chapter listed environment variables; the version Marta shipped after Aditi's first day also probes the registry with a cheap authenticated call and reports which scope came back missing, which converts a forty-minute investigation into a line of output.

There is a boundary worth marking here. This test says nothing about whether the skill produces good scaffolds, which is the subject 12.6 covered and which is answered by evaluating the skill's behaviour rather than its installation. A package can be flawlessly installable and produce a Dockerfile with the wrong base image on every run. Conversely, a skill with excellent behaviour that only installs on prepared machines is a private tool with a public README. The two properties fail independently, and the clean-room run only addresses one of them.

It also says nothing about what happens next. Aditi installed version 0.4.0 on a Monday; by the end of the quarter the repository was at 0.9.2, and four teams had it in their setups. At that point the interesting failures stop being about installation and start being about change — someone tightens a description, someone alters a response shape, and the people affected have no error message to read, because nothing errored.