Cleanup at the end of a run is the code least likely to execute

The alert fires on a Wednesday most weeks. /srv/agent crosses 95% of its 520 GB, the on-call engineer runs a find for directories older than a day, deletes about sixty of them, recovers 70 GB, and the queue drains. The harness runs roughly 400 tasks a day. Its cleanup step reports a failure rate of 6%, which is 24 workspaces a day, and 24 workspaces at 1.1 GB each is 26 GB a day — enough to fill the remaining headroom in about twenty days rather than seven. The numbers do not agree, and the proposal on the table is to raise the quota by 1.5 TB, which by the same arithmetic buys about three weeks.

The 6% is measured over the wrong denominator#

A cleanup step that runs at the end of a task can only report on tasks that reached the end. Instrumenting the harness for a week and classifying every run by how it terminated produced the following, and the shape of it is the whole chapter.

The reported 6% is a true statistic about 354 runs and says nothing about the other 46, which are the runs the harness stopped by force and the runs the kernel stopped for exhausting the memory it had been given — an OOM kill, for out of memory. Those 46 leak at 100%, because a SIGKILL does not run a finally block, an atexit handler, a defer or a context manager's __exit__; it removes the process without giving it a chance to do anything at all. They also leak more per run than average: a task killed at the 45-minute budget has been building for 45 minutes, so its workspace carries a full node_modules, a populated build cache and often a running container, which is why 46 runs out of 400 account for 55 GB of the 82 GB. At 82 GB a day against 520 GB of space the disk fills in 6.3 days, which matches the Wednesday alert, and 1.5 TB of extra quota moves the alert out by 18 days without changing the rate at all.

The instinct at this point is to make the cleanup step more robust — add --force, add retries, catch more exception types, wrap it in a second try. Every one of those improvements applies to the 24 runs that reached the code and none of them to the 46 that did not. Hardening a path that is being skipped rather than failed is work that cannot move the dominant term.

# the shape that cannot be fixed from the inside
def run_task(task):
    ws = create_workspace(task.run_id)
    try:
        return agent.execute(task, ws)
    finally:
        cleanup(ws)          # not executed on SIGKILL, SIGSTOP+OOM,
                             # host termination, or a panic in the runtime

A reaper reads the host, not the run#

Removal has to be performed by something that outlives the run and can discover leftovers from the outside. The name for that in systems work is a reaper: a process that runs on a timer, looks at what actually exists, and disposes of whatever belongs to an owner that is gone. The reaper takes no input from the process it is cleaning up after; it lists what exists on the host, joins it against a registry of runs, and removes anything that is both attributable to a run and safe to touch.

Two transitions carry the safety argument. Orphaned exists because a run that was killed never updates its record, so terminal state cannot be the only trigger; the heartbeat is written by the harness every 30 seconds and a workspace whose last heartbeat is older than twice the task budget is orphaned regardless of what its record says. The step from Terminal or Orphaned to Reapable is a check against the host rather than against the registry, because a record can be wrong and a running process cannot.

WS_ROOT = pathlib.Path("/srv/agent/wt")
RUN_DIR = re.compile(r"^run-[0-9a-f]{6}$")

def reap_once(now: float) -> None:
    for path in sorted(WS_ROOT.iterdir()):
        if not RUN_DIR.match(path.name):
            continue                                  # never touch strangers
        rec = registry.get(path.name)
        if rec is None:
            if now - path.stat().st_mtime < 3 * 3600:
                continue                              # racing with creation
        elif rec.pinned_until and now < rec.pinned_until:
            continue
        elif not rec.terminal and now - rec.heartbeat_at < 2 * TASK_BUDGET_S:
            continue                                  # still alive
        if in_use(path):
            metrics.incr("reaper.deferred"); continue
        remove_workspace(path)

def in_use(path: pathlib.Path) -> bool:
    if subprocess.run(["fuser", "-s", "-m", str(path)]).returncode == 0:
        return True                                   # open fd or cwd
    return any(str(path) in m for m in open("/proc/self/mountinfo"))

def remove_workspace(path: pathlib.Path) -> None:
    run_id = path.name
    containers.kill_and_remove(label=f"agent_run={run_id}")   # no-op if absent
    umount_below(path)                                        # ignores ENOENT
    subprocess.run(["git", "-C", REPO, "worktree", "remove", "--force", str(path)])
    shutil.rmtree(path, ignore_errors=True)
    subprocess.run(["git", "-C", REPO, "worktree", "prune"])
    broker.revoke_run_identity(run_id)                        # idempotent
    registry.mark_removed(run_id)
    log.info("reaped", extra={"run_id": run_id, "freed_bytes": ...})

Every operation in that function tolerates having already happened, which is what makes it safe to run twice — and it will run twice, because a reaper that deferred a workspace on one pass will find it again five minutes later, and because two reapers will eventually run concurrently after somebody restarts the service during a pass. Container removal by label finds nothing if the container is gone. umount on an already-unmounted path is a no-op. git worktree remove on a missing path fails and the following prune fixes the administrative entry that chapter 1 described, which is the specific thing an rm -rf alone leaves behind. The regular expression on the directory name is not decoration: a reaper is a program whose whole purpose is deleting directories on a schedule, and the guard that stops a bad registry lookup from turning it into a program that deletes /srv/agent costs one line.

The registry-miss branch is the subtle one. A directory with no record could be a leak from before the registry existed, or a workspace created three seconds ago whose record has not committed yet. Waiting three hours before removing an unattributable directory resolves the race in favour of the running task, and the workspaces that fall into that branch are rare enough that three hours of extra disk is not worth a more clever answer.

Two removals that fail and keep failing#

Idempotence — the property that performing an operation a second time leaves the system in the same state as performing it once, so a retry is never worse than the original attempt — assumes the removal eventually succeeds, and two cases break that assumption in ways worth separating from the ordinary deferral above.

The first is two reapers deleting the same tree at once, which happens the first time somebody restarts the service mid-pass and again every time an operator runs the reaper by hand because the disk graph looks wrong. Both processes walk /srv/agent/wt, both see run-b71e04, both call rmtree, and the loser raises FileNotFoundError on a directory the winner has already unlinked. ignore_errors=True hides that, and it also hides a half-removed workspace, which is the more expensive outcome: the run directory still exists with a few hundred files in it, so the next pass treats it as a workspace to reap rather than as debris, and the registry never records it as removed. Claiming the directory by renaming it makes the ownership question atomic, since exactly one rename on a given source path can succeed:

def claim(path: pathlib.Path) -> pathlib.Path | None:
    dest = TRASH / f"{path.name}.{os.getpid()}"
    try:
        os.rename(path, dest)      # atomic within the filesystem
    except OSError:
        return None                # another reaper got there first
    return dest

Removal then happens under TRASH, where a partially deleted tree is unambiguous, and the run directory disappears from the listing at the moment it is claimed rather than several seconds later when the last file goes.

The second case is a workspace that cannot be deleted by the account trying. A container that runs as root and bind-mounts the workspace writes its build output as UID 0, so rmtree running as the agent user gets EACCES on node_modules/.cache and stops there. It is not busy, fuser reports nothing, and the reaper will attempt it every five minutes until somebody notices. On the harness this affected 3 of 400 runs a day — the ones whose task involved a Docker build — and 90 undeletable workspaces had accumulated in the month before anyone connected them to the disk alerts. The structural fix is user namespace remapping so the container's root maps to the harness user; the operational one is a reaper.failed counter kept separate from reaper.deferred, because deferral is normal and expected to resolve on its own while a failure is a workspace that will still be there tomorrow.

Cleanup is more than disk#

Space is the failure that pages someone, and the leaks that do not page anyone are worth listing because they degrade the system slowly. Every run creates a branch, so after a year of unreaped runs the git server was carrying 47,000 agent/* refs and a fresh git fetch had gone from 1.4 seconds to 19 seconds for every developer in the company, because ref advertisement is proportional to ref count. Every run holds a broker identity, and identities that are never revoked mean the revocation list is not a meaningful control. Containers that outlive their runs hold network namespaces and proxy sessions, which is how a machine ends up with 300 idle veth interfaces — the virtual ethernet pairs that connect a container's network namespace to the host's — and a five-second docker ps. The reaper is the right place for all of it, because it already knows which run identifiers are finished.

A reaper is also not a substitute for a per-workspace quota. It runs every five minutes, and a runaway build that writes a 400 GB log fills the disk in eight minutes, so the reaper arrives after the outage. An XFS project quota, which is a size limit the filesystem enforces on a named directory subtree rather than on a user, or a sized loopback filesystem per workspace, caps that at, say, 20 GB, which converts a host-wide outage into one failed task. The two controls address different failures and neither one covers the other's.

The reaper needs its own instrumentation, and the metric to alert on is not disk usage. Disk percentage is a lagging indicator that tells you the accumulation has already happened; the leading indicator is the age of the oldest non-pinned workspace, which should sit at a few minutes and starts climbing the moment something is stuck. A rising reaper.deferred count is the other one, because a workspace that is deferred on every pass for an hour has a process holding it that nobody knows about — usually a test runner that survived its parent, which is the failure that start_new_session=True in chapter 4 exists to make killable. Alerting on those two turned the Wednesday page into a Tuesday ticket with three hours of slack in it.

Moving to containers does not make any of this go away, which surprises teams who expect the orchestrator to handle it. A container that exits still leaves its anonymous volumes, and docker volume ls on the harness host showed 2,300 of them holding 180 GB. A Kubernetes pod with a finalizer that never completes — a finalizer being a cleanup hook the API server waits on before it will actually delete an object — sits in Terminating indefinitely and keeps its PersistentVolumeClaim bound. The mechanism differs and the design error is identical, so the answer is identical too: a process outside the run that lists objects by label, joins them against a registry of runs, and removes what is attributable and idle. docker container prune on a schedule is that process, badly, because it has no idea which runs are still alive.

Keeping one on purpose#

There is a case for not removing a workspace, and it is the run that produced something a human needs to look at. A task that failed in a way nobody can reproduce, a patch that behaves differently outside its workspace, an agent trajectory under investigation after an incident — for these, the state on disk is the evidence, and reconstructing it from a transcript is unreliable because the build cache and the untracked files are exactly what the transcript does not contain.

The way to do it without reopening the original problem is a pin with an owner and an expiry, which is the Pinned state in the diagram. agentctl pin run-b71e04 --hours 72 --reason "PLAT-2301 flake" sets pinned_until, records who asked, and leaves the workspace in the same listing as everything else. The cost is real and bounded: six pinned workspaces is about 13 GB, and the pin appears on the same dashboard as the disk metric so it is visible when space gets tight. A pin without an expiry becomes indistinguishable from a leak within a month, which is how the original problem gets rebuilt under a friendlier name.

What the reaper demonstrates is that a workspace stopped existing. It says nothing about whether the boundaries around the next workspace are still attached — whether the proxy is still the only route out, whether the credential minting still targets the development account, whether the seccomp profile survived the base image upgrade. Those are properties of a running system that was configured correctly at some point in the past, and the only thing that can establish them today is a run that tries to cross the boundary and is stopped.