Per-run directories still share the cache, the temp dir and the config

Running several agents at once on one machine creates a second class of concurrency bug, distinct from the shared database row of the previous chapters. Nothing here is a resource the runs meant to share. They collide over a package cache, a temporary file, a configuration file in a home directory and a TCP port, and in each case the sharing was inherited from a default written years earlier by somebody assuming one user doing one thing at a time. This chapter is about finding those defaults and removing them, which is a different job from the locks and conditional writes of chapter 4 — the aim is not to order the access but to arrange that there is nothing shared to access.

The batch fixer this chapter works from ran ten coding agents at a time on one 16-core box, each one taking a failing test from a different repository, editing code until it passed, and opening a pull request. Each run got /work/run-<id>, a fresh clone, and cwd set before the process started. On the Wednesday it went to ten concurrent runs from four, the failure log looked like this:

run-3   ERR_PNPM_BAD_TARBALL  integrity check failed for registry.npmjs.org/esbuild/0.21.5
run-3   ERR_PNPM_BAD_TARBALL  (retry) integrity check failed
run-6   Error: ENOENT: no such file or directory, open '/work/run-6/repo/node_modules/.bin/vitest'
run-7   Error: listen EADDRINUSE: address already in use :::3000
run-9   Error: listen EADDRINUSE: address already in use :::3000
run-2   PR opened by "batch-fixer-4 <ci+4@example.com>"     ← wrong author

Six failures across three categories, none of them in the agents' reasoning, none of them reproducible when a run was retried on its own. Every one came from a resource that all ten processes were using and none of them had asked for.

The working directory is the smallest part of a process's identity#

Setting cwd isolates paths that are written relative to it. Every other path a tool touches is resolved from the environment, and the environment was inherited from the supervisor, which means all ten processes had the same HOME, the same TMPDIR, the same XDG_CACHE_HOME, and the same everything downstream of those.

Read the third column rather than the first, because that column is what makes the table a boundary map instead of a list of paths. Almost every row is governed by one of three variables, and a run that inherits those three variables inherits every row governed by them, including the rows for tools it has not installed yet. The two rows whose third column is neither HOME nor an XDG variable are the two that no environment change will fix — a port written into an application's own config, and a daemon socket at a fixed path — and those need a different remedy, which the next section comes to.

Three of those rows explain the Wednesday log entirely. The EADDRINUSE pair is two runs starting a dev server on the port the repository's config hardcodes. The wrong PR author is one run executing git config --global user.email as part of its setup and rewriting a file the other nine also read.

The ENOENT on run-6's vitest binary is the interesting one, and it needs its chain of causes laid out, because the deleted file is inside a directory that belongs to a run which did nothing wrong. It was caused by run-3 finishing and running its teardown, which was rm -rf /tmp/agent-* && pnpm store prune. Follow what prune does. A content-addressed store keeps one copy of each package's files and hands them out by hardlinking, so every node_modules/.bin entry in every workspace was a hardlink into that store, which is a directory entry pointing at bytes the store also points at rather than a copy of them. run-6's vitest binary and the store's copy were two names for one set of bytes on disk. prune removes store entries with no references it could see — and the references it can see are the ones the store recorded, which do not include a workspace it was never told about. So it deletes bytes that a live run is still using, and because a hardlink is a name rather than a copy, the deletion is visible inside /work/run-6. Pruning a shared store deletes files that live, from the filesystem's point of view, inside other runs' directories.

The pattern behind all three is that a default is a form of coupling that does not appear in any code you wrote. Nobody typed /tmp, 3000 or ~/.gitconfig into the agent; they were typed years ago into pnpm, into a Vite template, and into git, by people assuming one user doing one thing at a time. That assumption held for the four-concurrent version because collisions need overlap and four runs installing at different moments mostly missed each other. Concurrency exposes defaults the way load exposes an N+1 query.

Building the environment rather than adjusting it#

The instinct is to override the ones you know about: set TMPDIR, set HOME, move on. That is the version that fails six months later when somebody adds a tool with its own cache variable, because an inherited environment with a few overrides silently passes through everything you did not think of. The two arrangements differ in which direction a mistake points. With an inherited environment, forgetting a variable produces a tool that quietly writes into a shared location and works, so the mistake is invisible until two runs overlap on it. Starting from nothing and adding what is needed inverts which mistakes are possible, so a forgotten variable becomes a tool that cannot find its config rather than a tool that quietly shares one — and it fails immediately, in the run that forgot it, at the moment somebody is watching:

WS=/work/run-$RUN_ID
mkdir -p "$WS"/{home,tmp,cache,store,repo}

PORT=$(python3 -c 'import socket;s=socket.socket();s.bind(("",0));print(s.getsockname()[1]);s.close()')

exec env -i \
  PATH=/usr/local/bin:/usr/bin:/bin \
  HOME="$WS/home" \
  TMPDIR="$WS/tmp" \
  XDG_CACHE_HOME="$WS/cache" \
  XDG_CONFIG_HOME="$WS/home/.config" \
  GIT_CONFIG_GLOBAL="$WS/home/.gitconfig" \
  GIT_CONFIG_SYSTEM=/dev/null \
  npm_config_cache="$WS/cache/npm" \
  PNPM_HOME="$WS/home/.pnpm" \
  PORT="$PORT" \
  RUN_ID="$RUN_ID" \
  agent-runner --workspace "$WS"

env -i is the line that does the work, since it starts the child with an empty environment rather than the supervisor's, and every variable after it is one somebody decided to grant. The port is allocated by binding to :0, reading back what the kernel assigned, and closing it, which leaves a small window where another process could take it and is dramatically better than a range assigned by run index that breaks when a run is retried. GIT_CONFIG_SYSTEM=/dev/null matters because /etc/gitconfig is shared and writable by anything running as root in the image, and pointing it at an empty device removes a file no environment variable would otherwise redirect. Everything the process later shells out to — the package manager, the test runner, a language server, a browser download — inherits this environment, so a tool nobody anticipated lands in the workspace by construction.

The 7.6 machinery is what makes this a boundary rather than a convention: a mount namespace that only shows the run its own workspace, or a container per run, is what stops a process from writing outside the paths above whether or not it consults the environment. The distinction is worth holding onto, because the two do different jobs — the environment tells a cooperating tool where to write, and the namespace makes a non-cooperating one unable to write anywhere else. The point worth adding to that is that an isolated sandbox with a shared cache volume mounted into it has re-created every row of the table inside a boundary that looks correct on the diagram. The volume is the coupling, and the sandbox has no opinion about it.

Names outside the filesystem need the same treatment and are easier to forget, because no environment variable governs them. Containers started by a run are named postgres-test in nine repositories out of ten, so the second run to start one gets a conflict or, worse, connects to the first run's database and passes its tests against somebody else's data. Docker networks, compose project names, systemd unit names and abstract unix sockets are all flat namespaces shared by everything on the host. Prefixing every such name with the run id costs one line in the setup and removes a class of failure whose symptom — tests passing against the wrong data — never appears in a log.

There is a cheap way to discover what you have missed, which is to make the shared locations unwritable and see what breaks. Running one agent with HOME pointed at a read-only bind mount and /tmp mounted noexec,ro produces a list of every tool that expected to write outside its workspace, in the form of stack traces, in about the time it takes for the run to fail. What makes that work is that it converts a silent property into a loud one: a write to a shared location is indistinguishable from a correct write until somebody else writes there too, and a read-only mount turns every such write into an error with a path and a stack in it. The Playwright browser cache in the table above was found that way rather than by reading anyone's documentation.

The cache is the one you actually have to think about#

Isolating the cache is not free, and this is where the honest trade-off lives. Three arrangements, measured on the same repository, a Next.js app with 1,240 dependencies:

Read that table as two gaps rather than three options. The first gap is what safety costs when it is bought naively: the middle row is safe and pays roughly thirteen times the install time and the whole dependency tree in disk, and multiplied across the ten concurrent runs this box was built for, ten cold runs cost 9.4 GB of disk and thirteen minutes of install time that nobody's agent gets credit for. The second gap is the one worth having: the third row is also totally safe and sits within a few seconds of the shared writable store, which says that almost all of the middle row's cost was paying twice for bytes that are identical in every run.

The overlay arrangement — the shared store mounted read-only, with an overlayfs or copy-on-write upper layer per run, so a run reads every unmodified file straight from the store and gets a private copy of any file the moment it writes to it — gets most of the speed and all of the safety, at the price of a store that only a separate, serialised warming job may write to. That warming job is a single-writer queue, which is chapter 4's mechanism showing up in the place where the resource genuinely is shared: there is one store, everybody benefits from it being populated, and exactly one process may populate it. The reason the read-only mount is safe where a shared writable store is not is contained in the failure column of the first row — the integrity errors come from concurrent writes, and a mount that permits no writes has no concurrent writers to arbitrate.

Repository state has its own version of the same trade. Ten full clones of a large monorepo is minutes of I/O per run, and git worktree add /work/run-7/repo -b agent/run-7 gives each run its own working tree, index and HEAD while sharing the object database, which is the arrangement Git Worktrees Explained: The Feature That Unlocks Parallel AI Agents is about. What worktrees share is more than it looks: refs, config and the object store are common, so two runs cannot create the same branch name, a git gc triggered in one worktree touches objects every other worktree depends on, and .git/index.lock contention appears when two runs commit at the same instant. The git gc case is the store-prune failure again in a different tool, and for the same reason — a reclamation pass reasoning about references it can see, in a store whose readers it was never told about. Setting gc.auto=0 on the shared repository and deriving branch names from the run id removes both.

Teardown is part of the boundary#

rm -rf /tmp/agent-* is a write to a shared resource, performed by a process that has no idea what else matches the pattern, and pattern-matched cleanup is the single most common way a well-isolated system corrupts itself. The defect in it is that a glob is a query about names and ownership is a fact about history, and the two are only related by convention — every other run's directory matches the pattern just as well as this run's does. The alternative is that every run records what it created and the reclaim path reads that record:

@dataclass
class Manifest:
    run_id: str
    lease_until: float
    paths: list[str] = field(default_factory=list)
    containers: list[str] = field(default_factory=list)
    branches: list[str] = field(default_factory=list)
    ports: list[int] = field(default_factory=list)

async def reap(now: float) -> None:
    for m in load_manifests("/work/manifests"):
        if m.lease_until > now:
            continue                      # still owned, even if the process is gone
        for c in m.containers:
            await sh(f"docker rm -f {c}")             # idempotent
        for b in m.branches:
            await sh(f"git -C /srv/repo.git branch -D {b} || true")
        for p in m.paths:
            if p.startswith(f"/work/run-{m.run_id}/"):   # ownership check, not a glob
                shutil.rmtree(p, ignore_errors=True)
        os.unlink(f"/work/manifests/{m.run_id}.json")

The lease is what makes this work when a run is killed rather than finished, and the reason it is needed follows from two failures that cannot both be avoided by any timer alone. A finally block does not execute when the pod is evicted, the box reboots, or the supervisor is SIGKILLed, so a cleanup that only lives in the run's own process leaves orphans forever. A cleanup that runs on a timer without ownership goes wrong in the opposite direction and deletes live work. The lease resolves both because it is a claim that expires: it is renewed while the run is alive and expires shortly after it stops, which gives the reaper — the sweep that runs on its own schedule and reclaims what finished or dead runs left behind — a safe rule that needs no knowledge of whether a process is still breathing. Reclaim what nobody currently owns. Teardown is also a step with external side effects — containers, branches, rows in a staging database, objects in a bucket — so it belongs in the durable workflow from 10.16 rather than in application code, and every action in it has to be idempotent, because the reaper will run again.

Where isolation stops helping#

Isolation removes sharing that was accidental. It does nothing about sharing that is the point. The customer record two runs are both updating, the rate-limit budget on the partner API, the single staging database every branch deploys against, the shared git remote all ten runs push to — none of those can be given a per-run copy without changing what the system means, and all of them still need the locks, queues and conditional writes from chapter 4. A team that isolates workspaces well can end up believing the concurrency problem is solved, having removed the ninety per cent that was accidental and left the ten per cent that corrupts customer data.

The other cost is that isolation defers conflicts rather than preventing them. Ten worktrees can each edit package.json, each run its tests, and each report success, and then four of the ten pull requests fail to merge because they changed the same file. Isolation guaranteed that the runs did not interfere; it did not, and cannot, guarantee that their outputs compose. The two guarantees are about different moments — one about what happens while the runs execute, the other about what happens when their results are combined — and only the first is a property a workspace boundary could ever have.

The part most likely to be got wrong is not in the design at all, which is worth stating plainly because it is where the incidents come from. Setting HOME and TMPDIR is a morning's work and it is the part everyone remembers. The teardown path, and the tool nobody knew writes to the home directory — docker login dropping a token into ~/.docker/config.json, a test framework downloading browsers into the cache, a linter writing a global config on first run — are what leaks, because they were never in the list of things to isolate.

That is the argument for the two mechanical habits in this chapter over any amount of care. An environment built from empty catches the tool you did not know about, because the tool announces itself by failing rather than by sharing. A teardown driven by a recorded manifest catches the path you did not know a run had created, because the record is written by the run rather than guessed at by the sweeper. Neither requires anybody to have a complete list, and a complete list is the thing nobody has. The way to find what remains is not to think harder about the list; it is to run ten of these at once, on purpose, repeatedly, with something checking afterwards that each run's outputs are exactly its own.