Every run needs its own working tree, not its own clone

The harness — the program that pulls a task off a queue, starts an agent on it, gives that agent somewhere to work and collects the diff it produces — had been running two tasks at a time for three weeks without anyone noticing a problem, because for three weeks the two tasks had always touched different parts of the repository. On the Tuesday it broke, run A was assigned PLAT-2231 (add an idempotency key to the refund endpoint) and run B was assigned PLAT-2244 (bump the currency-formatting helper), and both of them touched src/billing. Both were pointed at the same checkout: /srv/agent/platform, a 4.1 GB clone of the monorepo that the harness had created once at install time and reused ever since.

Run B finished first and produced a 61-line diff. The diff applied cleanly to main and the tests passed in CI, and when the reviewer opened it there was a function in it that nobody on run B's task had any reason to write — a RefundIdempotencyKey type, half-defined, referenced from a file run B had genuinely edited. Run A's diff, produced four minutes later, was missing that type entirely, so run A's branch did not compile. Neither run had done anything wrong by its own lights. Both had read the tree, edited files, and asked git for a diff.

Two things in that story are worth naming before the fix will make sense. The first is the working tree, which is the directory of ordinary files git materialises for you to edit, and which is a separate object from the repository — the commit history, the file contents git stores as compressed objects, and the branch pointers — that git keeps in the hidden .git directory beside it. A working tree has exactly one current branch, recorded in a pointer called HEAD, and git checkout moves that pointer for every process working in the directory rather than only for the one that asked. The second is the unit the whole topic turns on. A sandboxed developer workspace is a private place for a single agent task to run: its own copy of the files, its own scratch space, and only the credentials and network reach that this one task was authorised to have. Sandboxed is the security half of the name, because the workspace is meant to be a boundary, so that a run which goes wrong damages what is inside it rather than the machine, the other runs, or production. Per-task is the operational half, because the workspace is created when the task starts and destroyed when the task ends, so nothing one run leaves behind can condition the next one.

The problem such a workspace solves is the one already visible above: with a single shared checkout, "run B's diff" is not a well-defined thing, because run B never had a tree of its own for a diff to describe. The same failure is familiar from CI systems that let two jobs share a build directory, and agent harnesses meet it sooner and harder, because runs are cheap enough to start eight at a time and each one spends twenty minutes editing files rather than two minutes compiling them. Where the reader meets this in practice is the moment a team stops running one agent at a time on a laptop and starts running a queue: on a build host, in a CI runner, or on the machine under someone's desk that grew into shared infrastructure. The rest of this topic works outward from the files to the credentials, the network and the cleanup; this chapter stays on the files, where the cheapest workspace available is a facility git has had for a decade.

Two runs, one working tree#

The sequence is easier to see laid out in time than described. The shared checkout is a single mutable object that both processes hold a handle on, and neither of them has any way to notice that the other one moved it.

The git checkout -b in the middle is the whole event. It moved HEAD for everybody sharing that directory, so when run B ran git diff it was diffing its own edits plus whatever of run A's edits were on disk against run A's branch tip. The resulting patch describes a transition between two trees, neither of which corresponds to any state either task was asked to produce. Topic 13.5 covered what happens when a patch's context lines do not match the target — a hunk fails, you get a rejection, and you can reason about it. This failure is worse than a rejection, because the patch is internally consistent and applies without a single conflict. The corruption is in what the patch means, and git has no way to know that the base it was computed against was a chimera assembled by two processes racing.

A shared checkout also produces failures that never reach a diff at all. A build that reads a file while another run is halfway through writing it gets a truncated module and a parse error somewhere unrelated. A test runner that walks the tree picks up test files from a branch it has never heard of. An agent that greps for a symbol finds a definition that its own task did not create and writes code against it. Every one of these is an intermittent failure that reproduces about one time in six, which is the least useful failure rate there is.

The first fix most teams reach for is a copy: cp -r the checkout before each run and delete it afterwards. That does isolate the tree, and it isolates the repository as well, which sounds strictly better until you price it. Copying 4.1 GB of files, three million of which are loose objects and packfile fragments, took 2 minutes 20 seconds on the harness machine's SSD and produced a second full object database that immediately began diverging from the first, so a run that fetched a new base branch pulled the same 300 MB of objects that the original repository already had on disk. The copy is a clone with the deduplication turned off.

What git already has for this#

The mechanism has been in git since 2.5, which shipped in 2015, and it predates any of this by a decade. A git worktree is an additional working tree checked out from a repository that is already on disk: its own directory, its own HEAD, its own index, reading and writing one shared copy of the history. git worktree was built for the ordinary human case of wanting a second checkout — a hotfix branch open while a feature branch stays half-finished — without paying for a second clone. The design assumption it was built on happens to be exactly right for parallel agent runs: several checkouts, one history, all on the same machine and all belonging to the same person.

$ cd /srv/agent/platform
$ git worktree add -b agent/2244 /srv/agent/wt/run-8f3a2c origin/main
Preparing worktree (new branch 'agent/2244')
HEAD is now at 4c19ab7 chore: bump pnpm to 9.12.0
$ git worktree add -b agent/2231 /srv/agent/wt/run-b71e04 origin/main
Preparing worktree (new branch 'agent/2231')
HEAD is now at 4c19ab7 chore: bump pnpm to 9.12.0

Each of those directories is a complete, ordinary working tree. cd into one and every git command behaves the way it does in a clone: git status shows that tree's changes, git checkout moves that tree's HEAD and no other, git stash stashes that tree's index. What is not duplicated is the part that costs money.

$ cat /srv/agent/wt/run-b71e04/.git
gitdir: /srv/agent/platform/.git/worktrees/run-b71e04

$ ls /srv/agent/platform/.git/worktrees/run-b71e04/
HEAD  ORIG_HEAD  commondir  gitdir  index  logs/

$ du -sh /srv/agent/platform/.git/objects
3.2G	/srv/agent/platform/.git/objects
$ du -sh /srv/agent/platform/.git/worktrees/run-b71e04
1.1M	/srv/agent/platform/.git/worktrees/run-b71e04

The .git entry in a worktree is a file rather than a directory, and it holds one line pointing back at a per-worktree administrative directory inside the main repository. That directory holds HEAD, the index, the reflog and a handful of pointers, and comes to about a megabyte. Everything expensive — the object database, the packfiles, the refs — stays in one place and is read by all of them. The per-run cost is the checked-out files themselves, which for this repository is 890 MB, plus that megabyte of bookkeeping.

The time difference is what makes the pattern usable inside a task's setup budget. Cloning the monorepo fresh from the internal git server takes 3 minutes 40 seconds on a good day and produces 4.1 GB. Adding a worktree from an existing local repository takes 11 seconds, almost all of it spent writing the 890 MB of files, and produces no new history at all. Eight concurrent tasks under the clone model means 33 GB and roughly half an hour of aggregate setup; under the worktree model it means one 4.1 GB repository plus 7.1 GB of checkouts, created in about a minute and a half of aggregate setup.

Wiring it into a harness#

The convention that makes this work is one directory and one branch per run identifier, both derived from the same string, so that any workspace on disk can be traced back to the run that owns it without consulting a database.

#!/usr/bin/env bash
set -euo pipefail

RUN_ID="$1"                       # e.g. run-b71e04
BASE_REF="${2:-origin/main}"
REPO=/srv/agent/platform
WT="/srv/agent/wt/${RUN_ID}"

git -C "$REPO" fetch --quiet origin
git -C "$REPO" worktree add --detach "$WT" "$BASE_REF"
git -C "$WT" switch -c "agent/${RUN_ID}"

echo "workspace=${WT} branch=agent/${RUN_ID} base=$(git -C "$WT" rev-parse HEAD)"

The base commit is recorded at creation time because it is what the eventual patch has to be interpreted against, and it is the one thing that cannot be recovered later once origin/main has moved. When the run finishes, the diff it publishes is git -C "$WT" diff "$BASE" HEAD, computed inside a tree that no other process has touched.

Two details bite people the first week. The first is that git refuses to check out one branch in two worktrees at once, which is a deliberate guard against exactly the class of confusion that opened this chapter:

$ git worktree add /srv/agent/wt/run-c02f11 main
fatal: 'main' is already used by worktree at '/srv/agent/platform'

That error appears the moment a harness tries to give a run the default branch, which is the natural first thing to write. Creating each run's branch immediately, or using --detach as the script above does and branching inside the worktree, avoids it. The second detail is that the shared object store means a git gc in any worktree operates on everybody's objects, and a run that calls git gc --prune=now while seven other runs hold loose objects they have not yet referenced can delete work in progress. Modern git guards most of this, but the general shape holds: commands that write to the repository rather than to the tree are shared operations, and a harness should not let a task run them.

There is a diagnostic benefit to the layout that is worth having even when nothing goes wrong. git worktree list prints every live workspace with its path, its commit and its branch, so an operator looking at a machine with disk pressure can see which runs are holding space and how old each one is without grepping process arguments. The same listing is what makes an out-of-band cleaner possible at all, because it turns "which of these directories is still in use" from a guess into a query.

What a worktree is not#

It is not a clone. There is no second copy of history, no independent remote configuration, and no way for one worktree's objects to diverge from another's, which means an operation that rewrites shared state — git filter-repo, a forced git reflog expire, a repack — reaches every run at once. If a task genuinely needs to mutate the repository rather than the tree, it needs a clone and there is no way around it.

It is also not a branch, and confusing the two produces a harness that creates worktrees it never removes. A branch is a ref, which is 41 bytes in a file; a worktree is 890 MB of checked-out files plus an entry in .git/worktrees that git will keep honouring until something explicitly removes it. Deleting the directory with rm -rf leaves the administrative entry behind and git worktree list keeps reporting a workspace that no longer exists. Chapter 8 is about what happens when that removal is attached to the wrong place in the lifecycle.

The saving on disk is also smaller than the object-store arithmetic suggests, because build output is untracked and therefore per-worktree. pnpm install in this repository produces 1.9 GB of node_modules, so eight concurrent runs carry 15 GB of dependencies whatever git does, and on a cold worktree the install takes 2 minutes 50 seconds — fifteen times the cost of creating the worktree itself. A content-addressed package store with hardlinks, or a warmed cache volume mounted into each workspace, is a separate problem from repository isolation and has to be solved separately.

The limit that matters most is that a worktree isolates a tree. Two runs in two worktrees still share the object database, the repository config and its hooks, every directory on the machine outside their own workspace, the process table, the network stack and the credentials in the environment they were started from. For the failure in this chapter — two colleagues' tasks stepping on each other by accident — a tree is exactly the right unit, because the collision was between files. Whether it is the right unit for the next failure depends on something a worktree cannot tell you, which is who wrote the task.

Before that question is worth asking, though, there is a cheaper one to settle. A worktree per run is one of at least three ways to stop two runs colliding, and the argument for it is not that it is the strongest. Ranking the options against each other is what the next chapter does, and the ranking is more interesting than it looks, because the cheapest scheme and the safest scheme are not the same scheme and the gap between them is where the real decision lives.