Isolation you cannot afford is isolation you will not have

The capacity plan said eight concurrent tasks. That number came from the queue: about 190 agent tasks a day arrived from the internal tooling, the median one took 14 minutes of agent time, and eight in flight was what kept the queue from growing during the European afternoon. The plan had one open decision left on it, which was how each of those eight got its own space, and three proposals had been written against it. Proposal A was the shared clone the harness already used, with a file lock around it. Proposal B was a worktree per task, as in the previous chapter. Proposal C was a container per task with a fresh clone inside it, which the security engineer who reviewed the design had asked for by name.

The argument in the review went the way these arguments usually go. Someone said isolation should never be a trade-off and that C was obviously correct, and nobody in the room could name a principled objection to that, so C went into the plan. Eleven days after it shipped, 70% of runs were going through a --reuse-workspace flag that a staff engineer had added on the Thursday because tasks were spending longer waiting for a clone than doing work. The flag was not a policy decision and nobody reviewed it. It was the predictable outcome of shipping a control whose cost the people paying it had not agreed to.

Two rankings over the same three options#

Rank the three on cost per task and the order is A, B, C from cheapest to dearest. Rank them on strength of isolation and the order is A, B, C from weakest to strongest. The two orderings are identical, which is exactly the problem: whichever end you optimise, you get the other end's worst option, and there is no scheme in the list that is best at both.

The row that decides against A is not an isolation row. It is effective concurrency, because a lock around one working tree means one task at a time, and the whole reason the number eight exists is that one at a time does not clear the queue. Eight tasks at 14 minutes each run in 112 minutes under A, 14.2 minutes under B and 17.7 minutes under C, which is 4.3 completed tasks per hour against 33.8 and 27. On setup cost per task A is the cheapest thing on the table, and on cost per completed task it is by a factor of six the most expensive. Those two ways of measuring cost disagree with each other before isolation enters the conversation at all, and the disagreement is the reason a lock feels free and is not.

That leaves B against C, and this is where the real decision sits. C is stronger on every isolation row and costs twenty times as much setup and three times as much disk. The question is not which is safer, because C plainly is. The question is whether the safety is worth 3 minutes 40 seconds and 4.1 GB per task on this workload, and answering it requires knowing what failure the isolation is preventing.

The clone is the expensive part, and it is negotiable#

Before conceding C's cost, it is worth checking how much of it is actually inherent. Almost all of it is the clone, and a fresh full clone is the least efficient way to get a repository into a container.

# baseline: what proposal C was measured at
$ time git clone git@git.internal:platform/platform.git /work/repo
real	3m41.204s
$ du -sh /work/repo
4.1G	/work/repo

# blobless partial clone from the same server
$ time git clone --filter=blob:none git@git.internal:platform/platform.git /work/repo
real	0m24.880s
$ du -sh /work/repo
641M	/work/repo

# reference an on-host mirror, refreshed by a cron job every 5 minutes
$ time git clone --reference /srv/mirror/platform.git --dissociate=false \
      git@git.internal:platform/platform.git /work/repo
real	0m19.640s
$ du -sh /work/repo
902M	/work/repo

A blobless clone fetches commits and trees but no file contents, and pulls blobs on demand as the working tree materialises, which for a task that touches thirty files means fetching thirty files instead of three million. Twenty-five seconds and 641 MB is a different proposition from 3 minutes 40 and 4.1 GB — it is roughly 2.5 times B's setup cost rather than twenty. The --reference variant is cheaper still on a host that already keeps a mirror, meaning a local bare clone — history and objects, no working tree — that a scheduled job keeps in step with the server, and it borrows that mirror's objects instead of transferring them. The price is a shared object store between the container and the host mirror, which gives back one of the isolation rows above and needs to be a deliberate choice rather than an accident.

The general lesson is that a cost column in a table like this one is a measurement of a particular implementation, and the ranking can move when the implementation does. Blobless clones do have a failure mode worth knowing: a command that walks history over file contents — git log -p, git blame on an old file, a full-tree git grep against a past commit — triggers on-demand fetches and can take minutes where it used to take seconds, and inside a sandbox with no network it fails outright. Agents run git log -p more than people do, so this is not a theoretical objection.

Why the cheap scheme wins arguments it should lose#

The --reuse-workspace flag is the mechanism that matters here and it deserves a name of its own. A control that imposes a cost on the person who benefits least from it will be routed around, and the routing will not look like a security decision when it happens. It looks like a flag, a cached base image that stops being rebuilt, an exception for the two tasks that were timing out, a CI job that reuses a runner because provisioning was slow. Each of those is a locally reasonable change made by someone solving a locally real problem, and the aggregate is a boundary that exists in the design document and not on the machine.

The arithmetic that drives it is worth doing explicitly. On this workload, C at its measured cost adds 3 minutes 42 seconds to a 14-minute task, which is 26% overhead — visible to every engineer waiting on a result, several times a day, forever. B adds 11 seconds, which is 1.3%, and nobody will ever file a ticket about it. The strength difference between them is invisible until the day it matters, and the cost difference is visible on every single run. Controls with that shape lose, because the pressure to remove them is continuous and the pressure to keep them is episodic. Carelessness about security on the part of engineers has nothing to do with it.

This is the specific hole in "containers with fresh clones for everything, isolation should never be a trade-off". It is not wrong about which scheme is stronger. It is wrong about what shipping means, because the scheme that gets deployed and then bypassed provides less isolation than the weaker scheme that stays attached, and the comparison that decides between two designs has to be between what each one will look like in six weeks rather than what each one does on the day it merges. A worktree per task that nobody has any incentive to disable is a boundary you can reason about. A container per task with a documented escape hatch used by 70% of runs is a boundary you cannot.

What this argument does not license#

The affordability argument is bounded, and the bound is the reason it is not simply an excuse for the cheaper option. It applies while the failure being prevented is collision and mistake — two runs writing the same file, a leftover node_modules from a previous task, a branch left checked out. Against that class of failure the strength difference between B and C is nearly zero, because both of them stop it completely, and paying twenty times more for an equal outcome is what makes the flag inevitable.

The argument says nothing at all about a run that is actively trying to leave its workspace. Cost only decides between two controls that are both adequate; between an adequate control and an inadequate one it decides nothing, and if B does not hold against the threat you actually have, then B being cheap is irrelevant and the correct move is to make C affordable rather than to pretend B is enough. The partial-clone numbers above exist precisely so that C is available when it is needed, at 25 seconds rather than 3 minutes 40.

There is also a cost column that none of these tables contains, which is what it is like to operate each scheme. A worktree is a directory an engineer can cd into with their normal editor when a run produces something baffling; a container that has already exited is a log file and whatever you remembered to copy out of it. Teams underinvest in that difference and then find that debugging agent behaviour in scheme C takes three times as long, which is a real cost even though no dashboard shows it.

What none of this has settled is the word "strength" in the isolation column. The table asserts a ranking without a scale behind it, and a ranking without a scale is an opinion. The scale turns out not to be a property of the mechanism at all — the same worktree is a solid boundary against one kind of failure and no boundary whatsoever against another, and which one you are facing depends on a fact about the task that no amount of measuring git will reveal.