Shared namespaces and warm pools

Per-session isolation is a claim that gets made about a layout. Each session gets its own directory, its own cookie jar, its own cache, its own download folder, and nothing in the code ever constructs a path into another session's directory. That is a true description of the code and says nothing about what the code could name, which is the only question a boundary answers.

This chapter runs that distinction over two designs at opposite ends of the strength scale: a browser automation service that separates sessions by directory inside one container, and a pool of warm Firecracker microVMs handed out to successive executions. They fail in the same way for the same reason, which is why the useful discipline is not "how strong is our sandbox" but "what does it share, with whom, and for how long".

What the session next door can name#

Take the browser design first. One container per host runs one Chrome instance, and each user session gets its own profile directory:

/sessions/
  a3f1c9/profile/    <- user 4471, logged into supplier-portal.com
  b7d204/profile/    <- user 9102, logged into a bank's treasury portal
  c1e88a/profile/    <- user 4471 again, a second concurrent task

The sessions are separated by a naming convention, and a naming convention constrains code that is trying to behave. Every session in that container runs as the same uid — the numeric user id the kernel uses to decide who owns a file and who may open it — and in the same mount namespace, the same PID namespace and the same network namespace, so the set of filesystems, the list of processes and the set of network interfaces and ports are one shared set for all of them. The agent driving each session executes steps derived from page content, which means an attacker who controls a page controls, to a useful degree, what those steps are. Ask what such a step can reach.

# running inside session b7d204's automation step
$ ls /sessions/
a3f1c9  b7d204  c1e88a

$ sqlite3 /sessions/a3f1c9/profile/Default/Cookies \
    "select host_key, name, length(encrypted_value) from cookies limit 3;"
supplier-portal.com|session|196
supplier-portal.com|csrf|84
.google.com|SID|312

# on Linux with no keyring available, Chrome's v10 cookie encryption
# uses a fixed passphrase, so 'encrypted' is not a meaningful obstacle
$ python3 -c "from Crypto.Cipher import AES; ..."   # decrypts in ~2 ms

$ cat /proc/$(pgrep -f 'user-data-dir=/sessions/a3f1c9')/environ | tr '\0' '\n' | grep TOKEN
SESSION_TOKEN=eyJhbGciOiJIUzI1NiJ9...

$ curl -s localhost:9222/json/list | jq -r '.[].url'
https://treasury.examplebank.com/payments/new
https://supplier-portal.com/rooms/booking

That last command is the one that ends the argument. The Chrome DevTools protocol is the debugging interface a browser exposes on a local port so that a client can list open tabs and drive any of them; it listens on loopback, the machine's own 127.0.0.1 address, and loopback is a property of the network namespace, which every session in that container shares. Any session can enumerate every target in the browser, attach to one belonging to a different user, evaluate JavaScript in its page, and read whatever that page can read. The profile directories are irrelevant to that path; it never touches the filesystem. Two of the other paths — reading /proc/<pid>/environ, where /proc is the kernel's filesystem view of running processes and environ is the environment a process was started with, for a process owned by the same uid, and the fact that files under /sessions are readable by a process running as the same uid — are equally indifferent to how the directories are laid out.

Written as a chain, the DevTools path has four links and every one of them is checkable. The debugging interface binds to loopback, which is an address rather than a policy. Loopback is not a global object — each network namespace has its own — so which processes can reach a loopback listener is decided entirely by which processes are in that namespace. Every session in this container is in one namespace, because the design put them in one container and a container is one namespace set. Therefore the listener that session a3f1c9's browser opened is, from session b7d204, simply a port on its own machine, and connecting to it needs no privilege, no bug and no filesystem access. Nothing in that chain mentions directories, which is why no rearrangement of /sessions affects it. The same shape explains the two same-uid paths just above it: a filesystem permission check compares the opener's principal against the file's owner, and when every session runs as one uid the comparison has the same inputs whichever session is asking. A check that receives identical inputs returns identical answers, so it cannot be the thing separating them.

The strongest objection to the design, then, is that one session's cookies and tokens are readable by every other session in that container, because a directory is a label and not a boundary against code running beside it. Filesystem permissions become a boundary when the principals differ; running every session as the same uid removes exactly that. Even with distinct uids per session, the shared network namespace keeps the DevTools path open, the shared PID namespace keeps process listing open, and /tmp stays common ground.

The same reasoning at the other end of the spectrum#

It is easy to read the browser example as a story about a team that chose a weak technology. The more useful reading is that they chose to share, and the sharing determined the outcome. That reading transfers directly to the strongest technology available.

Chapter 1, on containers, gVisor and microVMs, left a workload of 40 executions a second needing a microVM boundary and unable to pay 125–200 ms of boot on every request. The standard resolution is a warm pool: keep 60 VMs booted, hand one to an execution, take it back when the execution finishes, hand it to the next execution. Assignment costs about 3 ms instead of 150, the latency budget survives, and the design document still says Firecracker.

Whether that final line returns anything belonging to tenant 1 depends on what the arrow labelled rm -rf /workspace/* actually restores. A workspace wipe leaves /tmp, leaves the pip and npm caches under $HOME, leaves whatever a long-lived warm Python process is holding in its heap, leaves the guest page cache, leaves environment variables inherited by the supervisor, and leaves any file descriptor or TCP connection the previous execution opened. A VM reused 500 times over an hour is a shared machine for that hour, and it is shared between whichever tenants the scheduler happened to place on it.

The arithmetic makes this concrete. It gives two answers rather than one, and only the second of them describes the system in front of you. At 200 ms of work plus 3 ms of assignment a single VM can take 4.9 executions a second, which is a capacity figure: a VM that never went idle would get through roughly 17,600 executions in an hour. That is not the number this system produces. 40 executions a second arrive for the whole pool and sixty VMs share them, two thirds of an execution a second each, so sixty of them is far more capacity than 40 executions a second needs; the pool is running at well under a fifth of what it could take, and it is sized for bursts and for failures rather than for throughput. Spread evenly across it, one VM still runs about two thousand four hundred executions in an hour. If your tenant mix is 200 customers, that VM has, in an hour, run code on behalf of most of them. The blast radius of one execution leaving a credential in /tmp — blast radius being the set of principals and data a single compromise ends up reaching — is not "the next execution" but every execution that runs on that instance before it is destroyed, and the population of principals affected is the population that shares the pool.

Notice that the conclusion does not depend on which of those two figures you believe. The lower one — the one that describes an under-used pool doing exactly the work it was built for — already puts thousands of executions from most of two hundred tenants through one instance in an hour. Take the pool to saturation and the number grows by a factor of seven and the conclusion does not change, because it was already "most of them" at the low end. That robustness is what makes the argument worth having in a design review: somebody who disputes the utilisation figure, the tenant count or the work duration is disputing a quantity the finding is insensitive to. What the finding is sensitive to is the one thing nobody has written down, which is how long an instance lives before it is destroyed.

The naive strong answer — take the microVM because it is the strongest boundary — arrives at a system where the boundary is real between VMs and absent within one, and where nobody has written down which of those two cases a given pair of executions falls into. The browser design at least had the virtue of being obviously shared. A pooled design looks isolated on the architecture diagram and is shared in the same way, over a window nobody has stated.

Tracking sharing instead of tracking technology#

The useful discipline is to stop asking "how strong is our sandbox" and start asking, for each resource, two questions: which principals can name it, and over what window. Principals here means tenants, users, and executions, whichever is the unit you owe separation between. The window means the interval over which a resource is reachable by more than one of them — one execution, one session, one pod lifetime, forever.

Run those questions over the browser design and the answers are: cookie jars, one container's worth of sessions, for the life of the container. DevTools endpoint, the same. Downloads directory, the same. /tmp, the same. Every answer is the same because everything is inside one shared namespace set, and the directory structure changed none of them.

Run them over the pooled microVM design and the answers separate. Guest RAM is per-VM and reset only by reboot. The rootfs is read-only and shared by every VM, which is fine because nothing can write it. Writes land instead in an overlay, which stacks a writable upper directory over that read-only lower one so the base image is never modified; that upper layer is per-VM and persists across executions unless explicitly discarded. /tmp is per-VM and persists likewise. The tap device and its IP are per-VM and get reused, which matters if anything downstream authorises by source IP. Writing the answers down is what turns "we use Firecracker" into a claim someone can check.

The property that makes a resource dangerous to share is not that it is writable. A read-only cache directory shared across tenants is fine right up until one tenant can poison it, and a warm interpreter process shared across executions is fine right up until one execution can leave state in it. What matters is whether a principal can influence the resource in a way a later principal observes, and whether the earlier principal's data can be recovered by the later one.

Producing the inventory is mechanical enough to do from inside the sandbox in about ten minutes, and doing it from inside is the point, since the list you want is the list of things the untrusted process can name. findmnt -R gives every mount visible in the namespace, including the ones somebody added for convenience. ls /proc/[0-9]*/cmdline gives every process the namespace exposes, and if that list contains anything belonging to another principal, so does /proc/<pid>/environ. ss -ltnp gives every listener reachable on loopback, which is where debug endpoints, language servers and browser control ports tend to sit. ip addr and a traceroute to anything on the internal network tell you what the network namespace reaches. Each command's output is a set of names, and the names are the attack surface; the design document is not.

Legitimate sharing, and the authority a session already holds#

None of this is an argument that sharing is forbidden. Almost every economically viable sandbox shares something, because the alternative is booting fresh hardware per request. A read-only base image shared by ten thousand executions is sharing, and it is safe because no execution can modify it and no execution's data is in it. A pool is a legitimate design; chapter 7, on signing one off, works through what it takes. The argument is that sharing has to be a decision with a stated scope rather than a side effect of an optimisation, and that the technology label on the design document tells you nothing about which sharing decisions were made underneath it.

The pooled case admits a partial mitigation that is worth knowing about because it trades a quantity you can measure. Pinning instances to tenants — a VM, once it has run tenant 41's code, only ever runs tenant 41's code until it is destroyed — collapses the cross-tenant window to zero while leaving the cross-execution window intact for that one tenant. The cost is pool efficiency, since you now need warm capacity per active tenant rather than in aggregate. With 200 tenants and 60 warm VMs, a pinned pool either holds far more instances or accepts a cold boot whenever a tenant arrives without a pinned instance available, and the p99 of the feature becomes a function of how many tenants are concurrently active rather than of total throughput.

That last shift is the whole trade and it follows from one comparison. A pinned instance belongs to one tenant, so a pool of sixty instances can serve at most sixty tenants without a cold boot, and there are two hundred tenants. Whether the design works therefore stops depending on the arrival rate, which the unpinned pool absorbed comfortably, and starts depending on how many distinct tenants are active at the same moment. Below sixty, every request finds a pinned instance and the feature is as fast as it was. Above sixty, some tenant arrives to find nothing pinned for it and pays the full boot, and it pays it however quiet the system is overall — a pool at a fraction of its throughput capacity can still be out of instances for the tenant asking. The quantity you now have to forecast is a count of concurrent tenants, which is a harder thing to measure than requests a second and a worse thing to be wrong about, since it moves with your customers' working hours rather than with your own load.

It is also not a claim that removing all cross-session reachability makes the browser feature safe. Give each session its own container, its own uid, its own network namespace, and a session driven by an injected instruction still holds that user's live authenticated session to their bank's treasury portal, and can initiate a payment with it. That is the ambient-authority problem from topic 7.4, on architectural mitigations, where a component gets to act with whatever permissions its surroundings already carry rather than with permissions granted for the specific action, together with the tool-execution controls from topic 7.5, and the sandbox has no opinion about either. Isolation bounds what one principal can take from another; it does nothing about what a principal can be made to do with its own authority.

The narrowing is worth stating precisely, because it changes what you look for in a review. Isolation answers cross-tenant questions. It does not answer within-tenant questions, and it does not answer questions about the tools you deliberately handed the agent. Within that remit, the question to carry into a review is the pair above — which principals can name this, over what window — asked of each resource in turn rather than of the system as a whole, which is exactly the level at which "we use Firecracker" sounds like an answer.

There is one resource that no amount of per-session directories, per-session uids or per-session VMs partitions on its own, and it is the one every design in this chapter left completely open: the network. The browser container's sessions all shared an egress path, and so do all sixty VMs in the pool, and an attacker who can name nothing on the local filesystem can still resolve a hostname and open a socket. That is the third leg of the lethal trifecta from topic 7.3 — private data, untrusted content, and a way to communicate outwards — and turning it off is the subject of the next chapter, along with the harder question of how you prove it stayed off.