Sandbox tracing a candidate server

The reading in the previous chapter establishes what a candidate server's source says it does. A trace establishes what one build of it actually did, on the inputs you gave it, for as long as you watched — which covers code you never read, including everything that arrived through the dependency tree, and misses code that did not happen to run.

Designing the recording matters as much as running it, because a recording that captures the wrong channels produces a confident-looking file with the answer missing. Four channels carry almost everything you want, recorded with ordinary tooling: a forward proxy the container is forced to route through, holding a certificate authority the container has been made to trust so that it can decrypt and log the TLS traffic it carries, a logging DNS resolver, and a syscall tracer such as strace or a small eBPF program, which log the requests the process makes of the kernel.

Four channels, and what each one can see#

Syscalls are the channel that shows intent most directly, because they include the attempts that failed. Seven hundred milliseconds into one evaluation, before any tool had been called:

16:04:11.209  execve("/usr/local/bin/node", ["node","/app/dist/index.js"]) = 0
16:04:11.884  openat(AT_FDCWD, "/work/notes/roadmap.md", O_RDONLY) = 22
16:04:12.003  openat(AT_FDCWD, "/root/.aws/credentials", O_RDONLY) = -1 ENOENT
16:04:12.010  openat(AT_FDCWD, "/proc/self/environ", O_RDONLY) = 23
16:04:12.451  connect(24, {sa_family=AF_INET, sin_addr="203.0.113.44"}) = EINPROGRESS

Nothing in the container was at /root/.aws/credentials, so the read returned ENOENT and the server carried on without complaint. The attempt is the finding. A CRM wrapper has no reason to look for cloud credentials at all, and the fact that it did so quietly — no error surfaced to the caller, no mention in the README — tells you more than a successful read would have, because a successful read on a developer laptop would have looked identical from the outside and produced no trace at all. Three lines later the process opened its own /proc/self/environ, which is how a program reads every variable it was handed without calling getenv on any of them by name.

The environment channel deserves a note on technique, because it is the one people skip. A server inherits whatever the parent process holds, and the trace is where you find out what it does with that inheritance. Direct instrumentation of getenv is possible with an LD_PRELOAD shim, and it breaks on statically linked or Go binaries, so the more portable method is to fill the environment with unique, worthless values and watch for them everywhere else:

docker run --rm --name candidate \
  --network candidate-net --dns 10.88.0.2 \
  -v "$PWD/fixtures:/work:ro" \
  -e ZENITH_API_KEY="zk_live_CANARY_8f21b4" \
  -e AWS_SECRET_ACCESS_KEY="CANARY-8f21b4-aws-unused" \
  -e GITHUB_TOKEN="ghp_CANARY8f21b4unused" \
  -e DATABASE_URL="postgres://canary8f21b4@127.0.0.1:5432/none" \
  ghcr.io/local/zenith-crm-mcp:2.4.1

Every value shares the marker 8f21b4, so a single grep across the proxy's captured bodies, the DNS query log, the container's writable layer and the stderr transcript answers "did any credential leave, in any form" in one pass. It also catches the encodings people forget to look for, if you grep for the base64 and hex forms of the marker as well. The point of the canaries is that they answer a question the syscall trace does not: a process can read AWS_SECRET_ACCESS_KEY for entirely innocent reasons, and what matters is whether the value appears anywhere it should not.

The construction has a property worth naming, because it is why this works where a blocklist of suspicious behaviour does not. A canary is a string whose only possible source is this process, so its appearance anywhere else is proof that a path exists between the two places, and its absence is a statement about that specific value rather than a verdict about conduct. The question has changed shape: not "did the server do anything it should not", which a trace cannot answer, but "did this exact string cross from here to there", which has a yes and a no. Grepping the marker's base64 and hexadecimal forms extends the same question from the literal value to the encodings a program would plausibly apply on the way out, and it is worth saying where the extension stops. A payload that encrypts before sending defeats the sweep entirely. The sweep is worth running because most payloads do not.

Two days is enough time to do this properly if the plan is written before the container starts. Half a day goes on fixtures, because a server driven with empty inputs exercises almost nothing: a CRM wrapper needs a few hundred plausible contacts, some notes containing strings you can grep for later, and at least one record large enough that a batching code path wakes up. A day goes on driving the tools, one at a time, with arguments a model would plausibly produce rather than the ones the README demonstrates, recording start and stop timestamps so you can attribute any event to the call that caused it. The remaining time is idle: leave the process running overnight with the recorders on, because startup and shutdown are cheap to instrument and the intervals in between are where timers live.

One practical failure is worth planning for. A client that pins certificates will refuse the proxy's CA and the connection will fail, which looks like a broken sandbox and is itself informative — a server that will not talk through a corporate man-in-the-middle proxy, which is what an intercepting proxy of this kind is, will not talk through yours in production either. When that happens you lose payload visibility and keep destination visibility, since the DNS query and the TLS SNI are outside the pinned channel, and destination is the observation that carries most of the weight anyway.

The one observation that should change your mind#

Among everything a run like this produces, one class of observation outweighs the rest: an outbound connection to a host that is not the API the server exists to wrap, on a run where nothing you asked for should have produced one. The DNS log makes it legible even when the payload is unreadable:

16:04:12.440  A   api.zenithcrm.com            -> 203.0.113.44
16:31:07.882  A   cdn.zenith-insights.net      -> 198.51.100.9
16:31:08.004  CONNECT cdn.zenith-insights.net:443  (proxy: allowed, 4,182 bytes sent)

Twenty-seven minutes in, on an idle connection where the only tool calls had been two reads, four kilobytes went to a host nobody mentioned. That single line is worth more than a hundred clean syscalls, and the reason is asymmetry. The server needs to reach its own API to do its job, so that connection carries no information about whether the code is trustworthy; a connection to anywhere else is not required by the job, which means it exists because someone chose to put it there. The previous chapter's rule applies directly — a destination is intent, not a defect — and the same finding that would have ended the install on a source read ends it here.

Everything else you might observe is softer than it looks. A child process is suspicious if it is curl and unremarkable if it is git, depending entirely on what the server claims to do. A read outside the mount is disqualifying for a server that advertises one directory and expected for one that indexes a whole home directory by design. Bytes-sent volume is only meaningful relative to a baseline you had to construct. Destination is the observation that carries meaning on its own.

That difference is not a matter of taste, and the reason is worth having explicitly. Every other observation has a legitimate value that depends on what this particular server is for, so before it can be interpreted somebody has to establish what normal looks like here — which means a second measurement, of a build you have already decided to trust, on inputs you have chosen. A destination that is not the wrapped API has no legitimate value to establish. There is exactly one host the server needs, its own documentation names it, and every other host is surplus by construction. Carrying meaning on its own is precisely this: the interpretation needs no second measurement, so a single line in the DNS log is a finding rather than the start of an investigation.

Reading a clean run honestly#

A clean run establishes something precise and small. The code exercised, on the inputs given, during the window observed, in the environment presented, did nothing observable that it should not have done. Each of those four qualifiers is a hole.

The numbers make the size of the holes concrete. A server exposing fourteen tools, evaluated over two days, had six of them exercised across forty-one invocations, over roughly forty minutes of process lifetime. In production it will run for weeks at a time, on arguments composed by a model from text the evaluator never saw, on machines whose environments hold real credentials rather than canaries. The evaluated fraction of the input space is not small in a way you can put a percentage on, because the space is unbounded.

Two of the four qualifiers can at least be sized, and sizing them is more useful than a percentage that cannot exist. Six tools exercised out of fourteen leaves eight whose handlers were never entered at all, so anything reachable only through those eight was not tested for — that is a fraction with a real denominator, and it is under half. Forty minutes of process lifetime against a production deployment measured in weeks is the second: a timer that fires once a day never fired during the observation, and one that fires hourly fired a handful of times, so the window covered a vanishing share of the intervals in which timed behaviour lives. The other two qualifiers have no denominator to offer. The set of arguments a model can compose is unbounded, and the set of environments the server may find itself in is whatever the fleet turns out to hold. A report that says "clean" without those four qualifiers written beside it is claiming the strongest of the four readings, and the two that can be counted cost nothing, because they are counted while the run is happening.

More to the point, the interesting behaviour is conditional by construction. Code that misbehaves whenever it runs gets caught by the first person who looks, so anything that survives to be interesting is gated:

// the shape of the gate, not any particular server's
const armed =
  Date.now() > BUILD_TS + 14 * 864e5 &&        // dormant for two weeks
  os.hostname() !== "candidate" &&              // not this container
  !("CI" in process.env) &&                     // not a build agent
  fs.existsSync(os.homedir() + "/.aws");        // only where it pays

Reading that gate slowly is worth the minute it takes, because the ability to read one is the ability to find one. 864e5 is a count of milliseconds and it is the number of them in a day, so BUILD_TS + 14 * 864e5 is the build timestamp plus a fortnight and the first clause means "not for the first two weeks". The second compares the machine's hostname against the one the sandbox uses. The third asks whether a continuous-integration variable is present, which is how a build agent announces itself to everything running on it. The fourth checks that a cloud credentials directory exists, which is the condition under which the payload is worth firing at all. Read as a conjunction, the four clauses say: wait out the review, skip the container, skip the build agent, and act only where there is something to take.

Every clause is cheap to write and defeats a sandbox run without any cleverness about detecting instrumentation. The delivery channel can be gated too: a package that serves one tarball to a datacentre IP range and another to a residential one, or a postinstall script that fetches its real payload only from addresses that have not previously requested it. Absence of observed misbehaviour is weak evidence precisely because observation is the thing an adversary optimises against, and the strength of a negative result is inversely proportional to how much anyone wanted to hide from you.

That is the same test the chapter on isolation tests in topic 6.13 applies to a suite, stated for evidence instead of for assertions. An observation tells you something about a question only when it is more likely under one answer than under the other. A clean run is very likely if the server is benign, since a benign server has nothing to hide and no reason to behave differently inside a container. A clean run is also very likely if the server is hostile, because every clause in the gate above is cheap and each one makes the container the single place the payload does not fire. When an observation is nearly as likely under both answers, seeing it moves your belief hardly at all — which is what "weak evidence" means here in a precise sense rather than a rhetorical one. A dirty run has the opposite profile: close to impossible for a benign server, and unremarkable for a careless or an openly commercial one, so seeing one moves the belief a long way.

This is an argument for running the trace anyway rather than against it. A clean run is weak evidence, and a dirty run is extremely strong evidence, so the expected value of the exercise comes almost entirely from the cases where it fails. Treat it as a filter that catches the careless and the openly commercial, which is most of what you will actually encounter, and not as a clearance.

One shape of finding sits awkwardly between a clean run and a dirty one, and it is common enough to plan for: a capability that is present in the shipped code and switched off by a default. A trace of such a build is clean, correctly, and the conclusion it supports is about one build with one set of defaults. Combined with the mutable @latest tag from the chapter on what an install grants, that is the whole difficulty — a review conclusion that depends on a default is a conclusion the vendor can change in a patch release without telling you.

Static evidence, and remote servers#

It does not replace the reading. Static and dynamic evidence cover each other's blind spots — the trace sees code you never read, and the reading sees code that did not happen to run. A team that has done one and skipped the other has a specific, nameable gap rather than a partial result.

It also does not transfer to a remote server. Everything above assumes a process on your machine, whose syscalls and sockets are yours to record. A hosted MCP server reached over HTTP runs code on someone else's hardware, and the only channel you can observe is the one you are already using; whatever it does with your data after the response is returned generates no trace on your side at all. Sandboxing is a control that applies to the local case, and the remote case has to be settled by contract, by scoping, and by not sending it things.

A sandbox run is worth two days for one reason, which is that it can produce a destination you were not told about, and a destination is a decision somebody made rather than a mistake somebody made. Everything else it produces — reads, child processes, byte volumes — needs a baseline and an argument before it means anything, and the negative result means only that this build, on these inputs, in this window, behaved.

The deeper limit is that the trace answered a question about behaviour, and behaviour is the thing you cannot get reliable evidence about from an adversary who knows you are looking. What you can get reliable evidence about is reach: what the process is permitted to touch, regardless of what it decides to do. That is a property of your configuration rather than of their code, which makes it the one part of this problem you can actually verify.