Tracing output to its sinks

A trust boundary, in this setting, is any call where a value passes from a component that treated it as data into a component that will interpret it. The interpreter is what makes the boundary: an HTML parser interprets markup, a SQL engine interprets a query, a shell interprets a command line, a filesystem resolver interprets a path, an HTTP client interprets a redirect location. Each of these turns a string into behaviour, and the boundary is the call that hands the string across.

That is why "audit the model call" is the wrong instruction for the class of bug in this topic. The model call is where the untrusted string is born, and the string is inert there — a value in a variable, no more dangerous than a log line. It becomes dangerous somewhere else entirely, in code that usually belongs to a different team, was written before the model existed, and does not look like a security boundary. This chapter is about locating those crossings, which is a procedure you can run rather than a knack; what to do at each one is the next chapter's subject.

Data and code are the same bytes until something interprets them#

The reason the crossing is hard to see is that nothing changes about the string itself when it crosses. The bytes that spell out <img onerror=...> are identical whether they sit in a log file, a database column, or the argument to innerHTML. In the log they are data; a log viewer will show them as text. In innerHTML they are code, because an HTML parser will build DOM from them. The string did not change. What changed is the interpreter waiting at the other end of the call.

Stating that as a chain makes it checkable. A string is a sequence of bytes with no behaviour of its own; bytes do nothing until something reads them. An interpreter is a program that reads bytes and derives structure from them — elements, statements, arguments, path components. Deriving structure means some of those bytes stop being content and start being boundaries, which is the promotion. Therefore the promotion is performed by the reader, at the moment of reading, and cannot have happened earlier. Therefore the only place a control can prevent it is the call that delivers the bytes to that reader. Every rule in the next chapter is a consequence of that last line, and so is the reason a check placed anywhere upstream can be sidestepped: upstream code is guessing which reader will eventually be handed the string, and the sink knows.

The dangerous assumption is introduced at that call, and it is usually introduced by code whose author was thinking about formatting rather than interpretation. A template that builds an HTML fragment reads as "put the title here." A query builder that assembles a filter reads as "add this condition." A tool wrapper that turns a model's argument into a shell invocation reads as "run the command the model asked for." In each case the code is composing a string for an interpreter, and the composition step is exactly where a control belongs, but the code does not look like a security boundary. It looks like plumbing.

Tracing one response to every place it lands#

Take a report-generation agent as the worked example, because it spreads a single response across several sinks and shows the trace clearly. The model is asked to produce a JSON object describing a report:

{
  "title": "Q3 refunds for account <script>…</script>",
  "filename": "q3-refunds-../../etc/passwd",
  "filter": "region = 'EU' OR 1=1",
  "next_url": "https://reports.internal/done?x=//evil.example"
}

Now follow each field. The application code that consumes this object looks entirely reasonable:

resp = json.loads(model_output)

# 1. Title into an HTML email body
html = f"<h1>{resp['title']}</h1>"

# 2. Filename for the generated artefact
path = os.path.join(WORKSPACE, resp["filename"])

# 3. Filter fragment into a query
sql = f"SELECT * FROM refunds WHERE {resp['filter']}"

# 4. Redirect after the report is built
redirect(resp["next_url"])

There is no model call in this listing at all. The model call happened one line earlier, at json.loads, and produced a dictionary of strings sitting harmlessly in memory. Every dangerous thing in the program is on the four lines after that, and each is a trust-boundary crossing dressed as formatting. The f-string on line 1 crosses into the HTML parser of whatever mail client opens the email. os.path.join on line 2 crosses into the filesystem resolver, which will happily walk out of WORKSPACE following the .. segments. The f-string on line 3 crosses into the SQL engine, where OR 1=1 stops being a value and becomes structure. redirect on line 4 crosses into the browser's navigation, sending the user to a location the model chose. Four crossings, four interpreters, one response.

The belief that quietly protects this listing in a reviewer's head is that json.loads did something. It is a parser, parsing sounds like checking, and the call will genuinely fail on a malformed document — so it earns credit for rejecting things. What it rejects is a document that is not JSON. It says nothing about the contents of a document that is. Run the hostile object above and the benign object a well-behaved model would have produced through the same call, and json.loads returns successfully for both, with the same type, the same four keys, and string values that differ only in their characters. A check whose result does not move when the thing it is supposed to discriminate moves is not discriminating; it is a shape test that happens to sit near the danger. That is the general form of the mistake this topic keeps returning to, and it will appear again as a schema validator at a tool boundary.

The two middle lines are worth taking apart individually, because each is protected in a reviewer's head by a different piece of folklore. os.path.join is believed to keep a name inside WORKSPACE, and it does not. It composes strings with the platform's separator, and .. is an ordinary pair of dots to it, meaningful only later to the resolver. It also has a documented behaviour that makes matters worse: an absolute second argument discards the first entirely, so joining a workspace to a name that begins with a separator yields that name rather than a path underneath the workspace. The f-string on the query line is protected by nothing at all, and it looks protected for a different reason — the value it interpolates came out of a structured object through a named field, and a value that arrived through a named field feels vetted. The name is yours; the characters are the model's. Neither line is a security control that has been misconfigured. Neither line was ever attempting the job the reviewer is crediting it with.

The diagram makes the pattern visible: json.loads is a fan-out point, and every arrow leaving the dictionary is a place the string might be promoted from data to code. The controls do not belong at the model call, and they do not belong at json.loads, because the string is still data there. They belong on the four red edges, one per edge, each matched to the interpreter it feeds.

Enumerating sinks and marking where data stops being data#

The trace is a repeatable procedure rather than an inspiration, and it is worth doing the same way every time. Start at the model response and treat it as a single tainted value. Follow every assignment, every function call, every field access that carries any part of that value forward, and keep following until the value either reaches an interpreter or is discarded. Each place it reaches an interpreter is a sink. At each sink, mark the exact call where the string is handed across, because that call is where a control has to sit, and note which interpreter is on the other side, because the interpreter dictates which control is correct.

"Carries any part of that value forward" is the phrase doing the work, and it is worth making mechanical so that two engineers running the trace get the same list. A derived value is tainted if any of its bytes came from the tainted one. Slicing a string produces a tainted string. Concatenating, formatting or templating produces a tainted result even when the model's contribution is three characters in the middle of a hundred. Parsing produces tainted parts — the fields of a decoded JSON object are tainted, and so is a dictionary that holds them. Passing the value to a function taints whatever that function returns from it, which is why the trace crosses module boundaries rather than stopping at them. Two things end a branch. The value reaches an interpreter, in which case you have found a sink and you record it. Or the value is genuinely discarded — dropped, replaced by a value with no tainted bytes in it, or reduced to a decision that carries none of its characters, such as a length or a boolean. Anything else is still travelling, and the commonest way a trace misses a sink is an engineer stopping at a function call because the callee is somebody else's code.

The sinks that matter share a tell: the value flows into something that parses, executes, resolves, or navigates. A value that ends up in a log line, a metrics counter, or a JSON response body that the client will itself treat as data is not a sink in this sense, because nothing interprets it as code. Learning to tell the two apart is most of the skill, and it is why a generic "taint tracking" mindset — taint tracking being the practice of marking a value as untrusted at its source and following every value derived from it through the program — that flags every use of model output produces noise. The question is never "is this model output used here," it is "does this use hand the string to an interpreter."

There are a few call sites that hide the crossing well enough to be worth naming outright. String formatting is the most common: an f-string, a template literal, String.format, or a templating engine's raw-output mode all concatenate a value into a larger string that some interpreter will later read, and none of them announces that it is doing so. Serialisers are the second: a function that builds a shell command from an argument list, or that renders a config file from a dict, is composing input for an interpreter even though its name suggests it is just producing text. The operative property of serialisers is that they emit a grammar, and their names describe the output format rather than the fact that something downstream will parse it back. Query builders are the third: a method on an ORM (an object-relational mapper, the library layer that turns objects and method calls into SQL on your behalf) that accepts a raw fragment, or a helper that concatenates a WHERE clause, crosses into SQL at a call that reads like assembling a filter. Tool wrappers are the fourth and the most dangerous in an agent, because they exist precisely to turn a model's chosen argument into an effect — a file written, a command run, an HTTP call made — and the whole point of the wrapper is to interpret the model's string.

What the trace leaves behind should be one row per crossing, and three columns are enough: the call site as a file and line, the interpreter on the other side of it, and who owns the code. The interpreter column is what the next chapter turns into a control, since the grammar decides what "safe" means. The owner column is what turns the finding into work, because a crossing in a data-access layer another team maintains is a conversation and a crossing in your own view layer is a patch. Recording the sink without the interpreter produces a list nobody can act on, and recording it without the owner produces a list nobody does act on.

Downstream of the injection question#

This trace is not the same activity as prompt-injection analysis, where prompt injection means an attacker planting instructions inside text the model will read — a document, a ticket, a web page — so that the model follows those instructions instead of the ones you gave it. Conflating the two wastes effort. Topic 7.7, on exfiltration channels, dealt with the ways a compromised agent moves data out, and the injection topic before it dealt with how hostile instructions get into the model in the first place. Both of those are about influencing what the model produces. The trace here assumes the model has already produced something hostile, by injection or by accident or by a genuinely malicious retrieved document, and asks a strictly downstream question: given a hostile string in hand, which of your call sites will act on it. You can have flawless injection defences and still own every bug in this topic, because these bugs are about what your own code does with a string once it exists, not about how the string came to exist.

The independence runs in the other direction too, which is the part that makes this worth doing even in a product with no adversary in its threat model. None of the four crossings requires an attacker to have written anything. A model summarising a document about relative paths will write .. into a filename because the document was about ... A model titling a report on markup will put angle brackets in the title. A model asked for a filter over a field it has half understood will produce a fragment that is syntactically valid and semantically wrong, and the query will run. The sink does not check where the string's intention came from; it interprets the characters it is given. So the property the trace establishes — every crossing has a control matched to its interpreter — is what you need whether the hostile string arrived by injection, by accident, or by the model being ordinarily imperfect, and only the first of those three is addressed by anything upstream.

It is also not an audit that finishes. The trace is anchored to one model response in one version of the code, and every new consumer of that response adds an edge the old trace does not have. A field added to a prompt template, a new tool wrapper, a "small" export feature — each is a candidate crossing, and none of them shows up as a security change in a diff. Re-running the trace when the response grows a field is cheap; discovering the new sink from an incident is not.

Marking the edges tells you where the controls go; it does not tell you what they are. The four crossings in the report generator each need a different control, and the reason is not that four different libraries happen to be involved. It is that each interpreter parses by a different grammar, so "safe" means something different at each one — and for one of the four, the model composed structure rather than a value, which no amount of escaping can make safe. That is the subject of the next chapter.