Moderation you owe is separate from the refusal the provider gives

The email came from a customer's legal department, not from support. Their marketing lead had used the hero-image button on a pricing page, the generated illustration had included something close enough to a well-known cartoon mouse that a reader had noticed, and the page had been live for eleven days. The customer's question was straightforward and hard to answer: who had approved that image? The provider had not refused the prompt. The prompt had said "friendly mascot character, cheerful, cartoon style, waving", which is not a policy violation anywhere, and the model had drawn what that description most strongly resembles in its training data.

The team's first instinct was to file a ticket with the provider. That instinct is the mistake this chapter is about, because the provider had behaved exactly as documented. Their safety system exists to keep their model out of trouble that they are liable for, and it evaluated a prompt that was clean. Nothing in that system knows that the output is going onto a public pricing page belonging to a third company under that company's brand, and nothing in it was ever going to.

Two policies, two authors, two failure modes#

It helps to hold these apart by asking what each one is protecting. The provider's refusal protects the provider: it is calibrated to their jurisdiction, their public commitments, and their tolerance for a screenshot of their model's output in a newspaper. It is applied to what they can see, which is the prompt, any input images, and sometimes the output. It changes without a migration guide, both tightening and loosening, and it is deliberately conservative in ways that will annoy your users on legitimate requests.

Your policy protects your product and your customers. It covers the same subject matter but from the opposite direction, and the two overlap only partially. A prompt asking for a photorealistic image of a named public figure endorsing a product may sail through a provider's filter and be entirely unacceptable on a B2B marketing surface, business-to-business meaning the buyer is a company whose own legal department will read the page. A request to edit a photograph of a colleague, uploaded without their knowledge, is not a provider concern at all — the provider sees an ordinary photograph of an adult — and it is squarely yours.

The right way to think about the provider's filter is as the input validation on somebody else's server. You would not skip escaping user text in your templates because the database rejects some strings; the database's constraint is not your XSS policy, cross-site scripting being the injection of hostile markup through a field somebody else happened to validate first. The refusal is a floor beneath which nothing gets through, positioned by someone whose interests are adjacent to yours rather than identical.

What the input policy has to name#

A policy that says "no inappropriate content" cannot be implemented, so the useful exercise is writing the specific categories down and deciding the outcome for each. Four recur in every image feature and each has a different enforcement point.

Likeness of third parties is the one that surprises product teams, because it arrives through the editing path rather than the generation path. A background-replacement feature is a feature for uploading photographs of people, and a meaningful share of those photographs will be of someone other than the uploader. There is no reliable technical check for consent, so the practical controls are contractual and procedural: an explicit affirmation at upload, a record of who uploaded what and when, and a takedown path that a non-customer can use, since the person in the photograph is not your user and has no account.

Minors need their own rule rather than being folded into likeness, because the acceptable outcome is different. Generation involving minors, and edits applied to photographs of minors, are categories where "flag for review" is a poor answer, since review means a human on your team looking at the material. A hard refusal at the input gate, with no queue and no stored copy beyond what you need for an abuse report, is the ordinary policy.

Trademarks and named brands are what the opening incident was about, and they are the category where the input gate is weakest. The prompt did not name a brand. Blocking prompts that name brands catches the deliberate case and misses the incident entirely, so this is a category where the check has to be on the output and, for anything published beyond the requester, on a human.

The input gate itself is two mechanisms rather than one. Before a prompt is sent, a cheap check runs over the text for the categories where you have decided the answer is always no, which for most products is a short list and a regex or a small classifier — a model that takes one input and returns a score per category, "sexual: 0.02, violence: 0.71", rather than any text of its own — instead of anything elaborate. Before an upload is accepted, the user affirms in a checkbox that they have the right to edit the image and that it does not depict a minor, and that affirmation is stored with the upload record rather than discarded after the request. Neither mechanism is a real barrier to someone determined to get around it, and both are what makes an incident answerable afterwards, which is most of what a policy is for.

NSFW output — not safe for work, the standing industry label for sexual and graphic material — is usually well covered by the provider's own filter for generation, and much less well covered on the editing path, where a benign prompt applied to an uploaded image can produce something the prompt does not describe. Running your own output classifier is not redundant with the provider's here; it is checking a different artefact.

The check that has to happen before anyone else sees it#

The rule worth building the pipeline around is that a generated asset may be visible to the person who requested it before it passes your checks, and to nobody else until after. That single boundary makes the policy implementable, because it separates the fast path a user experiences from the slow path a publish requires.

private means visible to the requesting user in their own workspace, with a signed URL and no public route. published means the asset has left that boundary — embedded in a customer's site, sent in an email campaign, included in an export. Most assets never leave private, which is what makes a human review step on the in_review transition affordable: if 6% of generations are ever published, a reviewer is looking at 6% of the volume.

In the job pipeline from the previous chapter, this sits between storing the bytes and reporting success:

def finalize_generation(job, image_bytes: bytes) -> str:
    asset_id = store_asset(job.id, image_bytes)

    verdict = output_classifier.check(
        image_bytes,
        categories=["sexual", "violence", "minors", "known_marks"],
    )

    if verdict.max_score >= 0.85:
        set_visibility(asset_id, "quarantined")
        record_incident(job, verdict)          # no user-visible image
        return asset_id

    set_visibility(asset_id, "private")        # requester only
    return asset_id

The threshold is a product decision with a measurable cost on both sides, and it should be set from a labelled sample rather than from the default in the documentation. On a sample of 2,000 production generations from a marketing tool, a threshold of 0.85 quarantined 14 images of which 9 were genuine policy hits; dropping to 0.60 quarantined 61 of which 12 were hits. The second setting catches three more real problems and adds 47 false quarantines, each of which is a confused user and a support ticket. Which side you err on differs between a consumer app and a tool whose output goes onto customer websites.

Retention is a promise you cannot fully make#

Deciding how long you keep assets is straightforward until you notice that you are not the only holder. Providers commonly retain inputs and outputs for a period — thirty days is a widely used figure — for abuse monitoring, independently of your deletion. If your privacy policy says an uploaded photograph is deleted immediately on request, that statement is false in a way that matters, because the copy at the provider outlives your delete. Zero-retention arrangements exist on enterprise contracts and are worth asking for before you write the policy that assumes them.

The retrieval question is separate and is usually answered by accident. A support engineer debugging a complaint wants to see the image the user generated; that is a reasonable need and it means someone other than the user can open a private asset. Decide whether that is allowed, log it when it happens, and keep the log where the user can be told about it, because a background-replacement product holds photographs of people who never agreed to anything and an unlogged internal viewer is the part of that which is hardest to defend afterwards.

Labelling the output, and the limits of doing so#

Provenance metadata is the set of fields written inside an image file that record where the picture came from — which tool made it, from what, and when — travelling with the bytes rather than living in your database. It is cheap to add and worth adding. The two mechanisms in practice are C2PA Content Credentials, from the Coalition for Content Provenance and Authenticity, which attach a cryptographically signed manifest describing how the asset was produced, and the simpler field digitalSourceType set to trainedAlgorithmicMedia, defined by the IPTC, the International Press Telecommunications Council, for exactly this purpose and readable by most tooling as a plain metadata tag.

from PIL import Image
import piexif, io

def tag_generated(png_bytes: bytes, model: str, job_id: str) -> bytes:
    img = Image.open(io.BytesIO(png_bytes))
    meta = img.info.copy()
    meta["Description"] = (
        f"AI-generated image. source=trainedAlgorithmicMedia "
        f"model={model} job={job_id}"
    )
    out = io.BytesIO()
    img.save(out, format="PNG", pnginfo=_png_info(meta))
    return out.getvalue()

Be honest with yourself about what this achieves. Metadata is stripped by the ordinary operations of the web: a resize in an image pipeline, an upload to most social platforms, a screenshot. Anyone wishing to remove it can do so in one command. Its value is in the cooperative case — an internal reviewer, a customer's asset library, a downstream tool that surfaces the credential — and not as an enforcement mechanism. The record that actually holds is the row in your own database linking asset, prompt, model version, requesting user and timestamp, which is why the job table earns its keep a second time.

None of this is a content-moderation system in the sense a large platform means, and pretending otherwise is a way to overbuild. You are not classifying a public feed at scale; you are gating a pipeline you already control, at two points, with a human on a small fraction. It also does not address quality: a quarantine verdict says nothing about whether an image is any good, and an approved asset can still be unusable.

The pipeline now knows how to handle an image it should not serve. It does not yet know how to handle a request that produced no image at all, or produced one that fails on arrival, and those are not one case but several — each of which deserves a different answer, and one of which must never be retried.