A flag that reads a mutable row cannot tell you what was served

The escalation-rate panel for the summarisation feature had two lines on it, split by the flag new_summary_prompt. For the first four days of February the treatment line sat at 2.6% and the control at 3.0%, on about 4,000 summaries a day in the treatment arm, which was 10% of the 40,000 the service produced. That is the shape of a modest win, and on the 5th someone put it in the weekly update. On the 7th the treatment line was at 4.1% and stayed there. The obvious reading is that the new prompt regressed, so the flag went off on the 10th, and the escalation rate for those accounts settled at 3.4% instead of returning to 3.0%.

That panel is a rollout being watched. Release and rollout is the part of shipping that begins after the code is already on the machines: deployment puts a new version of the service into production in an inert state, and release is the separate act of turning the new behaviour on for some set of users, at a pace somebody chooses, with a way to take it back. The switch doing that here is a feature flag, a named value the running service reads on every request and uses to decide which path that request takes. new_summary_prompt is one of those, and the two paths it selects between are the old summarisation prompt and the new one. Configuring it for 10% of accounts puts those accounts in the treatment arm and leaves the other 90% as the control arm, whose behaviour is meant to be unchanged so that the gap between the two lines on the panel can be read as an effect of the new prompt rather than of the week it was measured in.

Nothing about that sequence is explicable, because the flag was never identifying a prompt. It was identifying a database row, and two different people had edited the rows behind it during the window. On the 4th, someone tidying the stable prompt removed a paragraph about a deprecated policy_ref field, which is why the control arm moved. On the 6th, the author of the candidate prompt got feedback that the summaries read curtly to enterprise accounts and added two sentences about tone, which is why the treatment arm moved. Neither edit produced a new artifact. Both were UPDATE statements against rows the service reads on every request.

Flags earn their place on prompt and model changes because those changes cannot be finished before they ship. The evaluation suite from topic 3.16 runs the cases somebody thought of, production supplies the ones nobody did, and the practical question becomes how few accounts have to meet a change before its problems surface, together with how fast it can be withdrawn once they do. You meet the mechanism in three shapes: a hosted flag service such as LaunchDarkly, Unleash or Flagsmith; a table in your own database that every pod polls on a timer; or, for prompts specifically, a label in a prompt registry that you repoint from one version to another. Whatever the shape, the operations it owes you are the same — enable the change, target who receives it, and disable it again without shipping anything — and on new_summary_prompt all three worked as documented. What went wrong sits on the far side of the flag, in what the service did once the flag had answered.

The read path that destroys the experiment#

Here is the code, which is small enough that the problem is easy to miss when reviewing it.

def build_summary_prompt(account_id: str, doc: Document) -> str:
    channel = (
        "candidate"
        if flags.is_enabled("new_summary_prompt", account_id)
        else "stable"
    )
    row = db.fetchone(
        "SELECT body FROM prompts WHERE name = %s AND channel = %s",
        ("summary", channel),
    )
    return row["body"].format(document=doc.text, locale=doc.locale)

And here is the authoring tool's save handler, which is the other half of it.

def save_prompt(name: str, channel: str, body: str, author: str) -> None:
    db.execute(
        "UPDATE prompts SET body = %s, updated_by = %s, updated_at = now() "
        "WHERE name = %s AND channel = %s",
        (body, author, name, channel),
    )

The flag evaluation is correct. The bucketing is stable — the same account lands in the same arm every time — and the rollout percentage is honoured. What the flag returns is a boolean about which row to read, and the row's contents are a mutable global that anybody with access to the authoring tool can change at any moment, including in the middle of a measurement window. Topic 4.10 dealt with how prompts get stored and reviewed; the failure here is downstream of that, in how the running service binds to one. A prompt can live in a properly reviewed repository and still be delivered to production through a read path like this one.

The consequences are worth separating, because they are three distinct failures that people tend to collapse into one complaint about "the experiment being messy".

The treatment moved under the experiment. Accounts in the 10% received one artifact from the 2nd to the 6th and a different one from the 6th onwards, and both periods are aggregated into a single line on the dashboard labelled with the flag's name. The 4.1% figure is a blend of two prompts in unknown proportion, which means it is not a measurement of either.

The control moved too. This is the part that surprises people, because the mental model of a flag is that "off" means "the way things were". Off means whatever the stable channel currently contains, and the stable channel is edited on the same schedule as everything else — copy fixes, a new field in the output, a line removed because a downstream consumer changed. Turning the flag off on the 10th did not restore the 3.0% baseline, because the artifact that produced 3.0% stopped existing on the 4th.

The regression cannot be attributed. Given a specific bad summary from the 8th, there is no way to recover the prompt that produced it. The row has been overwritten twice since, and even if the authoring tool keeps an audit trail of edits, reconstructing "what was in the row at 14:22 on the 8th" is an inference from timestamps rather than a fact recorded on the request. Chapter 6 goes into what has to be recorded per request for that reconstruction to be a query; the point here is that no amount of logging on the request side helps if the thing being logged is the string stable.

This arrangement is not the result of carelessness, which is why it is so widespread. The mutable row exists because someone wanted a prompt change to take effect without a deploy, and a table the service reads on every request is the shortest path to that. It was probably built before there was any flag at all, when there was one prompt and editing it in place was an accurate model of what was happening. The flag arrived later, as a way to make the change gradual, and it was layered on top of the existing read path because rewriting the read path was not what the ticket was about. Every step is locally reasonable, and the combination is a system that cannot answer the only question anybody asks after a regression.

The cost is easiest to see in what the team actually did next. Somebody spent most of the 11th in the audit log of the authoring tool, matching edit timestamps against the escalation series, trying to establish which of the two moves was the prompt and which was seasonal. The conclusion, written into the incident notes, was that "the tone edit on the 6th probably caused the regression, though the stable-side change on the 4th makes the comparison unreliable". That sentence is the output of the whole investigation, and it is an assertion rather than a finding. The version of that system where the same investigation takes ten minutes is not a version with better dashboards; it is one where the request records what it was served.

Immutable artifacts and a pointer that moves#

The fix is a change to what "publish" means. An edit inserts a new row rather than replacing an existing one, versions are numbered and never reused, and the flag carries a version identifier rather than a channel name.

CREATE TABLE prompt_versions (
  name        text    NOT NULL,
  version     integer NOT NULL,
  body        text    NOT NULL,
  body_sha256 text    NOT NULL,
  created_by  text    NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (name, version)
);

REVOKE UPDATE, DELETE ON prompt_versions FROM app_writer;

The revoke matters more than the schema. Immutability that depends on everyone remembering to insert instead of update is not immutability, and the authoring tool is the place where the shortcut is most tempting, because "fix a typo" feels like it should not consume a version number. It should. Version 34 with a typo and version 35 without one are two different artifacts, and the four hours of traffic that saw the typo are four hours you can name.

The flag configuration then stops being a boolean and becomes a small payload:

{
  "flag": "new_summary_prompt",
  "unit": "account_id",
  "salt": "new_summary_prompt.2026-02",
  "default": { "enabled": false, "prompt": "summary@31" },
  "rules": [
    {
      "match": { "segment": "canary_accounts" },
      "enabled": true,
      "prompt": "summary@34",
      "percentage": 10
    }
  ]
}

and the read path resolves a version rather than a channel:

def resolve_summary_prompt(account_id: str) -> ResolvedPrompt:
    decision = flags.evaluate("new_summary_prompt", unit=account_id)
    name, version = decision.payload["prompt"].split("@")
    body = prompt_cache.get(name, int(version))  # immutable, cacheable forever
    return ResolvedPrompt(
        name=name,
        version=int(version),
        body=body,
        enabled=decision.enabled,
        assignment_reason=decision.reason,  # "rule:canary_accounts" | "default"
    )

Two properties follow from this that are easy to undervalue. The cache is now correct without invalidation, because summary@34 means one sequence of bytes forever, so a process can hold it for its lifetime and a rollback cannot be defeated by a stale entry. And the resolved object carries the version out of the function, which is what lets the request log record a fact rather than a channel name.

An edit under this scheme publishes summary@35 and leaves summary@34 in place. The flag payload still says summary@34 until somebody changes it, so the tone fix on the 6th does not silently enter the running experiment; entering it requires a deliberate edit to the flag, which is a reviewable, timestamped change with an author, and which everyone understands to end the current measurement window and start a new one.

A moving pointer is still useful — summary@production as a label that points at a version is how most prompt registries expose this, and Langfuse's prompt management builds instant rollback out of exactly that, a label you repoint without redeploying. The label is safe as long as two things hold: the versions it points at are themselves immutable, and each request records what the label resolved to at the time it was served. A label without recorded resolution is the mutable row again, wearing a better interface.

What each arrangement can answer#

The difference is easiest to see as a set of questions you will eventually be asked, usually by someone who does not care about your storage layer.

Where the boundary of this actually is#

Immutability buys attribution, and attribution alone. It does not make a rollout safe: summary@34 can be catastrophically worse than summary@31, and a flag that selects it correctly will deliver the catastrophe to exactly 10% of accounts with perfect fidelity. What changes is that you can name the thing that did it and return to the thing that did not.

It is also not the same problem as prompt versioning in source control, which topic 4.10 covered, even though the words overlap. A team can have every prompt in git with a review requirement on the directory and still serve production from a table that a separate authoring UI writes to, and in practice that arrangement is common, because the review process was designed for engineers and the authoring UI was built so that non-engineers would not need one. The runtime binding is what determines what was served. If the deployed service reads a mutable location, the repository history is documentation of intent rather than a record of fact.

There is a second thing it does not solve, which is that an immutable prompt is only half of what a request binds to. The rendered prompt that reaches the model is the template plus whatever the template interpolates — the document text, the locale, the retrieved passages, the tool schemas the runtime appends. Two requests can carry summary@34 and produce completely different byte sequences at the provider, and one of those byte sequences can be the one that fails. Version identity gives you the part of the input that you control and change deliberately, which is the part that release management is about, and the rest has to be captured separately.

Independent enablement, targeting and disablement are the three operations the flag has to support separately, and it is worth checking that they really are separate in whatever system you use. Enablement is the switch itself. Targeting is which units are in scope, and it needs a stable unit and a per-flag salt so that two flags at 10% do not select the same 10% of accounts and confound each other. Disablement has to be usable by someone who is not the person who set the targeting, at three in the morning, without understanding the rule structure — which usually means a single enabled: false at the top level that short-circuits every rule below it.

What none of this addresses is the case where the change you are rolling out is not one thing. The flag above selects a prompt version, and prompts are not the only artifact a request binds to at runtime. The next chapter takes the same flag and puts a model change behind it as well, which sounds like a small extension and turns out to remove your ability to fix the problem you are most likely to have.