Why a prompt in the application repo makes rollback a deploy

A prompt kept as a file or a constant in the application repository gets the four properties of the previous chapter: an author, a diff, an approval and a durable history. It also gets something nobody chose, which is the release's lifetime. The text ships when the service ships and it changes when the service changes, so the set of operations available for changing it is exactly the set available for changing code.

That identity is invisible while things are working and expensive the first time they are not. A deploy pipeline — the sequence of build, test and rollout stages between a merge and production traffic — is designed around the risk profile of code, and it applies that profile to a text constant with no ability to tell the two apart. This chapter is about what that costs during an incident, measured rather than asserted, and about the three ways teams try to undo a prompt change when the release is the only unit of undo.

Where the clock goes when the prompt is in the image#

The prompt in question is a Python constant, compiled into the container image along with everything else:

# app/agents/orders/prompts.py

ORDER_AGENT_SYSTEM = """
You are an order support assistant for Acme.

Tool selection:
When the customer supplies a tracking number, prefer lookup_shipment, since it
resolves carrier state directly. Use lookup_order when the customer refers to an
order by number or by the item they bought, and when you need line items,
payment state or refund eligibility. Only one lookup is usually required.
...
"""

Changing that string means shipping a new image, and a typical pipeline for a service of this shape takes fourteen minutes end to end:

$ gh run view 5512819 --json jobs
build image           4m12s   (deps cached; layers rebuilt from app/)
unit + contract tests 5m48s
eval suite (frozen)   1m30s
push to registry      1m04s
rolling deploy (12 pods, maxSurge 25%)  1m26s
                      ------
                      14m00s

None of those stages is obviously wasteful, and none of them is doing anything useful for a change that consists of restoring a bulleted list to a text constant. The eval suite is the closest to relevant, and it is running against a frozen set that already passed for the current text. The build is recompiling nothing, since Python, and is dominated by layer assembly and registry round trips. From the engineer's point of view all fourteen minutes are fixed cost paid to change a string.

Why it is fixed cost is a property of pipelines rather than a complaint about this one. A pipeline is triggered by a merge and parameterised by nothing else. It does not read the diff, it holds no classifier that could separate a Python function from a docstring from a text constant, and every stage in it was added because some class of change needed that stage — the build because code becomes an artefact, the tests because code has behaviour worth characterising, the registry push because the artefact has to get somewhere before it can run. Each is correct for the change it was added for, and each runs identically for every other change, because the alternative is a pipeline that decides which stages a diff needs and that is a classifier nobody wants to be wrong. The fourteen minutes is therefore not waste that better engineering would remove. It is the price of sending a text edit down a path built for code, and the only way to stop paying it is to stop sending it there.

There is a governance wrinkle sitting on top of the pipeline that teams discover during their first night incident. The branch protection from the previous chapter requires an approving review before anything reaches the main branch, and at two in the morning there is nobody to approve. Either the organisation has a break-glass path — an administrator override, an emergency label, a second on-call who can be woken for a rubber stamp — or the engineer discovers two minutes into the fix that it cannot merge. Both outcomes are bad in different ways: the override exists to be used and every use erodes the guarantee that production is reviewed, and its absence turns a fourteen-minute recovery into however long it takes to wake a colleague.

The erosion in the first of those deserves precision, because it is easy to file as a cultural worry. Branch protection's value is a universal claim: everything on main was reviewed by somebody other than its author. An override does not weaken that claim, it falsifies it, and a falsified universal claim is not repaired by being rare. What remains is a weaker claim — everything on main was reviewed except the commits where it was not — and the only way to act on the weaker one is to read the override log, which nobody reads until something has already gone wrong. That is the argument for designing the emergency path rather than improvising it. An override that records a reason, notifies the owning team and opens a follow-up review turns the weaker claim back into something checkable. One invented at 02:10 by whoever has the admin bit does not.

Three ways to undo it, and what each one drags along#

There are three plausible moves, and they differ less in speed than in what else they touch.

The third row is the fastest and the one most teams reach for, and it is the one that makes the schema question sharpest. A migration in the same release has already run — alembic upgrade head executed during the deploy, and the column and index exist in the database regardless of which image is serving. Rolling the image back therefore leaves the schema ahead of the code, which is fine for a purely additive migration and is not fine in general. If the release had dropped a column, renamed one, or backfilled a value the old code does not understand, the image rollback is now a correctness problem layered on top of a quality one, and the engineer discovering that six minutes in is discovering it in the worst possible circumstances.

The chain under that is short and every link is checkable. A migration is applied to the database, and the database is not part of the image; alembic upgrade head ran during the deploy, and its effects now sit in a system the deploy pipeline has no lever over. Redeploying the previous tag therefore restores the code and leaves the schema exactly where the migration put it, which is what the schema is ahead of the code means. Whether that is a problem is decided entirely by what the migration did. A nullable column and an index are invisible to code that never mentions them, so the old image runs unchanged and the rollback is clean. Dropping a column, renaming one, or rewriting values into a new encoding is not invisible, because the old code names the thing that is no longer there or reads bytes it no longer understands. The engineer at 02:10 is being asked to make that distinction from memory, about somebody else's pull request, before the rollback rather than after it.

The cherry-pick row deserves its own note because it is the option that looks most professional and behaves worst under pressure. It requires the prompt commit to be separable, which is a property of how the author happened to organise their work rather than something the on-call engineer can rely on — had the author also touched app/agents/orders/tools.py in the same commit, the revert carries code with it and the conflict resolution happens at 02:16 with no reviewer awake. It also routes around the approval machinery from the previous chapter: the fix that reaches production is a commit nobody reviewed, authored by the person with the least sleep, and the record of what production is actually running diverges from the main branch until somebody reconciles it on Monday.

The wrong granularity in both directions#

Reverting a release to undo a prompt is oversized and undersized at the same time, and both halves matter.

It is oversized because the unit of undo is the release, and the release contains a schema migration. Undoing a text constant should not be an action with a data-loss failure mode; the moment those two things share a rollback unit, every prompt rollback inherits the risk profile of the riskiest change in the release. That coupling grows with release cadence in the wrong direction: a team that deploys twice a day has, on average, half a day of unrelated change welded to any given prompt.

It is undersized because fourteen minutes of a raised error rate is a choice the mechanism forces on you. The offending change is forty lines of text with a known previous value, and there exists a version of this system in which restoring it is an operation measured in seconds. The engineer is not paying fourteen minutes because the change is complicated; they are paying it because the only lever attached to the prompt is the release lever.

The coupling also gets worse as a team gets better at shipping. A service that releases once a fortnight has an enormous amount of unrelated change welded to any prompt rollback, which is obvious enough that nobody is surprised by it. A team practising trunk-based development with ten deploys a day has a small release, which sounds like the problem shrinking, and it means instead that the prompt edit is one of three commits rather than one of forty while the fourteen-minute pipeline is unchanged. Continuous delivery reduces the blast radius of a rollback and does nothing at all about its latency, because the pipeline is the latency.

Faster pipelines and looser storage#

The two responses that follow naturally from that timeline are both worth working through, because both are attempts to buy the fourteen minutes back and neither changes the shape of the problem.

The first is to make the pipeline faster. Shaving it to six minutes by parallelising tests helps every deploy a little and leaves the coupling intact: the prompt still cannot move without a release, and the release still carries the migration. Teams that respond by optimising CI end up with a faster version of the same arrangement, and the cherry-pick is still branch surgery at 02:16.

The second is to move the prompt somewhere a deploy is not required to reach, and the intermediate step most teams try first is worth examining, because it gets halfway and stops in an awkward place. Putting the text in an environment variable or a mounted config map means it can change without a rebuild, and a config-map update plus a rolling restart takes about three minutes rather than fourteen. What that arrangement lacks is everything the previous chapter was about: the value now lives in a cluster resource that anyone with kubectl access can edit directly, the change has no diff against a reviewed baseline, the audit trail is whatever the platform's admission logging happens to keep, and no identifier ties the text serving at 02:20 to anything a reviewer approved. It is faster and less accountable, which is the trade this topic exists to avoid making. The stronger form of the same mistake is moving the prompt to a database row editable from an admin page, which trades a slow rollback for no rollback at all, since the row has one value and no history.

The sentence a team writes after making that change is the prompt is configuration now, so we can roll it back in three minutes, and the first half of it is true. What has gone missing is neither speed nor availability. It is the answer to which text was approved. A config map holds one value and no lineage, so an engineer at 02:20 can change what is serving and cannot compare what is serving to anything a reviewer ever looked at. The fast rollback and the reviewed artefact have become properties of two different systems, and the team now holds exactly one of them.

It also does not follow that a faster prompt rollback makes the incident short. Detection and decision were ten of the twenty-four minutes, and no storage mechanism touches those; a control plane that flips the prompt in three seconds still leaves you with a fourteen-minute incident if the alert window is five minutes and the on-call engineer needs six minutes to read the release. Fixing the rollback path moves the ceiling, and the floor is set by observability and by how quickly a human can attribute a symptom to a change.

Two teams with identical prompts, identical review standards and identical eval suites will have wildly different incident durations if one of them can change the serving prompt through configuration and the other cannot. Nothing about the quality of the prompt engineering distinguishes them, which is why this is worth treating as an architectural property rather than as a matter of care. It is also why the number is worth measuring rather than estimating: time a revert of a one-line prompt change through your own pipeline on a quiet afternoon, from branch to the last pod serving the new text, and the figure you get is the floor on every prompt regression you will ever have. Most teams guess low, because they are thinking about the edit rather than about the fourteen minutes of machinery the edit has to travel through, and because nobody has ever had a reason to time it while nothing was on fire.

What has to change is the relationship between where the prompt is authored and what the serving path reads at request time. Right now those are the same thing — the text in the image is the text the model sees — and that identity is what makes every prompt change a deploy. Breaking it, without giving up the author, the diff and the approval, is the subject of the next chapter.