Convenience becomes approval when nobody names an owner

Here is the timeline of a promotion that nobody authorised. On 20 March the invoice PoC finished and, unlike the one in the previous chapter, it passed. On 27 March an AP supervisor asked whether the engineer could "just run last week's batch through it, since it's already set up", and he could, in about twenty minutes. On 3 April he put it in a cron entry on the analytics VM so he would stop being asked. In late April someone in AP built a spreadsheet that pulled from the output table. By the end of May the month-end close checklist had a step that read "check extraction output before keying exceptions", and two clerks had been moved onto other work.

On 14 June the engineer was on holiday when the cron job died on a malformed attachment. Nobody was watching, because there was nothing to watch with; the failure was discovered fourteen hours later when the AP supervisor noticed an empty table, and month-end closed two days late with 1,240 invoices keyed by hand at short notice. The incident review asked who owned the service, and the answer was that nobody did — the engineer had been reassigned in May and had not been told the job still mattered, and his manager had never been asked to staff it.

At no point did anyone decide to put this in production. Each individual step was helpful, small, and correct in isolation, which is what makes the pattern so hard to interrupt: there is no moment where a reasonable person should have said no.

Production is defined by dependence, not by deployment#

The mental model that fails here is the one where production is a place — a cluster, a namespace, an environment with a deploy pipeline in front of it. Under that model the invoice job was never in production, since it ran on an analytics VM from a personal cron entry, and its output landed in a table nobody had reviewed.

The model that matches reality is that a system is in production when a business process depends on it. By that definition the promotion happened somewhere in late April, when the spreadsheet started pulling from the output table, and it was complete in May when two clerks were reassigned. The deployment mechanics were never the question; the dependency was. Everything organisations put around production systems — on-call, change control, capacity planning, backups — exists to manage dependence, and dependence is created by users, not by pipelines.

What was missing was never a code review. It was a moment where a specific named person was asked whether they would carry this, with the answer written down. Ownership is not a formality attached to systems that are already important; it is the mechanism by which a system's importance becomes visible to the organisation before an incident makes it visible. The absence of an owner is also why every other control was absent, because alerting, runbooks and capacity are things owners ask for and unowned systems never acquire.

Separability is what keeps the decision available#

The control that would have worked is structural rather than procedural. If, on 27 March, running last week's batch had required a credential the engineer no longer had, the conversation would have been "who authorises this and who will run it" rather than twenty minutes of convenience. That is the entire design goal: make the experiment something that cannot quietly become a dependency, so that the decision to promote has to be made explicitly by somebody with the standing to make it.

Four separations do most of that work, and all four are cheaper to set up on day one of a PoC than to retrofit on day sixty.

-- Provisioned with the PoC, not after it. Ten lines, one day-one decision.
CREATE ROLE poc_invoice_extract LOGIN PASSWORD :'pw' VALID UNTIL '2026-03-30';

GRANT USAGE ON SCHEMA ap_readonly TO poc_invoice_extract;
GRANT SELECT ON ap_readonly.supplier_invoice_index TO poc_invoice_extract;
-- deliberately absent: any GRANT on ap.* , any INSERT/UPDATE anywhere in ap

CREATE SCHEMA poc_invoice;
GRANT ALL ON SCHEMA poc_invoice TO poc_invoice_extract;
COMMENT ON SCHEMA poc_invoice IS
  'EXPERIMENT. No SLA, no backups, dropped after 2026-03-30. Do not build on this.';

ALTER ROLE poc_invoice_extract SET statement_timeout = '30s';

VALID UNTIL is the line that does the work, because it converts a policy into a mechanism. On 30 March the credential stops working whether or not anyone remembers the policy, and the AP supervisor's request on 27 March runs into a wall three days later that produces exactly the conversation that never happened. Renewing it is trivial and that is fine — the point is that renewal is a request somebody makes to somebody else, which leaves a trace, rather than a default that persists through inattention.

The same principle applies to the scheduled run. A prototype job that is going to exist at all should be built to stop on its own.

# poc/schedule_guard.py — imported at the top of the batch entrypoint
import datetime as dt
import sys

REVIEW_DATE = dt.date(2026, 3, 20)
GRACE_DAYS = 10

def guard() -> None:
    today = dt.date.today()
    if today <= REVIEW_DATE + dt.timedelta(days=GRACE_DAYS):
        return
    sys.exit(
        f"PoC window ended {REVIEW_DATE}. This job is unowned and has no SLA.\n"
        f"To keep it running, a promotion decision is required: see "
        f"docs/promotion-checklist.md and name an owner in decisions/."
    )

Fifteen lines, and the failure mode where a prototype outlives the experiment by fourteen months becomes an error message pointing at the decision that has to be made. The message matters as much as the exit code, because the person who hits it at 06:00 on a Monday is often not the person who wrote it, and what they need is the name of the document that unblocks them.

None of this is expensive. Provisioning a separate service account, a poc_ schema on a replica and a repository takes under a day for most teams, and it is a day spent at the point in the project where nobody is under delivery pressure yet.

Most organisations already have several of these systems and no inventory of them, which is worth checking before the next incident supplies one. The query that finds them is not about code: list the scheduled jobs and long-lived service accounts whose credentials were issued to an individual rather than a team, then ask for each one whether a business process would notice if it stopped. On a team of thirty engineers that search usually returns somewhere between two and six jobs, at least one of which somebody depends on. Each of those is a promotion decision that was never made, and the cheap remedy is the same conversation held late — name an owner, or turn it off and see who complains, which is an unpleasant but fast way to establish whether the dependency is real.

The demand that sounds right and does nothing#

The instinctive response, when a prototype is heading for production, is to demand a rewrite: proper structure, test coverage, code review, error handling, before anything ships. It is the wrong lever, for three reasons that are worth separating.

It is easy to refuse, because it reads as an engineering preference during a conversation about business value, and "we don't have six weeks" is an answer that ends the discussion with the prototype still running. It is also easy to grant in a way that changes nothing: a team can spend four weeks tidying the prototype, add unit tests for the JSON parsing, get two approvals on the pull request, and arrive at a well-structured unowned system that still runs on curated inputs with no alerting, no cost cap and no one on call. Every problem in the June incident survives that rewrite untouched. And the demand misdescribes the risk, because prototype code being messy is a maintenance cost that grows slowly, while prototype code being unowned and unbounded is an operational exposure that arrives all at once.

The demands that matter are about ownership, reversibility and the conditions the system runs under. Whether the code is pretty is a question for whoever ends up maintaining it, and that person cannot be identified until the ownership question has been answered anyway.

What separability is not#

It is not a claim that the prototype code is bad. Plenty of PoC code is perfectly reasonable — the invoice extractor was four hundred lines with a retry, a schema validator and a test — and the argument here would be identical if it had been written to the highest standard the team is capable of. The problem is structural rather than qualitative, which is precisely why a quality remedy does not address it.

It is also not a veto, and treating it as one is how engineers lose this argument. The goal is not to prevent the system from ever reaching production; the invoice extractor was worth promoting, and the two clerks who moved to other work represented real value that the company got early. What separation preserves is the ability to make that call as a decision, with someone's name on it, at a moment when saying no is still possible. An engineer who uses these controls to block things will find them removed; an engineer who uses them to force a conversation will find them tolerated, because the conversation is one most sponsors are willing to have when it is asked for clearly and early.

There is a real cost, and it is worth being honest about it rather than pretending the controls are free. Read-only access to a replica means the PoC cannot measure write-path latency into the ERP, which is one of the five criteria from chapter 1, so the integration spike needs a separate, time-boxed, supervised window against a staging instance. Expiring credentials will occasionally expire in the middle of something. These are the costs of keeping the experiment an experiment, and they are small next to two clerks reassigned against an unowned cron job.

Refusing promotion by default is only the defensive half of this. Once the conversation actually happens — and the point of all this machinery is to force it to happen — somebody will ask what the prototype has to become before it is a product, and the answer had better be a list rather than a mood. Which items belong on that list, and which two you keep when you are told you can have two, is the question the next chapter settles.