Named prompt sections

A named section is a labelled region of a prompt that you wrote yourself — ## Rules, <redaction>, <output_format> — holding one kind of instruction. It looks identical on the page to the delimiters of the previous chapters and does an unrelated job: those mark where text from outside the system begins and end, this marks where one topic in your own text ends and the next starts.

What the convention buys is a property of review rather than of inference. A reviewer asked "does this prompt handle X" can open the section where X would live and answer from eight lines; if the section exists and does not mention X, the gap is a sentence they can point at, and if the section does not exist at all, that is louder still, because every other prompt in the repository has one. Without sections the same question is an open-ended search over prose, and a search that comes back empty supports no conclusion.

The absence looks exactly like a presence somewhere else#

Consider a reviewer trying to establish whether a summarisation prompt says anything about redacting personal data. They search for "redact" and find nothing; the prompt says "do not include". A second reviewer searches for "PII", the standard acronym for personally identifiable information, and finds nothing; the prompt says "identifying details".

Neither search failing tells you anything, because the space of phrasings is open, and after two failed searches the honest conclusion available is "I did not find it", which is not the same claim as "it is not there" and cannot be turned into one by reading harder. What both reviewers end up doing instead is reasoning about the intent of a paragraph they can see — a sentence about third parties mentioned in the transcript, say, which may or may not extend to the customer's own card number — because a prose prompt gives them nothing else to reason about. The argument then has no resolution procedure, and the way those arguments end is that somebody observes the question is unrelated to the change under review and approves it.

Compare what happens when the same prompt has a fixed set of named sections and one of them is ## Redaction. The question becomes a lookup: open that section, read eight lines, answer. That is the entire benefit, stated plainly — consistent headings create reviewable anchors, and an anchor is what converts an unbounded search over prose into a bounded lookup whose failure is informative.

Why the second failure is informative and the first is not comes down to a constraint on the writer, which is the step most descriptions of this leave out. A reviewer searching prose is guessing which words an author reached for, out of a set nobody bounded; a guess that misses tells you about the guess. A reviewer opening <redaction> is relying on something else entirely — that whoever wrote a redaction rule had to put it there, because the repository admits no other place for one and a check in CI says so. Absence from that section is then evidence about the prompt rather than evidence about the reader's vocabulary. Take the constraint away and the property goes with it: sections adopted as a reading habit, with authors free to write a redaction rule into <output_format> when it suits them, restore the unbounded search with extra steps. This is why the lint later in this chapter is not tidiness. It is the thing that makes a negative answer mean anything.

What the same prompt looks like sectioned#

Here is a prose summarisation prompt, abbreviated to its first third:

You are a summarisation assistant for the support team. Read the transcript
and produce a summary of no more than three sentences, plus an escalation
flag. Write in past tense and in the third person, and avoid speculation
about what the customer meant. Do not include identifying details about
third parties mentioned in the transcript. If the customer became abusive,
note that the conversation was difficult but do not quote the language used.
Summaries are read by team leads during weekly review and occasionally
pasted into incident reports, so they should stand alone without the
transcript. Where the customer asked a question that was not answered, say
so. Do not invent ticket numbers...

Every one of those sentences is defensible. Read as a whole they are eight different kinds of instruction — role, output format, style, redaction, tone, audience, coverage, a prohibition — arriving in the order they occurred to the four people who edited the file over nine months. There is nowhere for a ninth kind to go except the end, which is where every subsequent edit went.

The restructured version uses the six sections that became the house standard:

<role>
You summarise support transcripts for team leads.
</role>

<output_format>
Three sentences or fewer, past tense, third person, plus an escalation
flag (true/false). The summary is read without the transcript and must
stand alone. Do not invent ticket numbers or dates.
</output_format>

<redaction>
Remove payment card numbers, national identifiers, and full postal
addresses, whether they belong to the customer or to anyone else, and
replace each with a bracketed placeholder such as [card number].
Do not include identifying details about third parties the customer
mentions. Do not quote abusive language; note that the conversation
was difficult instead.
</redaction>

<coverage>
State any question the customer asked that was not answered.
Note unresolved commitments made by the agent.
</coverage>

<transcript>
{{transcript}}
</transcript>

<escalation_rules>
Set the escalation flag when ... 
</escalation_rules>

Nothing was rewritten. Sentences moved, one rule was added, and the card-number case now has an obvious home. The value shows up on the next review rather than on this one: a reviewer asked whether the prompt handles a customer's own identifiers opens <redaction> and answers in eleven seconds.

The right-hand column is what makes the standard hold. Without it, output_format silently becomes the section where anything formatting-adjacent lands, redaction rules migrate into it because a placeholder looks like formatting, and a year later you have prose again with headings on top of it.

Follow that drift one step at a time, because it is slow enough to look like nothing happening. A rule about replacing a card number with [card number] is genuinely about both redaction and formatting, so the author who files it under <output_format> is not being careless. The next author, looking for precedent, finds a redaction-flavoured rule already living there and adds a second. Within a few edits <redaction> is thin, <output_format> is long, and the section names have stopped predicting the contents. At that point the lookup fails in the worst available way: it comes back empty and the reviewer believes it, because the structure is still there and still passes every check that only counts sections. The right-hand column is the part of the standard that says where a rule with two natures goes, and it exists because that ambiguity is where the drift starts.

Six sections, and why adding a seventh should hurt#

The number is not magic and the specific names are not either. What matters is that the set is small enough that an engineer holds it in their head, and that it is the same set in every prompt in the repository. A reviewer looking at their fourth prompt this month should not have to learn a new layout; the whole return on the convention comes from the fact that <redaction> is in the same place in all of them and means the same thing.

That implies adding a section is a repository-wide change rather than a local one, and the friction is the point. When somebody proposes <tone> because their prompt has three sentences about tone that fit nowhere, the options are to file them under <output_format>, to argue that every prompt in the repo needs a tone section, or to admit that the standard is wrong and change it deliberately. All three are better outcomes than a seventh section appearing in one file, because a section that exists in one prompt and not the others is invisible again — a reviewer who does not know to look for it will not notice its absence anywhere else.

Rolling the standard out is the part teams get wrong by being too ambitious. A big-bang migration of forty prompts produces forty simultaneous behaviour changes of the kind the next chapter is about, all landing in the same week, with no way to attribute anything. Migrate on touch instead: the lint — a check in continuous integration that reads each prompt file and reports which of the agreed sections are missing, empty or out of order — runs in warn mode over the whole repository and in fail mode only over files changed in the current pull request, so a prompt gets restructured when somebody is already editing it and already has to test it. On the summarisation repository that took about five months to reach full coverage, which felt slow and produced exactly one behaviour regression rather than a fortnight of them.

Runbooks, ADRs and structured logging#

The pattern is older than prompts and it is the same pattern every time. An incident runbook has fixed headings — detection, immediate mitigation, rollback, comms — because at 3am the reader needs to find rollback without reading detection, and because a runbook with an empty rollback section is a visible defect during the review that happens in daylight. The author's own memory is not what the headings are for. An architecture decision record has context, decision, consequences, in that order, for the same reason: the consequences section being thin is the thing a reviewer is supposed to notice.

The closest analogue in code is the move from ad-hoc log lines to structured logging. print(f"failed for {user} after {n} retries") and log.warn("retry_exhausted", user_id=uid, attempts=n) carry the same information to a human reading one line. They differ entirely in what can be asked of a corpus of them — the first supports grep with a guessed phrase, the second supports a query. Prose prompts and sectioned prompts stand in the same relation, and a prompt repository is a corpus.

That comparison is worth pushing on, because it is exact in one place and misleading in another. Where it is exact: the second line names its fields in advance, so a question like "how many retry exhaustions did this account see" is a filter rather than a search over phrasings, and the answer "none" is trustworthy for the same reason the empty <redaction> section is — the emitter was constrained to use that field name. Where it misleads: structured logs are consumed by a query engine that fails loudly on a field that does not exist, and prompt sections are consumed by a person. Nothing in the reading of a prompt errors when a section is missing. The lint is what supplies the missing half of the comparison, and a team that adopts the structure without it has the ad-hoc log lines wearing field names.

What the sectioned form replaced, in most teams, is not a different structure but an implicit one that lived in someone's head. The person who wrote the original prompt knew there was a redaction rule in paragraph three, and while they were on the team the ambiguity never surfaced because reviews routed through them. Structure is what that knowledge looks like when it is written down.

Headings do little to the model#

The tempting error is to believe the headings do something to the model. They mostly do not, and treating them as behavioural is how teams end up with <CRITICAL_RULES_DO_NOT_VIOLATE> sections that read as shouting and buy nothing.

Section structure does have a modest effect on model behaviour — a rule under a heading is easier to attend to than the same sentence buried mid-paragraph, and the next chapter is entirely about that effect being real enough to break things. But the reason to do it is the review property, and if you justify structure to your team on behavioural grounds you will lose the argument the first time someone A/B tests it and finds a two-point difference on an eval.

Losing that argument is not bad luck, and it is worth seeing the shape of it in advance. An argument made on behavioural grounds is settled by a behavioural measurement, and the behavioural effect here is small, which means it sits close to the noise any eval carries and can land on either side of zero depending on which cases the suite happens to contain. The review benefit, meanwhile, is not measured by that suite at all — it shows up months later, in a question answered in seconds rather than argued for two days, on a prompt nobody has touched since. So a team that justified the convention behaviourally has staked it on the one number that will not support it, and will roll the convention back on evidence that was never about the reason they adopted it.

The related error is thinking the structure is self-enforcing. It is not, and the fix is a test:

REQUIRED = ["role", "output_format", "redaction", "coverage",
            "transcript", "escalation_rules"]

SECTION = re.compile(r"<(?P<name>[a-z_]+)>\n(?P<body>.*?)\n</(?P=name)>",
                     re.DOTALL)

def test_prompt_structure():
    for path in PROMPT_DIR.glob("*.txt"):
        found = {m.group("name"): m.group("body") for m in SECTION.finditer(path.read_text())}
        assert list(found) == REQUIRED, f"{path}: sections {list(found)}"
        for name, body in found.items():
            if name != "transcript":
                assert body.strip(), f"{path}: <{name}> is empty"
        # nothing outside a section
        stripped = SECTION.sub("", path.read_text()).strip()
        assert not stripped, f"{path}: text outside any section: {stripped[:80]!r}"

Twenty lines, and it turns three separate conventions into things CI enforces: every prompt has all six sections, in the same order; none of them is a hollow placeholder; and no rule has been appended to the bottom of the file where nobody looks. That last assertion is the one that pays for the test, because appending to the end is what everyone does under time pressure, and it is exactly the edit that recreates the original problem.

It is worth seeing why that third check has to be written the way it is, since the obvious version does not work. Checking that the six sections are present cannot detect an appended rule, because the six sections are still present — the rule is somewhere else. What the test does instead is delete every recognised section from the file and assert that what remains is empty, which is a check on the complement rather than on the parts. That inversion is the whole trick, and it generalises: any convention that says "everything of this kind lives in one of these places" is enforced by looking at what is in none of them. Without it, the file's end is the one region of the prompt the standard does not cover, and it is also the cheapest place to type.

Measuring whether any of it helps is worth an afternoon. Six engineers were given four prompts from the repository — two in the old prose form, two sectioned — and five factual questions about each, of the form "does this prompt say anything about X". On the prose versions they answered 61% correctly with a median of 48 seconds per question; on the sectioned versions, 93% with a median of 11 seconds. The interesting number is not the speed. It is that on the prose prompts, four of the fourteen wrong answers were confident assertions that a rule existed when it did not, which is precisely the failure mode this chapter is about.

Two conversions make those results say more than they appear to. Read the accuracy as an error rate and the gap grows: getting sixty-one percent right means getting a bit under two in five wrong, and getting ninety-three percent right means getting about one in fourteen wrong, so the mistakes fell by something like a factor of five rather than by the thirty-two percentage points the correctness figures show. Read the timings as a ratio and forty-eight seconds against eleven is a bit over four times faster, which is the difference between a question a reviewer answers and a question a reviewer defers. Neither conversion is the point on its own. The point is the kind of error underneath them: a wrong answer that asserts a rule exists is a reviewer who will approve the change, and the reason prose produces that error is exactly the reason a failed search over prose supports no conclusion — a reader with nothing to look up reasons from intent instead, and intent can be read either way.

A topic boundary is not a provenance boundary#

The headings here do a different job from delimiters around a payload, and the two get conflated because they look identical on the page. A <transcript> tag marks a provenance boundary — everything inside came from outside the system, and that fact has consequences for escaping, for testing, and for what an attacker can do. A <redaction> heading marks a topic boundary in text you wrote entirely yourself, and no payload will ever be rendered into it. The first needs collision analysis; the second needs a naming convention. Applying the collision-scanning discipline to your own section headings is wasted effort, and applying a relaxed attitude to a payload boundary is the early-closing region of two chapters ago.

Structure is also not a check on correctness. A <redaction> section containing a rule that is wrong, or contradicts something in <output_format>, passes every test in this chapter and reads beautifully. Anchors make omissions visible; they do nothing about rules that are present and mistaken, and a team that adopts sections and then stops reading the sections has traded one blindness for another.

So the claim is narrow and worth keeping narrow. A fixed set of named sections does not make a prompt better, shorter, or more obeyed. It makes a specific question — is the rule about X in here — answerable in seconds instead of arguable for two days, and it makes the answer "no" a thing a reviewer can establish rather than merely suspect. That is worth a lint and five months of migrating on touch.

And it is not free to adopt on an existing prompt. Moving sentences into sections changes nothing about their wording, which makes it feel like a formatting change that could not possibly alter behaviour. That intuition is wrong in three specific ways, and the next chapter is about what evidence you should demand before believing a restructure was behaviour-neutral.