Auditing a prompt's referents

A stale prompt does not fail the way stale code fails. Code that references a deleted function stops compiling; an instruction referencing a deleted tool, a renamed field or a label set that has since grown two values keeps running, keeps being obeyed exactly, and keeps producing output that passes every validator downstream of it. The instruction is not wrong in any way that reading will reveal — it is wrong about the world, and the world is in a different repo.

That makes prompt staleness a resolution problem rather than a comprehension problem. The move is to stop reading a stale prompt for sense and start resolving it for referents: for each instruction, name the thing outside the prompt that it depends on, name the live source of truth for that thing, and check whether the two still agree. What follows is the shape that check takes, the class of defect it catches, and the specific reason a healthy-looking metric is often the strongest evidence that something is wrong.

An instruction and the thing it points at#

Here is a classifier prompt and the schema its output is validated against.

# prompts/ticket_classify.md, line 12
Classify the ticket into exactly one of the following categories:
billing, account_issue, bug_report, how_to, other.
Return only the category name, lowercase, with no other text.

// contracts/ticket_label.schema.json, current
{ "properties": { "label": { "enum": [
  "billing", "account_issue", "bug_report", "how_to", "other",
  "security_incident", "data_deletion_request" ] } } }

The instruction names five labels; the schema defines seven. The model is not confused by this. It reads "please delete my account and every record you hold about me", finds no category in its list that fits, and does what the instruction tells it to do, which is pick one of five. The label it returns is a lowercase string from the permitted set, so the schema accepts it, the router accepts it, and the ticket goes to general support.

Every observable in that diagram reports normal. The work gets done, because a deletion request routed to general support is read by a human within a few hours and that human answers it — agents are competent and the message is unambiguous to a person. What does not happen is the deletion queue's clock starting, and that queue exists because a statutory response window applies to those requests and somebody has to be able to prove it was met. The failure is not that the work went undone; it is that it went undone in the one dimension nobody was watching.

The metric deserves its own paragraph, because it is the part that keeps a defect like this alive. An accuracy panel of this kind reports agreement between the model's label and a gold set of hand-labelled tickets, and that gold set was sampled when the taxonomy had five values. The confusion matrix rendered on the panel, which counts each predicted label against the label a human assigned, is 5×5. There is no row for data_deletion_request, no column for it, and no case in the gold set carrying it, so a ticket of that class cannot be scored as wrong — it can only be absent. The number is not reporting that the classifier is fine; it is reporting agreement on the subproblem the classifier still knows about. A metric computed against the same assumption the prompt encodes cannot see a defect in that assumption, and the longer both hold still, the more confident the number looks.

Why an absent class cannot be scored wrong is worth one more step, because it is what makes the panel silent rather than merely inaccurate. Scoring accuracy means comparing a predicted label against a reference label and counting the agreements. A ticket has a reference label only if a human assigned one, and the humans who built this gold set assigned labels from the taxonomy as it stood then. So a deletion request that is in the set carries whichever of the five old labels a person picked for it, and the model choosing the same one is recorded as agreement. A deletion request that is not in the set contributes to neither the numerator nor the denominator. There is no arrangement of the data under which the panel's number can fall. The defect is not that the metric is computing something wrong; it is that the defect lies outside the metric's domain entirely.

That shape recurs well outside prompts. It is the same as a monitoring check that pings the health endpoint of a service which returns 200 whenever the process is up, or a foreign key that is documented in the entity-relationship diagram and not enforced in the database. In each case the check and the thing being checked share a premise, so the check goes green precisely when the premise fails. Backend engineers usually meet it first as a test that asserts against a fixture generated by the code under test.

Stated as a step, the reason agreement between two such artefacts carries no information is short. A measurement tells you something about the world only if the world could have made it come out differently. An instruction and a metric written at the same time from one understanding, neither touched since, stay consistent with each other whatever the world does, because nothing connects either of them back to it. Their agreement is therefore guaranteed rather than observed, and a guaranteed observation has no content. That is also the exact reason the fixture-generated test keeps passing after the code stops being correct.

Ranking defects by whether the output looks wrong#

A stale prompt usually carries several defects at once, and they are not equivalent. Ranking them is what a review will actually argue about, and the column that decides the order is not blast radius or cost — it is whether the output the defect produces is distinguishable from a correct one.

The label instruction goes first, and the reason is the third column rather than the first. The model is obeying it, the obedience produces a well-formed label, and nothing downstream can flag a well-formed label as wrong. Every other defect on the list either shows up in a number somebody looks at or does nothing at all.

Enumerating the referents#

The generalisable move is that almost every instruction points at something outside the prompt — a set of labels, a tool name, a field, a document type, a policy, a threshold, a team, a queue. Those referents live in systems that change on their own schedule.

The enumeration is manual once and cheap thereafter. Number every instruction, write down its referent kind and value in a small table beside the prompt, and let a test resolve that table against the live sources on every build.

import json, pathlib

SCHEMA = json.loads(pathlib.Path("contracts/ticket_label.schema.json").read_text())
LIVE_LABELS = set(SCHEMA["properties"]["label"]["enum"])
LIVE_TOOLS = {t["name"] for t in json.loads(pathlib.Path("agent/tools.json").read_text())}
LIVE_FIELDS = set(json.loads(pathlib.Path("contracts/ticket_payload.json").read_text()))

# maintained by hand, one row per instruction, reviewed with the prompt
REFERENTS = [
    {"rule": 3,  "kind": "label_set", "value": ["billing", "account_issue",
                                                "bug_report", "how_to", "other"]},
    {"rule": 11, "kind": "tool",      "value": "pricing_lookup"},
    {"rule": 18, "kind": "field",     "value": "ticket.body_text"},
    {"rule": 24, "kind": "policy",    "value": "POL-114", "reviewed": "2024-02-11"},
]

def unresolved():
    for r in REFERENTS:
        if r["kind"] == "label_set" and set(r["value"]) != LIVE_LABELS:
            yield r, f"drift: {LIVE_LABELS ^ set(r['value'])}"
        if r["kind"] == "tool" and r["value"] not in LIVE_TOOLS:
            yield r, "tool not in registry"
        if r["kind"] == "field" and r["value"] not in LIVE_FIELDS:
            yield r, "field not in payload contract"

def test_prompt_referents_resolve():
    problems = list(unresolved())
    assert not problems, "\n".join(f"rule {r['rule']}: {why}" for r, why in problems)

Three details in that test carry its usefulness. The label check compares sets rather than lists, because the order the labels are written in is not part of the contract and a list comparison would fail on a reordering that means nothing. The ^ operator is a symmetric difference: it reports the values on each side that are missing from the other, which is what makes the failure message actionable — a rename appears as one value gone and one value arrived, where a bare "these do not match" sends somebody to compare two enum lists by eye. And the live sets are built by reading the contract files themselves rather than by being retyped into the test, so the assertion fails in the build of whatever repository holds the referent table on the day the contract it points at changes. That last property is the entire point. The failure lands somewhere a person is looking.

Existence resolves, meaning does not#

Run against that classifier as inherited, the referent test fails on three rows in about eight milliseconds, and it would have failed on the label row the day the trust-and-safety pull request merged, in the build of a repo the authors of that pull request never touched. The policy rows cannot be resolved automatically, so they carry a review date instead and the test warns when one is older than two quarters, which converts an unbounded question into a calendared one.

What the pass resolves is existence, so it catches an instruction pointing at something that is gone and misses an instruction pointing at something that is still there and now means something different. A tool keeping its name while its arguments gain a required field, a policy document at the same identifier that counsel tightened in March, a label that survived a rename in the schema and quietly changed which tickets belong to it — all of those resolve cleanly and all of them are the same class of defect. Nothing mechanical closes that gap; the review date on policy rows is the crude substitute and it works only because somebody eventually reads the row.

Keeping the table honest is its own small discipline, because a hand-maintained list of referents drifts from the prompt in the ordinary way: somebody adds an instruction and not a row, and the test keeps passing on the rows that are there. The cheap guard is to assert the row count against the instruction count, so that adding a rule without a referent entry fails the build with a message naming the rule number. That is a weaker check than resolving the referent and it is the one that keeps the stronger check attached to reality.

It also says nothing about value. An instruction can resolve perfectly and still be redundant, still be in conflict with three others, still be the reason the prompt has saturated. The referent pass is orthogonal to the conflict hunting, bisection and provenance work of the earlier chapters in this topic, and it is cheaper than all of them, which is a reason to run it first on a prompt you have just inherited.

Which is where this ends up in practice. Take a prompt you did not write, number its instructions, and write the referent table — an hour on a thirty-rule prompt, and the first pass usually returns two or three rows that nobody can resolve. Put the test in the build, so that the next team to change a schema in a different repository breaks a test rather than a statutory clock.

Then look at whatever the prompt's health metric is and ask what it is computed against. A prompt and its metric are usually written at the same time by the same person from the same understanding of the system, which is exactly the condition under which they share a premise. The metric that has been steady for a year is the one most likely to share a premise with the instruction that has been wrong for a year, and stability is the symptom rather than the reassurance.