Policy text in context is not a control that holds

The finding that started this came out of a routine log review. An agent-authored change to the checkout error path had shipped three weeks earlier, and the line that did the damage looks like a hundred other lines in the codebase:

logger.error("checkout failed", {
  userId: user.id,
  email: user.email,          // ← customer PII, now in the log index for 90 days
  cartId: cart.id,
  reason: err.code,
});

Nobody wrote that maliciously and nobody caught it in review, because the reviewer was looking at the retry logic. The email addresses of about 4,000 customers sat in a searchable log index for three weeks. An email address is personally identifiable information, usually shortened to PII: data that identifies a particular human being, which an organisation makes explicit promises about in its privacy notice and to its regulator, and which nobody promised would be sitting in a search index that every engineer on call can query. The security engineer who found it opened a proposal the same afternoon: put the organisation's data-handling policy — 900 tokens of it — into every repository's config file, so that this cannot happen again.

What the proposal gets right#

It is worth being explicit about this, because the rest of the chapter takes the proposal apart and an engineer who has just handled an incident deserves better than a list of objections. The proposal identifies a real recurring failure rather than a one-off; a search of the last quarter's agent-authored diffs found four more log statements carrying customer fields, none of which had been flagged. It puts the guidance where the code is written rather than in a wiki nobody opens. And it names a norm that had, until then, existed only in the heads of three people, which is strictly worse than a written one. The instinct — the failure is systemic, so the fix must be systemic — is correct.

The defect is in the mechanism, and it is a specific one: text in the context window changes the probability of a behaviour, and the proposal is being sold as making the behaviour impossible.

Measure the residual, because it is not zero#

That claim is testable, and the test is worth running before the meeting rather than after. Take a task shaped like the one that failed — the checkout handler swallows errors from the payment provider; add structured logging so we can debug the 402s — and run it 200 times with the policy present and 200 times without, then grep the resulting diffs for customer fields in log calls.

The policy works. An 89% reduction is a large effect, larger than most config entries produce, and any argument that pretends otherwise is dishonest. What it does not do is reach zero, and the fourth row is the reason. When the task carries a competing demand for attention — a stack trace to read, a flaky test to chase, a deadline stated in the prompt — the violation rate climbs back toward a third of the untreated rate. The instruction is one of several thousand tokens competing to influence one decision, and on the run where something else wins, the agent writes the same line it would have written with no policy at all.

Scale changes what a 1.5% residual means. This organisation merges roughly 4,000 agent-authored pull requests a quarter, of which perhaps a fifth touch a logging path — so 800 opportunities, at 1.5%, is about twelve leaked-field incidents per quarter, down from a hundred. Twelve is a much better number than a hundred and it is not a number anyone would accept if it were stated out loud in the proposal, which it never is, because the proposal is phrased as "so that this cannot happen again".

Two failures the incident would not have prevented#

The residual is the first failure mode and the easier one to argue about, since it is visible in the table. Two others matter more, because the incident that motivated the proposal contains no evidence about either.

The policy covers the vocabulary it names. It says never log email addresses, phone numbers, physical addresses, or payment identifiers, which is a good list, and the leak that happens next month reads logger.info("checkout failed", { ...user }) — an object spread that includes an email the agent never typed. Or it reads logger.error(\payment declined for ${user.contact}`), where contact` is a field on an internal type that nobody classified as PII when it was added. In both cases the agent is not violating the sentence it read; it is writing code that the sentence did not anticipate, which is the ordinary condition of prose rules meeting real code. Every incident produces a policy that would have caught that incident.

The second is a distribution problem. Nine hundred tokens copied into every repository's config file is not one policy; it is thirty-one copies, and copies drift. Six months after the rollout, four repositories have a locally edited version — someone added a carve-out for a service that logs an internal customer ID that is not really PII — and two are still on the pre-amendment text because their config files were forked from an older template. The source of truth is now ambiguous in exactly the way that matters during an audit. Nothing about the original incident tells you this will happen, and it always does.

The belief is more expensive than the tokens#

The cost that outlasts all of these is what the proposal does to the team's reading of its own evidence. Once a policy is in every config file, the team believes the class of failure is handled, and the next violation is read against that belief. It arrives as a surprise, gets classified as a one-off, and gets a one-off's response: a fix to that line, maybe a note in the retro. The correct classification is that it is the expected residual of a probabilistic control, running at about the rate the measurement predicted, and the correct response is to ask what deterministic control should sit underneath.

That misreading compounds, because it also suppresses the follow-up work. A team that thinks the problem is solved does not build the redacting logger. A team that knows it has a 1.5% residual on 800 opportunities a quarter has a number to argue for engineering time with. The 900 tokens are cheap; the belief they buy is what costs.

Which is why trimming the policy to 300 tokens is the wrong kind of compromise, even though it is the one that gets proposed in every version of this meeting. It sounds proportionate, it addresses the cost objection, and it keeps "the important rules" always present. The table above shows what it actually buys: a 2.0% residual instead of 1.5%, for a third of the tokens. The trim is a sensible answer to a question about cost, and cost was never the objection. Both the 900-token and the 300-token versions rest on the premise that the guarantee comes from the text being in context, and preserving that premise more cheaply preserves it.

The same confusion produces the hybrid#

Chapter 4 left two staff engineers waiting on evidence about a 60-line API versioning section, and this is where the evidence comes back ambiguous. The skill fires on 78% of API tasks. The config version costs 60 tokens per session — 40 after tightening — and reaches an unglamorous 14% of sessions by the chapter 1 test. Neither prediction was right. Neither engineer is wrong.

The answer both of them reach for, within about ninety seconds of seeing the numbers, is to do both: a two-line summary of the versioning rule in the config file, and the full 60 lines in the skill. It sounds like the synthesis. It is two mechanisms with one owner, and it has the same defect as the duplicated review checklist from chapter 5, arriving by a more respectable route. The summary and the skill will be edited at different times by different people; in a year they will disagree; and the config file's version, being the one that is always present, will win the disagreements it is wrong about.

The hybrid also disguises the question the measurement was supposed to answer. Either the two lines are sufficient for the common case, in which case the skill is not needed for the common case and its 78% is a fact about a rarely-needed fallback, or the two lines are not sufficient, in which case the config entry is a comforting fragment that makes the 22% of API tasks the skill misses harder to see — because the agent has something about versioning in front of it, produces plausible output, and nobody notices the fuller rule never arrived. Deciding which of those is true is a day's work: run the API tasks with only the two lines and see how many come out right.

What the disagreement reveals is that the team's decision rule is a taxonomy. One engineer classified the content as constant-relevance and therefore config; the other classified it as a workflow and therefore a skill. Both arguments are about which category the section belongs to, and neither is about a measurement, which is why the evidence coming back in the middle left them with nothing to say. A rule that sorts by measured reach and measured firing has an answer for the ambiguous case — 14% reach is too low for always-loaded, 78% firing is too low to rely on alone, so the real work is raising the firing rate — while a rule that sorts by category can only produce a compromise between two labels.

What this chapter does not claim#

None of this says that standing instructions do not work. Chapter 1's billing invariant works, the correlation-ID line from chapter 4 works, and the security policy works to the tune of an 89% reduction. Text in context is a genuinely effective instrument for shifting the distribution of what an agent produces, and a team with no written norms produces worse output than a team with them.

The claim is narrower: a probabilistic instrument cannot deliver a guarantee, and PII in logs is a domain where the organisation has promised a guarantee — to its customers, in its privacy notice, and to a regulator. Using influence where enforcement is required is a category error, and it stays a category error at 900 tokens, at 300, and at three well-chosen sentences.

That leaves the security engineer holding an open incident and one fewer plan than she arrived with, which is not an acceptable place to end. The replacement is not a better-worded policy.