Ageing example sets

Every example in a few-shot block is a claim about the current state of the product, frozen on the day somebody pasted it in. Some of those claims are about form — this is the shape of a reply, this is the register — and they age slowly. Others are about fact: this discount exists, this queue exists, this screen has this button. Those age at whatever rate your product changes, which for most teams is faster than anybody re-reads a prompt.

The failure this produces is unusually quiet. A stale claim does not make the answer malformed or off-topic, so the eval keeps passing; it makes the answer confidently wrong about something the eval never checks. And because the block also anchors the model's sense of what an input looks like, a shift in who your customers are degrades answers for the new ones while the aggregate, still dominated by the old ones, barely moves.

Neither failure has a culprit, and that is the part that makes them hard to look for. A wrong label has an author and a moment: somebody typed it, and the day they typed it is in the file's history. A stale claim was correct when it was written and became false somewhere else entirely, in a promotions table or a taxonomy or a redesigned screen that has no connection to the prompt at all. Nothing in the prompt file changed on the day it stopped being true. There is no event, no diff and no author, which is why the ordinary instincts — read the change, ask who wrote it, check the tests — have nothing to attach to.

What a retired claim does to current answers#

The mechanism is the one a wrong label follows, applied to an assertion rather than to a label. A demonstration is the operative specification of what a correct reply looks like, so the model reproduces its properties. Consider one example from a twelve-example block in a support assistant:

<example>
  <ticket>This is the third order this year that's arrived damaged.</ticket>
  <reply>I'm sorry — I've flagged this with our fulfilment team. As a
  frequent customer you're also eligible for our 15% loyalty discount on
  your next order; I've added the code LOYAL15 to your account.</reply>
</example>

The model does not know that LOYAL15 was a specific code with an expiry. It reads the example as evidence that replies in this task offer a discount when a customer has had repeated trouble, and generalises accordingly. Where the programme behind that code has been retired, the demonstration keeps producing the offer, and the damage accumulates at the rate of traffic rather than announcing itself.

"Generalises accordingly" is where the explanation usually stops, and everything else here follows from taking it apart. A language model scores the next fragment of text given everything in front of it, and the example block is part of what is in front of it. What a stretch of text supplies to the fragment that follows is a regularity rather than a set of facts: replies in this document acknowledge the problem, name an action already taken, and offer a remedy where a customer has had repeated trouble. There is no separate operation in which the model believes a claim, and nowhere for the product of such an operation to be kept. Continuing the document is the whole of what it does, and continuing this particular document means writing a reply carrying those properties.

So what gets reproduced is the property and not the fact. The model is not holding LOYAL15 as something it takes to be available; it is writing the kind of reply that ends with a code, and one string of that shape is as easy to produce as another. Two things follow from that, and the second is what lets the failure survive for the best part of a year. The first is that the offer reappears on inputs resembling the demonstration — repeat complaints, damaged goods, a customer with history — because those are the inputs on which the property was established, and a property is applied where it appears to apply. The second is that the claim is not stored. There is no field holding the discount for a migration to update, no row for a retirement to delete, and no cached value to invalidate. The regularity is re-derived from the block on every single call, and it will go on being re-derived, correctly, for exactly as long as the block says what it says.

None of that shows up in any metric such a team watches, and the reason is instructive. The eval scores whether the assistant's reply addresses the right issue and follows the format, which it does — the discount sentence is an extra clause on an otherwise good reply. There is no check that asserted policies exist. So the stale claim degrades output quality in a way that is invisible to accuracy and visible only in escalation transcripts, which nobody joins back to the prompt.

The general form of that is worth holding onto, because it explains the blindness rather than reporting it. A stale example does not degrade into malformed output that a parser rejects, or into an off-topic answer a reviewer flags, or into a wrong destination that a downstream queue bounces back. It produces a reply that is well formed, on topic, correctly routed, and false in one clause, which is the single combination every instrument in this topic is blind to. Nobody forgot to close that gap. Each of those checks asks a question about the shape or the destination of a reply, and whether a claim inside it is true is a different question that none of them was ever asking.

Staleness of register is the quieter half and works differently. If all twelve examples are self-serve tickets — two or three informal sentences, first person, no reference numbers, one issue each — then the block anchors the model's sense of what an input looks like and what register a reply uses. When enterprise traffic arrives that is structured — a purchase order number, a contract reference, three numbered questions in one message, written by someone in procurement who signs with a job title — twelve demonstrations of informal single-issue tickets are a fairly strong prior that inputs are informal and single-issue, which is why the assistant answers the first of three numbered questions and stops. Nothing was mislabelled. The block is simply anchored to a population that was 95% of traffic when it was assembled and is now 78%.

Why that half works differently is worth being exact about, because it is an effect the chapter on accuracy against example count already described from the other side. A long example block carries a distribution and the model reads it: that is majority-label bias, where the distribution was over labels and a block holding mostly one label pushed close calls toward it. Here the distribution is over inputs. Twelve informal single-issue tickets establish, as firmly as twelve of anything establish anything, that inputs in this task are informal and single-issue. A structured procurement message then arrives as an unusual instance of a well-established kind rather than as a kind of its own, and the continuation most consistent with the document is a reply of the shape it keeps showing — one issue, addressed, in the register of the twelve. Answering the first question and stopping is the model being faithful to the block it was given.

How little the aggregate moves while that happens is worth computing rather than asserting, because the number is smaller than most people guess. An accuracy figure over a mixed population is the average of its parts weighted by the share of the population each part holds — the same arithmetic the per-slice chapter used, where one eval's accuracy figure was taken apart into the named groups composing it. Take the figures this chapter's own case produces: a new cohort scoring a little over seventy while everyone else scores a shade over ninety, a gap of roughly twenty-one points. That gap weighted by the cohort's share is its entire contribution to the mean. At one customer in twenty the aggregate falls by about a point; at one in ten, by about two. A one-point fall on a metric whose repeat runs already differ by half that much is a bad week rather than an incident, and it gets attributed to a model version, a traffic mix, or nothing at all. The cohort has to reach something like a quarter of traffic before the aggregate moves by five points — and by then the account team has been complaining for two quarters.

Why deleting the one example passes#

The natural response to finding a stale claim is to delete the offending example, re-run the eval, and move on if the score holds. It will hold. On the block above, that change gave 91.5 against a previous 91.4, with a run-to-run spread of 0.4, which is a green tick and a merged pull request.

Read those two scores against the spread before reading them against each other. Repeat runs of a byte-identical prompt on this set differ by four tenths of a point, and the change moved the score by one tenth, which is a quarter of the movement the prompt produces when nothing about it is touched. That supports exactly one claim: no regression was measured. It supports nothing at all about whether the stale claim is gone, because the eval was never in a position to answer that question.

The score holds for three separate reasons, none of which is that the fix was sufficient, and they are worth taking one at a time because each fails in a different way.

The first is that the eval set cannot see the failing population. It was sampled from production traffic in the same era the examples were copied, so it contains no enterprise cases at all and cannot measure the segment that is failing. A metric computed over a set that excludes a segment is not reporting that the segment is healthy — it is reporting on a different population and saying nothing whatever about this one.

The second is that the metric has no predicate a false claim could fail. It scores routing and format rather than the truth of claims, so it would not notice the discount either way. A check can only return false on a property it evaluates, and whether an asserted policy still exists is not among the properties this one evaluates. Adding enterprise cases to the set does not repair that, since the check would score the new cases by the same two questions it scores the old ones by.

The third is that the deleted example was a sample rather than the defect. The block still has eleven examples assembled by the same person under the same conditions with no review, which an afternoon of reading turns up in detail: four reference a settings screen that was redesigned last year, two route to a queue name that no longer exists in the taxonomy, one uses a tone from before the brand guidelines changed, and three are genuinely fine. The single deletion treats a provenance problem — nothing anywhere records where each example came from, who checked it against the product, or when — as a bad row.

There is a general version of this that applies to every measurement in this topic. An eval set ages the same way an example set does, and the two age together when they were drawn from the same traffic in the same month. The zero-shot baseline, the example-count curve, the per-slice report, the leave-one-out ablation: all of them are conditioned on the eval set being a fair sample of what the system sees now. When it is not, all of those instruments agree with each other and with the prompt, and the whole apparatus is internally consistent and wrong. Refreshing examples on a schedule while the eval set stays frozen moves the problem rather than fixing it.

Tracing what each instrument rests on turns that from a slogan into something checkable. The zero-shot baseline reports what the block is worth as a difference between two scores computed on one set, so an unrepresentative set makes both scores wrong in the same direction and the difference between them a fact about a population you no longer serve. The count curve locates a knee by finding where added coverage stops paying, and the coverage being measured is coverage of that set. The per-slice report can only print rows for slices it holds cases for, so a segment supplying none of the cases has no row and is therefore incapable of regressing. The leave-one-out ablation names the example responsible for a slice's failure by removing it and re-scoring, which requires the slice to contain the cases that example is damaging. And the pricing arithmetic converts accuracy points into money at a rate drawn from what one error costs, which assumes the errors being counted are the errors being made.

Not one of those is broken. Each answers its own question correctly on the set it is handed, which is precisely why they agree with each other and with the prompt: they share a premise, so their agreement is evidence about the premise rather than about the answer. That is the version of this failure that is genuinely difficult, and the only thing that detects it is a measurement of the eval set itself against current traffic — which is a different kind of measurement from every other one in this topic, because its subject is the instrument rather than the system.

Provenance is metadata, not review#

Putting the prompt in the repository behind code review does not catch any of this. A reviewer reading a diff sees text that has not changed, and a reviewer reading the whole file sees a plausible reply mentioning a discount, with no way to know that the programme ended. The information that would have caught it is not in the diff; it is where the example came from, when it was last checked against the current product, and who is accountable for it. That has to be stored next to the example.

The shape of that argument is worth making explicit, since it is the reason a process change does not work here and a data change does. Review is a function of a diff: a reviewer reads what changed and evaluates it against a model of what the text means. On the day the loyalty programme ended, nothing in the prompt file changed, so there was no diff, so review had no input to run on. The fact that went stale is not in the file either. It is a relationship between a claim written in one system and a state held in another, and a relationship that is written down nowhere cannot be traversed by anybody, however carefully they read either end of it. Making it catchable means storing the link, and once the link is stored the check stops being an act of attention and becomes a join.

- id: ex-047
  added: 2024-11-03
  added_by: r.okonkwo
  source: production_reply/tkt-882134     # verbatim agent reply, redacted
  covers_slice: repeat_damage_complaint
  label_rationale: >
    Acknowledges, states the action taken, does not promise a timeline.
  asserts_policy: [loyalty_discount_v2]   # named claims made in the reply
  verified_by: r.okonkwo
  verified_on: 2024-11-03
  review_after: 2025-05-03
  owner: support-platform

Four of those fields do work no review process provides. covers_slice states the example's job, so when the slice is retired or fixed the example is a candidate for deletion rather than a permanent resident; an example with no slice is an example nobody can justify. asserts_policy names the factual claims the example makes, which turns policy drift into a join — when loyalty_discount_v2 is marked retired in whatever system owns product policy, a scheduled job lists every example asserting it, and this one surfaces on the day the programme ends rather than eleven months later. verified_on and review_after make staleness a state the CI can see, so a check that fails when any active example is past its review date costs nothing to run and converts an unbounded liability into a recurring twenty-minute task. And owner means the task has somewhere to go.

Read as a set, those four turn four open-ended questions into closed ones. Whether an example is still needed becomes a lookup against the current slice report rather than a judgement. Whether it is still true becomes a join against whichever system owns the fact rather than a reading task. Whether anybody has looked at it lately becomes a build failure rather than a memory. And whether somebody will act on the answer becomes a name rather than a hope. None of the four requires a person to notice anything, which is the property that separates them from a review process and the reason they keep working in a quarter when nobody is paying attention.

The states an example moves through are worth being explicit about, because "in the prompt" and "correct" have been the same thing by default:

The transition into active carries the requirement that a slot be justified by a measured per-slice delta rather than an argument, and the transitions into flagged are the three ways an example goes bad: time, an external fact changing, and evidence from the eval itself. A block of four examples generates perhaps six of these transitions a year, which is a small amount of process for the size of the failure it prevents.

What triggers a review is not only the calendar, and tying it to events catches things a date never would:

The row about the eval set is the one most often skipped, and it is the one that keeps everything else honest. When a segment arrives that is 22% of traffic and 0% of the eval set, the first work is not on the prompt; it is labelling two hundred cases from the new segment and adding a slice for them, after which the per-slice report will say what the prompt needs.

One mechanical check is worth building even before the metadata exists, because it finds the damage that is already in production. Take every factual claim made in the example block — the discount, the delivery window, the queue names, the settings screen — and write each one as an assertion with a query that would falsify it against a current source of truth. The discount becomes a lookup in the promotions table, the queue names become a comparison against the live taxonomy, the settings screen becomes a link somebody clicks. On a twelve-example block that exercise takes about an hour and returns seven claims of which three are false, which is a far better return than any amount of re-reading. The output also seeds the asserts_policy fields, so the hour is spent once.

Provenance is not a guarantee#

A field saying an example was verified in March is a claim by a human that they read it and believed it, and a busy engineer clearing a queue of flagged examples will extend review dates without reading carefully. The checks that resist that are the mechanical ones — the policy join, the CI failure on an expired date, the leave-one-out ablation that reports a slice delta per example — because they produce evidence rather than an assertion of diligence. Everything else depends on somebody caring, which is what the owner field is really for.

The general claim underneath that is worth carrying past this chapter. A field recording that somebody verified something is evidence about a process, and a process is a statement about what people did. A policy join, a build failure on an expired date and a per-example slice delta are evidence about the world, and they hold whether or not anybody was paying attention on the day they ran. Where the two disagree the second one wins, and a system carrying only the first is one distracted quarter away from carrying nothing.

It is also worth being clear that a review cadence does not detect a shift in who your customers are. The trigger table has a row for a new segment being onboarded, and that row only fires if somebody tells the team, which on a self-serve product with a growing enterprise motion is exactly the thing nobody thinks to mention. The durable version of that check is not a calendar entry but a composition report: the share of production traffic falling in each eval slice, computed monthly, so that a slice at 0% of the eval set and 22% of traffic is visible as a number rather than as a complaint from an account team.

The difference between those two is which of them needs somebody in the loop in order to fire. A calendar entry asks a person to remember; the composition report computes a share of traffic per slice and prints it beside the eval score, so the segment nobody thought to mention appears as a row with a small number in it. That check is also the one keeping every other instrument in this topic honest, which is the second time here that the eval set has turned out to be the thing to fix before the prompt.

An example set is a small artefact with a long life, and the thing that goes wrong with it is neither an accident nor a mistake anybody made. It is that the world moved and nothing in the system was watching. Provenance fields, a review date, a policy join and a monthly composition report are the whole of the remedy, and together they cost a few hours to build and about twenty minutes a quarter to run.

What makes that remedy necessary rather than fussy is the mechanism this chapter opened on. The block is read from scratch on every request, and from it the model derives, faithfully, whatever regularity it currently contains. No step degrades a stale claim, no threshold announces it, and no component anywhere has the job of comparing what the block says against what is true. The fidelity is the difficulty: a block describing a product from eighteen months ago will go on describing it perfectly.

Put end to end, the routine this topic describes is short enough to run on any prompt you own. Measure the task with no examples at all, on a held-out set with fixed decoding and a known spread, so that any later claim has something to be a claim about. Sweep the count and find where the curve stops paying, then price the block you are considering in money, in prefill latency, and in context you are taking from the user. Cut the eval into named slices and read the errors into them, and spend the example budget on the groups that are actually failing, including the ambiguous and malformed inputs nobody wants to put in a demo. Treat every example as a specification you are shipping, with a label somebody can defend and a claim somebody has checked. Where you automate selection, check the boundary slice specifically and build the fallback before you need it. Then put a review date on all of it, because the version of this prompt that is wrong in eighteen months looks exactly like the version that is right today.