Mislabelled demonstrations

A mislabelled demonstration is an example in a few-shot block whose answer is not the answer your taxonomy prescribes. In a training set of 50,000 rows, one such row is statistically invisible; gradient descent averages it against every other row that disagrees with it, and nobody notices. A few-shot block does not average. The twelve examples are the entire evidence the model has about how your organisation maps inputs to labels, and it treats them the way you would treat a specification.

Both halves of that contrast are worth working out rather than accepting, because the instinct they overturn is a correct instinct almost everywhere else. Training adjusts the model's weights, and it does so by walking the rows and nudging the weights a little in whatever direction would have made each row come out right. One mislabelled row buys one nudge. Every row that disagrees with it buys a nudge the other way, and on a well-defined taxonomy with fifty thousand rows in it there are a great many such rows. The weights that survive the run are the residue of all of those pulls added together, so a single dissenting row moves the finished model by something on the order of its share of the corpus, which is one part in fifty thousand. Nobody notices it because there is almost nothing there to notice.

A prompt has no step that corresponds to any of that. Nothing in a request writes anything back: the loop reads the weights, scores the candidates for the next fragment of text, appends one, and reads the same weights again, so the model is byte for byte the same after the call as it was before. The twelve examples are not fifty thousand rows shrunk down to twelve. They are part of the document the model is being asked to continue. There is no corpus for a wrong label to be one part in, and there is no opposing pull either, because the eleven correct examples do not contradict the twelfth — they are about other inputs, and they agree with it by saying nothing wherever it speaks. The arithmetic that made the training case harmless has no counterpart here, and the wrong label arrives at the decision point carrying its full weight.

That difference is what makes annotation errors in a prompt a distinct failure rather than a smaller version of a familiar one. It also changes what the error looks like from outside: not a slightly higher error rate spread across everything, but a specific group of inputs going consistently to a specific wrong place while the aggregate barely moves. Recognising that shape is most of the diagnosis, because the shape is unlike anything model variance produces.

One wrong label is a rule, not a data point#

The model looks for the simplest rule consistent with what it has been shown. Eleven examples describe a taxonomy that mostly makes sense, and the twelfth says that a request for money back after returning an item is a billing_question. The rule it infers is not "one of these is a typo". It is something closer to "requests about getting money back go to billing", and it applies that rule to every input that resembles the demonstration.

Calling that an inference is the part worth slowing down on, because the model is not reasoning about your taxonomy. It is doing the one thing it does, which is scoring the candidates for the next fragment of text given everything before it. At the moment the label is due, what sits above is twelve inputs each followed by a label, and then a thirteenth input followed by nothing. The continuation that scores highest is the one most consistent with the pattern the document has already established, so the question the scores are effectively answering is: what mapping would make all twelve of those pairs come out the way they did? A mapping that files money-back requests under billing answers that question. So does "eleven of these are right and the twelfth is a typo", and that second answer never gets reached, because nothing in the document marks any example as less authoritative than any other. There are no line numbers, no confidence fields, and no record of who typed what or how quickly. Twelve pairs of equal standing admit the simplest rule covering twelve, and the simplest rule covering twelve is not the simplest rule covering eleven with an exception bolted on.

Stated plainly, the wrong label is not competing with the right ones at all. It is filling in a region of the input space that none of them speaks about, and it fills it in the only way the document leaves open. What catches people out is that the identical artefact — one labelled example, wrong — really is harmless in the place they are used to meeting it, so the instinct that says one bad row in twelve is noise is the same instinct that is correct about one bad row in fifty thousand. The two situations look the same on the page and differ entirely in what is done with the row. Averaged, it is a rounding error. Read as a specification, it is a clause.

The consequence is a concentrated, directional error rather than a scatter. Reading the eval per slice shows it immediately:

slice                  n    acc   errors   most common confusion
ALL                  400   88.7       45
en_single_issue      210   96.7        7   product_quality → delivery (3)
en_multi_issue        54   85.2        8   billing → delivery (4)
pt_br                 48   87.5        6   refund_request → billing_question (4)
caption_only          36   83.3        6   product_quality → delivery (2)
refund_request        24   54.2       11   refund_request → billing_question (11)

Eleven of the twenty-four refund cases are wrong and all eleven are wrong in the same direction, into the same bucket. That is the signature, and it has three parts worth learning to read together. The errors concentrate on inputs resembling one example; they all point at a single wrong label rather than spreading across the label set; and the model is not hesitant about them — sampling those cases at temperature 0.7 five times gives the same wrong label five times, because the model is not confused, it is following the rule it was given. Model variance looks nothing like this. Variance produces errors spread across slices, pointing in different directions, and flipping between runs.

The three parts of that signature each follow from the rule, and seeing them as three consequences of one cause is what makes the shape usable as a diagnosis. Concentration follows from the rule having a condition attached: it fires where the input resembles the demonstration and nowhere else, which is why the refund slice absorbs the damage while the other four sit close to where they were. Direction follows from the rule naming a destination. A rule sending money-back requests to billing sends every case it fires on to the same place, whereas a model with no rule and a genuinely close call would spread its mistakes over whichever labels happened to score well on each particular input. And the confidence follows from the size of the gap between the top two scores. Sampling draws a candidate in proportion to its share of the probability, so a label that comes back five times out of five at a temperature of 0.7 is one holding almost all of the distribution, and holding almost all of the distribution is what obeying a rule looks like from outside. A close call at that same temperature is what variance looks like, and it flips.

The slice figures repay being read as counts as well as percentages, because the counts are small enough to check by hand. Twenty-four refund cases with eleven wrong leaves thirteen right, and thirteen divided by twenty-four is a little over fifty-four in a hundred, which is the 54.2 in the table. The percentage and the error count are the same fact written twice. On a slice this size the count is the honest unit, since one case is worth a shade over four points on its own, and a metric where a single ticket moves the number by four points is not one to read to a decimal place.

The aggregate did move, by 2.3 points, and that figure is worth deriving rather than noting, because deriving it explains why the nightly eval raised no alarm. An accuracy figure over a set is each group's accuracy weighted by that group's share of the set, which means a group's power to move the mean is its size and nothing else. The refund slice is 24 cases out of 400, which is six cases in every hundred. Its accuracy fell from about 92% to the 54.2 in the table, a drop of roughly thirty-eight points. Six hundredths of thirty-eight is a shade over two, and a shade over two is what the aggregate reported. Nothing was hidden and nothing was smoothed away; the mean did exactly what a mean does with a large movement in a small group.

2.3 points is also exactly the size of change a team explains away: it is only four times the run-to-run spread, and the number carries no information about where the errors went. A movement four times the noise floor is large enough to be real and small enough that an innocent cause will fit it comfortably, and on any given week there is usually one to hand.

Meanwhile the damage in production is far larger than the eval suggests, because the eval set has 24 refund cases out of 400 and production traffic is about 18% refund requests. Running that through costs one multiplication. On 5 million calls a month, nearly one call in five is a refund request, which is about nine hundred thousand of them. At the old accuracy of 92%, eight in every hundred went to the wrong queue, and eight hundredths of nine hundred thousand is the previous 72,000. At 54%, forty-six in every hundred go wrong, and forty-six hundredths of nine hundred thousand is a bit over four hundred thousand. The gap between those two counts is roughly 342,000 additional misroutes a month, which is what the refunds team was complaining about weeks before anyone thought to look at the example block.

A couple of points on a dashboard and several hundred thousand misrouted tickets are the same failure measured against two different populations. The eval set gives refunds one case in seventeen; production sends nearly one in five. An eval set that under-samples a slice will always under-report a bias concentrated in it, and no amount of staring at the aggregate recovers the difference, because the thing that would recover it — how much of real traffic each slice carries — is not written down in the eval set at all.

Finding the example that did it#

Once you suspect the block rather than the model, the most direct test is a leave-one-out ablation. Run the eval once per example with that example removed, and compare each run against the full block on the slice you care about:

def leave_one_out(examples, cases, slice_pred):
    base = score(render(examples), cases, slice_pred)
    print(f"full block: {base:.1f}")
    for i, ex in enumerate(examples):
        reduced = examples[:i] + examples[i + 1:]
        s = score(render(reduced), cases, slice_pred)
        flag = "  <-- suspect" if s - base > 8.0 else ""
        print(f"  without #{i:>2} ({ex.gold:16}) {s:5.1f}  {s - base:+5.1f}{flag}")

full block: 54.2
  without # 0 (delivery        )  54.2   +0.0
  without # 1 (billing_question)  54.2   +0.0
  without # 2 (product_quality )  58.3   +4.1
  ...
  without # 7 (billing_question)  91.7  +37.5  <-- suspect
  ...
  without #11 (refund_request  )  50.0   -4.2

Removing example seven takes the refund slice from 54.2 to 91.7, and nothing else moves by more than a few points. On a 400-case set with twelve examples that is thirteen eval runs, 5,200 model calls, and at the token counts a six-example block carries it costs a few dollars and twenty minutes of wall clock. Running it on the whole eval set rather than one slice is the same cost and is the version worth wiring into continuous integration — the automated checks that run against every proposed change before it merges — on any change that touches the block, because it finds the example responsible without anyone having to suspect a particular one first.

Both ends of the jump from 54.2 to 91.7 are small whole numbers of tickets, and checking them is what makes the result trustworthy rather than merely large. Fifty-four point two on twenty-four cases is thirteen right and eleven wrong; ninety-one point seven on the same twenty-four is twenty-two right and two wrong. Nine cases changed hands, and nine out of twenty-four is the +37.5 the ablation prints. The run count is thirteen rather than twelve for the reason any measurement needs a control: twelve runs each missing one example, plus one run of the full block for them to be compared against. Thirteen runs over four hundred cases is thirteen lots of four hundred completions, which is the 5,200. The reason the diagnosis is worth automating is not that it is clever — it is that thirteen numbers with one obvious outlier in them are readable by anybody, while twelve examples with one wrong label in them are not.

There is a cheaper screen that catches most annotation errors before they ship, which is to have the model classify each example using the other eleven as context and flag disagreements. Twelve calls, a few cents, and it will surface exactly the cases where your label contradicts the pattern the rest of the block establishes. It produces false positives on the deliberately hard cases that coverage-driven selection puts in a block — a multi-issue ticket resolved by house convention is supposed to disagree with the naive reading — so it is a queue for human review rather than a gate, and the review is the point. An example where the model disagrees with your label and you cannot articulate why your label is right is an example that should not be in the block.

The same mechanism outside classification#

A wrong label is easy to talk about because the answer is one token from a closed set. The identical failure on a generative task is harder to see and does more damage, because there the demonstration's answer is a paragraph and every property of that paragraph is being specified at once, including the ones nobody chose deliberately.

A support-reply assistant makes this concrete. One of its six examples pairs a customer complaint about a delayed order with a reply that reads, in part, I've gone ahead and issued a full refund to your original payment method. The reply was a real one, written by a real agent who really did issue the refund from a system the assistant has no access to. The assistant cannot issue refunds, and nothing in the prompt says it can, and the demonstration is nonetheless a specification of what a correct answer to this task looks like — so the model writes replies asserting refunds it has not issued. On a 200-case eval scored for claims of completed actions, that single example took the rate of replies asserting an action the system did not take from 4% to 19%, which is roughly one in five customers being told their money is on the way by a service with no ability to send it. Nineteen replies in every hundred is where that "one in five" comes from, and the whole of the difference between it and the four in a hundred before is one demonstration.

What that example specifies divides into the part somebody chose and the part nobody did, and the second part is the one that shipped. Whoever pasted it in was choosing a tone, a length, and an ordering — the apology first, the action second. What they were also specifying, without deciding to, is that a reply of this kind states a completed action in the first person and does not hedge about it. A document establishes properties and a sentence states facts, and the model is continuing a document. So what carries forward is the property — replies here assert an action taken — while the particular refund in the example is not a fact being stored anywhere, it is one instance of the property being copied. Nothing in the prompt has to claim the assistant can issue refunds for the assistant to begin writing as though it had issued one.

The same mechanism produces unsupported detail in summarisation. If a summarisation example's output contains a figure that does not appear in its source document — because the human who wrote it knew the number from elsewhere — the demonstration teaches that summaries in this task may contain confident specifics beyond the source, and the model obliges on every subsequent document. Grounding instructions do not survive this, for the reason topic 4.1, on how a prompt is assembled, established about demonstrations overriding written rules.

The screen for it is the same shape as the label check and asks a different question of each example: is every claim in this answer derivable from the input beside it, and is every action it describes one this system can actually perform? Both questions are answerable by reading, and both are unanswerable by any automatic metric, which is why they belong on a review checklist rather than in CI.

Random labels and one wrong label are opposite interventions#

There is a well-known result that appears to contradict all of this, and it is worth handling directly because someone will raise it. Min and colleagues, in 2022, replaced the gold labels in few-shot demonstrations with random labels drawn from the correct label set and found that performance dropped only slightly across a range of classification tasks. The paper's conclusion was that much of what demonstrations provide is the label space, the input distribution and the format — roughly the anchoring jobs a first demonstration does — rather than the input-to-label mapping itself. It gets cited, second-hand, as "the labels in your examples don't matter much".

Randomising every label and mislabelling one are opposite interventions. Random labels destroy the mapping uniformly, leaving noise with no structure for the model to generalise from, so the model falls back on what it already knows about the task and loses only the small increment the mapping was contributing. One wrong label inside eleven right ones is a coherent signal: it is consistent with a rule, it is not contradicted by anything else in the block, and the model has every reason to take it seriously. Later work has also qualified the original result — Yoo and colleagues found that label correctness matters more as models get larger and on tasks where the mapping is not something the model already knows — and the practical version for a team shipping a prompt is that your organisation's taxonomy conventions are exactly the part the model cannot infer, which is the part your labels are carrying.

The word doing the work in that contrast is consistency, not quantity, and the way to see it is to ask the earlier question again: what mapping would make every pair in the block come out the way it did? Where the labels were drawn at random there is no such mapping, because they were assigned by a coin rather than by a rule, so no rule fits them and the highest-scoring continuation is the one that ignores the mapping and keeps whatever the block still does establish — the format, the label vocabulary, and the kind of input this task takes. Those are precisely the things the paper reported surviving. Where eleven labels are right and one is wrong, a mapping does fit all twelve, and adopting it costs the model nothing, since it is consistent with every piece of evidence in front of it. A block with one error in it is therefore more dangerous than a block with every label scrambled, which is a sentence worth reading twice before anyone cites that paper as grounds for relaxing about labels.

Where the diagnosis stops working#

Leave-one-out finds the culprit when the eval set disagrees with the example. It finds nothing when the eval set agrees, which is the case whenever the person who labelled the examples also labelled the eval cases and applied the same misunderstanding to both. Then removing example seven changes nothing measurable, the slice score stays at 91 in both configurations, and production keeps producing outcomes the refunds team knows are wrong while every dashboard reads green. The failure has moved from the prompt to the ground truth, and no prompting technique detects it, because the system is faithfully reproducing a definition your organisation wrote down incorrectly.

The available checks are unglamorous. Have a second person independently label thirty cases from a slice and measure agreement, which puts a number on whether the taxonomy is well defined at all. Compare eval labels against a downstream signal that comes from outside the labelling process, such as how often a routed ticket gets manually forwarded within an hour, since that number is generated by people who have no idea what the eval set says. Where the two disagree, the eval set is the thing to fix first.

There is also a limit to what a single-example diagnosis can explain. Two examples that conflict with each other produce a milder version of this pattern — the model becomes indecisive on the contested region rather than confidently wrong — and leave-one-out will show two examples each worth a few points with no obvious culprit, which is the situation the declining tail of the count curve describes from the outside. The remedy there is the same curation work, done with the conflict now identified.

The thing to carry out of this is a habit rather than a technique. One annotation error in a twelve-example block is not a rounding error, it is a rule you did not know you had written, and it will be obeyed on the slice that resembles it until somebody removes it. Read the eval per slice so that the shape is visible, keep a leave-one-out run cheap enough to be routine, and hold every label to the standard you would hold a line of code that decides where a customer's money goes.

All of this assumes a human chose the twelve examples and a human can be held responsible for each label. The next step teams reach for is to stop choosing, and to retrieve the examples per request from a pool of thousands. That moves the selection decision into an embedding model, and it changes which examples appear next to the hardest inputs in a way that is systematic and, at first, invisible.