The zero-shot baseline

A zero-shot prompt states a task and asks for an answer without showing a single completed instance: the instructions, the six queue names a support classifier routes into, the output format, and nothing else. A few-shot prompt is the same thing with several complete worked cases inserted — a ticket, then the queue it belongs in, then the next ticket, then its queue — so the model sees the task performed correctly a handful of times before it is asked to perform it on the real input. Each of those worked cases is a demonstration. Both names describe one underlying mechanism, in-context learning (ICL): a model that has already finished training can pick up a pattern from whatever sits in its prompt and apply it to the next input, with no change to its weights, no training run, and nothing to deploy beyond a text file.

That last sentence is the claim rather than the explanation, and the explanation is worth working through, because almost everything the rest of this topic measures is a consequence of it. A language model is a scoring function over the next fragment of text: everything written so far goes in, a score for every candidate fragment comes out, and a separate step of ordinary code then picks one, appends it, and runs the function again — the arrangement topic 1.0 sets out, where a model is that function and the loop around it. There is no slot anywhere in that arrangement for a task, a rule or an example. There is one string, and one operation performed on it.

What makes a string of worked cases produce labels is a property of the documents the scoring function was shaped by. Documents are internally regular. A numbered list continues as a numbered list, the fourth row of a table looks like the first three, and a glossary that has defined four terms in the form term — one sentence defines the fifth the same way. A function trained to continue text plausibly is therefore trained, among other things, to pick up a regularity established a few hundred tokens above and extend it, because extending it is what actually came next in an enormous number of the documents it was built from.

A few-shot prompt is a document with exactly that shape, and deliberately so. Ticket, then queue. Ticket, then queue. Then a ticket with nothing after it. The only continuation consistent with everything above is a queue name for the last ticket, so emitting the right label and continuing the document are the same act, performed by the same machinery that would finish the glossary. Nothing was installed and nothing was taught in any sense that persists; the regularity was read off the string, and it holds exactly as long as the string is in front of the model.

Two consequences follow, and both are load-bearing for everything after this chapter. Nothing gets written down, because no step in the loop writes back to the weights — the model is identical before and after the call, and the next request knows nothing about this one. And because the whole effect lives in the scores conditioned on a longer string, that string has to be present every time, which is why demonstrations are re-sent and re-billed on every call rather than paid for once.

The alternative is to put the pattern somewhere it does not have to be re-sent, which is what fine-tuning does — a further round of training that adjusts the model's own weights on a corpus of your examples, after which the examples themselves stay behind. Weights are read on every request and written on none, so a pattern living there costs nothing per call and cannot be changed without another training run. A pattern living in the prompt costs tokens on every call and can be changed by editing a file. Neither arrangement dominates the other, which is why both are still in use, and the question of which is cheaper turns out to be a question about call volume that a later chapter in this topic puts a number on.

Reading a convention off the string rather than training it in is why the technique exists at all. Teaching a system your particular six-queue taxonomy used to mean assembling a few thousand labelled tickets and fine-tuning a model on them, which is days of work plus a model to version, serve and monitor. Eight worked cases in a string do a weaker version of the same job in the time it takes to save the file, which is why nearly every production prompt you inherit has an Examples: heading somewhere below the instructions, why the vendor cookbooks lead with one, and why the prompting libraries ship a template class whose only job is rendering the block. The "few" in few-shot is literal — typically one to about sixteen demonstrations. What almost none of those prompts carries is the one number that says what the block is worth, which is the score of the same task with no demonstrations in it at all.

The measurement that has to come first#

The apparatus for this was established in topic 4.1, on the anatomy of a prompt — fixed cases, fixed decoding, one change per measurement, and a known run-to-run spread before you believe any gap. What an ablation adds is which two things you put on either side of the comparison. An ablation is an eval run with a single component removed, so that the difference names what that component was contributing; here the component is the example block. The zero-shot variant is not a different prompt. It is the same prompt with the block removed and nothing else touched: same instructions, same constraints, same output format, same model version, same parser, same scorer. If you rewrite the task line while you are in there because it reads better without the examples underneath it, you have measured two changes at once and the result attributes to neither.

A sweep worth trusting looks roughly like this, with the variants defined as a dictionary so the harness cannot accidentally drift between them:

import statistics

DECODING = {"temperature": 0.0, "top_p": 1.0, "max_tokens": 8, "seed": 7}

BASE = open("prompts/triage_v4.txt").read()          # role, task, labels, format
EXAMPLES = open("prompts/triage_v4_examples.txt").read()   # the 8 demonstrations

VARIANTS = {
    "zero_shot": BASE,
    "few_shot_8": BASE + "\n\n" + EXAMPLES,
}

def score(prompt: str, cases) -> float:
    hits = 0
    for case in cases:
        reply = client.complete(prompt=prompt + "\n\n" + case.ticket, **DECODING)
        hits += parse_label(reply) == case.gold
    return 100.0 * hits / len(cases)

for name, prompt in VARIANTS.items():
    runs = [score(prompt, HELD_OUT) for _ in range(5)]
    print(f"{name:12} mean={statistics.mean(runs):5.1f} "
          f"spread={max(runs) - min(runs):.1f} "
          f"tokens={count_tokens(prompt)}")

zero_shot    mean= 88.6 spread=0.6 tokens=612
few_shot_8   mean= 91.0 spread=0.5 tokens=4318

Two lines instead of one, and the change now says something. The eight examples are worth 2.4 points on this set, the run-to-run spread is around half a point so 2.4 is a real difference rather than noise, and the price is 3,706 additional input tokens per call. Whether that trade is good is a separate argument, and it is an argument a team can now actually have, because both sides of it are numbers. Without the first line of that output the discussion collapses into taste, and taste reliably favours whoever wrote the examples.

Both figures in that reading are subtractions, and they are worth doing on the page. The 91.0 on the second line minus the 88.6 on the first is 2.4, which is the accuracy column. The 4,318 tokens minus the 612 is 3,706, which is the token column. What licenses either subtraction is not the arithmetic, which anyone can do, but the way the two variants were constructed: everything except the example block is byte-identical across the two runs — same instructions, same constraints, same output format, same model version, same decoding, same parser, same scorer — so the block is the only thing in the system that could have moved either number. That is the whole logic of an ablation, and it is why one change per measurement is a requirement rather than tidiness. A variant that also rewrote the task line would produce two subtractions that look exactly like these and license neither of them.

The mistake this measurement exists to prevent is a specific one, and it is tempting for a good reason. A number reported next to a change reads as a statement about the change, and 91.0 arrives at the bottom of a diff that added eight examples. What the figure is actually a statement about is the whole prompt — instructions, labels, format and demonstrations together, run against one set of cases on one model version. The examples are one term in that total and the total does not decompose by being looked at. Recovering the term rather than the sum requires running the configuration with the term missing, which is the only operation that isolates it.

Five repeats at temperature 0 may look like superstition, and the reason for them is that temperature 0 fixes the sampling policy without fixing the arithmetic. The policy at that setting is to take the highest-scoring candidate, which is fully determined once the scores exist. What is not fully determined is the scores. Each one is the end of a very long chain of floating-point additions, and floating-point addition is not associative — adding a large value to a small one can round the small one away entirely, so the same set of numbers summed in a different order can land on a slightly different total. Batched inference on GPUs sums floating-point values in an order that depends on what else is in the batch, which is to say on whose requests happened to arrive alongside yours.

Most of the time that changes nothing anyone could observe, because the gap between the top candidate and the second is far wider than the last few bits of the sum. On a token where the top two candidates are within a hair of each other it decides which of them is on top, so two identical requests can diverge, and one divergent token in a classification reply is one flipped label — the reply is the label. On a 400-case set that shows up as a few tenths of a point of movement between runs of a byte-identical prompt. The intuition to carry away is that temperature 0 removes the randomness you asked for and leaves the randomness the hardware brings. The spread is the thing you compare gaps against, and a team that has never measured it will happily ship a 0.4-point "improvement" that is indistinguishable from doing nothing.

What a demonstration is actually good at#

Deleting the examples to get a baseline is not an argument that examples are waste. It is a way of finding out which of four specific jobs they are doing, because those four are what a demonstration does better than a sentence, and everything else in a prompt is usually cheaper as instruction.

The first is anchoring the shape of the output. A worked example fixes field order, key spelling, quoting, whether the reply is bare or wrapped in a code fence, and whether anything precedes it — all at once, in a form the model copies rather than interprets. Describing the same shape in prose takes several sentences and still leaves the model room to add a preamble.

The second is establishing a boundary convention on inputs that are genuinely ambiguous. Consider a ticket that says the customer was double-charged and the replacement unit arrived a week late. Is that billing or delivery? There is no fact of the matter; there is a house rule, and your house rule is that the queue with the money in it wins. Stating that rule in general terms means enumerating the priority order across every pair of your six labels, which is fifteen sentences that all read like legal drafting. Showing it means one example with a two-issue ticket and the label billing underneath. The example is shorter and it is also less likely to be misread, because a demonstration cannot contain an ambiguity about which of two rules applies.

The third is register. If the task produces text a human reads, the difference between a reply that opens "Thanks for reaching out!" and one that opens with the answer is a matter of a hundred small choices, and a single example of the voice you want carries more of them than a paragraph describing the voice.

The fourth is tie-breaking on inputs that sit near the model's own decision boundary, where it is close to indifferent and its answer moves with phrasing. A demonstration on that kind of input nudges it consistently, which is worth points on precisely the slice that generates escalations.

Those four jobs have a property in common that is worth noticing, because it predicts which prompts benefit from examples before you run anything. Each of them is a convention rather than a fact — something your organisation decided, which the model has no way to infer from the task description because it could reasonably have been decided the other way. The exact JSON key spelling, the priority order between labels, the house voice, and the direction to lean on close calls are all arbitrary in the sense that a different team would have chosen differently and been equally correct. Where a prompt is asking the model for something with a right answer in the world, examples add little, because the model either knows it or does not. Where the prompt is asking for compliance with something local and unwritten, examples are the shortest way to write it down.

Why a demonstration is the shortest way follows from the mechanism at the top of this chapter. The model is extending a local regularity, so whatever the demonstrations hold constant becomes part of the regularity being extended: the field order, the absence of a preamble, the way a two-issue ticket was resolved, the opening words of a reply, the direction a close call leaned. None of those has to be described, because none of them is being described. They are simply present in the text the model is continuing, and continuing text is the operation it performs.

An instruction reaches the same place by a longer route. It is a sentence about the output rather than an instance of one, so something has to convert a description into a thing matching the description. That conversion is what instruction tuning added — a training stage in which a model is taught to follow written directions rather than merely continue the text it is given — and a conversion is a step with somewhere to go wrong, which a copied pattern does not have. Where a property is easy to state and tedious to demonstrate, the sentence wins; where it is easy to demonstrate and tedious to state, the example does.

Set against what an instruction does, the division is fairly clean:

Two rows in that table answer "No" in the example column, and they answer it for the same reason, which is worth separating from the practical advice around it because it is not a limitation anyone can engineer away. A demonstration is one member of the set of legal outputs. An example ending in billing asserts that billing is a legal answer, and an assertion of that shape — this one is in the set — rules nothing out. What "the label set is exhaustive" requires is the opposite shape: nothing outside these six is in the set, which is a claim about every string you never wrote down. No quantity of claims of the first kind adds up to one of the second, since each new example can only ever put one more thing inside the set. Twelve demonstrations covering all six queues leave a seventh queue name exactly as available as one demonstration does, and only a sentence saying the six are the complete set closes it off. The competitor row is the same argument in negative form: a block of replies that all happen to avoid naming a competitor never says that naming one is forbidden, while a single reply that names one says, unmistakably, that it is allowed.

The rows where a sentence wins are the ones people most often try to buy with more examples, and topic 4.1, on how a prompt is assembled, already showed the sharp edge of that: a demonstration is read as the operative specification, so an example that quietly violates a stated rule repeals it. The corollary here is that every example you add is another chance to demonstrate something you did not intend, which is a cost that does not appear in the token count.

There is also a category of thing examples do not do at all, and it gets confused with the ones above often enough to be worth naming. Demonstrations do not install knowledge. An example whose answer asserts that refunds are available for ninety days does not teach the model your refund policy; it teaches the model that answers in this task assert a refund window with confidence. When the policy changes to thirty days, the demonstration keeps producing confident ninety-day-shaped answers, and it will do that until somebody reads the example block again. Few-shot prompting is a way to specify the form and the boundaries of an answer, and it is a poor and expensive substitute for retrieval, for a system prompt that states the current policy, or for fine-tuning.

When the labels already say what they mean#

The same question resolves in the opposite direction on a task that never needed demonstrations. A teammate is classifying inbound complaints into three categories — billing, delivery, product quality — and has gone straight to a twelve-example few-shot prompt. Nothing in that is unusual, which is exactly why it deserves a question: it was chosen because it is what everyone does, and no measurement was involved.

Three classes with self-describing English names is close to the easiest thing a current model does. The label delivery needs no explanation, and the model has been trained on a very large number of support taxonomies that look exactly like this one. A plausible zero-shot score on a task of that shape is in the low nineties, and the twelve examples are unlikely to be finding twenty points anywhere. Meanwhile the block runs to 600 tokens, and the service takes 5 million calls a month. Multiplication is the right operation on those two figures precisely because the block is fixed: it does not vary with the complaint, so every one of the 5 million calls carries the same 600 tokens, and the monthly total is that figure repeated once per call. Six hundred times five million is 3 billion input tokens a month, which is a real line on a real invoice.

That does not make the twelve examples wrong, and there are findings that would justify keeping some of them. If the zero-shot run's errors concentrate on complaints that mention two problems, examples are the cheap fix, because a boundary convention is what they are good at. If your product quality category includes packaging damage by internal convention, no amount of label-name reading will get the model there and it will route those to delivery forever, so a demonstration is doing genuine work. If the downstream consumer needs a bare lowercase token and the zero-shot replies keep arriving as Category: Billing, one example fixes the shape for a hundred tokens.

Equally, there are findings that would take the block to zero. A zero-shot score of 94 against a few-shot score of 94.2, with a run-to-run spread of half a point, says the twelve examples bought nothing measurable and cost 600 tokens on every call. That is a strictly worse system, and shipping it is a decision nobody made on purpose; it is what happens when the default is followed and nothing is measured.

The ablation also has a shelf life, which is why it belongs in the eval suite rather than in a reviewer's memory. A zero-shot score of 88.6 was measured against one model version, and the gap it establishes is a property of that version rather than of the task. Model upgrades have a habit of closing exactly this gap, because each generation follows instructions more reliably without being shown, and an example block that was worth 2.4 points in March can be worth 0.3 in September while nobody re-runs the comparison.

What all of this buys is the ability to say what a block of demonstrations is for. Without the zero-shot row you have one number that is compatible with every story about the examples, including the story where they are pure cost; with it you have a gap, a spread to judge the gap against, and a token price to set the gap against. Ten minutes of eval time is the whole of the investment, and one extra run a night is the whole of the upkeep.

Once the baseline exists and the examples have a reason, the remaining question is how many of them, and that turns out not to be a question with a monotone answer. Accuracy against example count has a shape, it flattens earlier than most people expect, and past the flat part it bends back down.