Nearest-neighbour example selection

Dynamic example selection replaces a fixed block with a query. Embed the incoming input — run its text through an embedding model, which returns a vector of a few hundred numbers positioned so that texts with similar meaning land close together — search a pool of labelled cases, take the nearest few by cosine similarity, which scores a pair of vectors by the angle between them and so ranks the pool by how alike it is to the input, render them as the example block, and classify. It is about forty lines of code, and it is the obvious next move once you have accepted that which examples matter more than how many.

The reasoning behind that move is sound as far as it goes, and it is worth putting in its strongest form before taking it apart. A demonstration helps a test case to the extent that the test case resembles it, which is what the earlier sweep of accuracy against example count established when it showed that which two examples you draw moves the score further than the choice between two of them and sixteen. If resemblance is what makes a demonstration useful, then choosing demonstrations by resemblance is the direct way to make every one of them useful on every input, rather than hoping four hand-picked cases cover the traffic. That argument holds on the great majority of inputs, and on the great majority of inputs a selector does beat a curated block.

It is also, mechanically, a k-nearest-neighbour vote with a language model attached, and it inherits that algorithm's failure mode along with its strengths. That inheritance is invisible in an aggregate score and obvious in a per-slice one, which is why a selector is the change most likely to ship on a green aggregate and make the inputs you care about worse.

Why the nearest neighbours carry the wrong label#

A k-nearest-neighbour classifier assigns a point the majority label of its k closest neighbours, and its characteristic weakness is well understood: it is accurate in the interior of a class and unreliable near the boundary, because near the boundary a point's neighbours include members of the adjacent class and the majority vote comes down to which side happens to be denser locally. That is exactly the algorithm a selector builds, with one change — the vote is not counted, it is shown to a language model as a set of demonstrations. That sentence is a statement of what the selector is rather than a comparison it happens to resemble, and the weakness it inherits is worth building up from the parts rather than cited.

Start with the embedding. An embedding model places a piece of text at a point in a space of a few hundred dimensions, and it was trained so that texts a person would call similar land near one another. What it means by similar is whatever the training made it mean, which is a property of language — shared vocabulary, shared subject matter, shared phrasing, the things that make two sentences read alike to somebody who knows neither ticket's history.

Your label boundary is not a property of language. Which queue a two-issue ticket goes to is a house convention, the queue holding the money wins, decided inside your organisation by people who could reasonably have decided it the other way. Nothing about that decision is present in the words of a ticket, and the embedding model was never shown it. So the surface along which your label changes is a surface the embedding model knows nothing about, and there is no reason for it to line up with the directions in which distance grows.

Two things follow, and the second is the one that costs you. In the interior of a class the mismatch does not matter: a ticket surrounded on every side by tickets of its own kind has neighbours that agree with it, whatever surface runs somewhere further away. A ticket sitting on the boundary has neighbours on both sides of it, all of them genuinely close, because closeness is measured in a space that does not bend around the boundary. The ball you draw around an input to hold its eight nearest cases can therefore cross the boundary while every member of it is a similar ticket.

Which label dominates that ball is then settled by which class happens to be denser where the ball sits, and local density in a pool of past tickets is a record of how often each kind of ticket arrived. It is a base rate. A base rate carries no information about which label your convention assigns to the input in the middle of the ball, and on the inputs where the two disagree it is the base rate that wins the vote.

Take a case from a slice where refund requests and billing questions are hard to tell apart: Charged twice for the annual plan on the 3rd, I want my money back. Surface similarity puts it near every ticket in the pool about duplicate charges on annual plans, and those are overwhelmingly labelled billing_question, because most double-charge tickets are questions about a charge rather than demands for a refund. Seven of the eight retrieved neighbours therefore carry billing_question. The model now sees seven demonstrations mapping this exact vocabulary to billing_question, plus the majority-label effect operating at full strength on a block that is 88% one class, and it answers billing_question. Under a fixed block, the model saw a balanced set of eight and had to reason from the taxonomy, which it did correctly on 39 of these 48 cases.

Two things happened there and both are worth naming. The retrieval did not merely fail to help — it assembled a block that is seven parts in eight one label, which is the majority-label pressure the earlier chapter on example count measured, arriving here by construction rather than by accident. A hand-assembled block that lopsided would have been caught in review; this one is rebuilt on every request by code nobody reads. The second thing is what the model was left without, which is any evidence at all pointing the other way.

The failure is invisible in an aggregate by construction, which is worth being plain about rather than treating as bad luck. Boundary cases are a minority of any set — being a minority is close to what makes them a boundary — so a selector that wins comfortably in the interior and loses badly at the edge wins on the mean. The aggregate is not failing to report the regression through any defect of measurement. It is reporting the weighted average it was asked for, and the weight sitting on the slice that broke is small.

The measurement that makes the failure visible in a running system is label purity of the retrieved block — the share of the retrieved examples carrying whichever label is most common among them, so 1.00 means the block is unanimous — logged per request alongside the outcome:

def select(ticket, pool, k=8):
    q = embed(ticket)
    hits = pool.search(q, k=k)                     # cosine, descending
    purity = max(Counter(h.label for h in hits).values()) / k
    return hits, purity, hits[0].score

# aggregated over the 400-case eval

purity   share of requests   accuracy   model agrees with majority label
1.00                  46%       93.1                            98.4%
0.875                 24%       90.6                            96.2%
0.75                  18%       88.9                            91.7%
<=0.625               12%       79.2                            77.1%

boundary slice only, purity 1.00 (n=21):  accuracy 47.6

Before the accuracy column, notice why the purity column holds the values it does. Purity is the size of the largest label group divided by the number of examples retrieved, and eight are retrieved here, so it can only ever be a count of eighths. Eight of eight is 1.00, seven of eight is 0.875, six of eight is 0.75, and the bottom row gathers five of eight together with everything below it. The seven-of-eight row is the same block shape as the refund case above, since seven parts in eight written as a percentage is the 88% that paragraph quotes.

Two readings of that table matter. The model agrees with the retrieved majority label 98.4% of the time when the block is unanimous, which means that on nearly half of all requests the classifier is a k-nearest-neighbour vote wearing a language model as a formality. And on the boundary slice, unanimity is a strong predictor of being wrong rather than right, because those are precisely the cases where local density lies. You have taken on kNN's failure mode without taking on its one redeeming property, which is that a kNN classifier at least reports its vote margin and can abstain.

That second reading inverts what a purity number looks like it ought to mean, so it is worth taking one step at a time. A unanimous block means the ball did not cross the boundary in the vote, and exactly two situations produce that. Either the input sits nowhere near the boundary, in which case unanimity is honest and the case is easy. Or the input sits on the boundary and one side is much denser there, in which case the vote is unanimous for the denser side while the input belongs to the other. A slice selected for sitting on the boundary is mostly the second situation, which is why unanimous blocks inside it score 47.6 while unanimous blocks across the whole set score 93.1. Confidence in the block and correctness of the block have come apart, and the purity figure on its own cannot tell you which situation produced it.

The three other things a selector gives up#

Boundary inversion is the failure people should look for first, and it is not the only difference between a fixed block and a dynamic one.

The prompt stops being an artefact. With a fixed block, the exact bytes sent to the model live in the repository, can be diffed, reviewed, and reproduced a year later from a commit hash. With a selector, the prompt is constructed per request from a pool and an index, so reproducing an incident requires the pool contents, the index build, the embedding model version and the query, none of which are in the repo. Logging the selected example identifiers and a pool version with every request is the minimum that makes an incident investigable, and it is cheap; teams routinely discover they need it during the incident.

The cache goes away. A fixed block is byte-identical on every call and therefore cacheable, at a hit rate of 78% and an effective input rate around 0.35 of base. A per-request block shares no prefix with the previous request, so the hit rate on that segment collapses to roughly zero and the block returns to full price and full prefill. On the extraction service's numbers that is a move from $2,540 back to $7,200 a month, plus about 450 milliseconds of prefill returning to every request, plus 25 milliseconds for the embedding call and 8 for the vector search. A dynamic selector is not a cost-neutral swap for a fixed one, and it is worth pricing that at the same time as the accuracy, because a three-point gain and a $4,700-a-month increase is a trade someone should make deliberately.

Why the hit rate collapses rather than merely falling deserves a step, because a selector that changes only one example in eight looks as though it ought to keep seven eighths of the discount. A cache holds computed state for a prefix of the prompt, and that state is reusable only when the bytes it was built from are identical, since every part of it was computed from every byte before it. Change the first retrieved example and the state after that point differs everywhere, so nothing from the second example onward is reusable either. A prefix is cacheable or it is not, and a block assembled per request is not.

The money follows from that without any further reasoning. The block goes back from its cached price to its full one, and taking two thousand five hundred and forty dollars from seven thousand two hundred leaves a rise of just over four and a half thousand dollars a month, which is the increase quoted above. It arrives at the same time as a three-point accuracy gain, and pricing the two together is what makes it one decision instead of two taken by different people in different weeks.

The pool becomes an unaudited liability. One wrong label among twelve does concentrated damage when a human chose all twelve; a pool of 3,000 labelled cases will contain wrong labels — a 2% annotation error rate is respectable and means sixty of them — and every one is eligible for retrieval. Worse, a wrong label in the pool is retrieved precisely for the inputs that resemble it, so it is not diluted across the traffic; it concentrates on its own neighbourhood exactly as a hand-placed mislabel does. The leave-one-out diagnostic that names the culprit in a twelve-example block does not scale to 3,000, so the equivalent has to be built differently: log the retrieved identifiers, join them against outcomes, and rank pool entries by the error rate of the requests that retrieved them.

Two of the claims in that paragraph are arithmetic rather than opinion. Two wrong labels in every hundred, across three thousand cases, is sixty wrong demonstrations, every one of them live and every one retrievable. And leave-one-out fails for a reason you can count, since it re-scores the whole eval once per example — a dozen or so runs for a twelve-example block, and three thousand of them here. What sits between those two figures is a mechanism rather than a coincidence: resembling a demonstration is the condition under which that demonstration weighs most heavily on an answer, so the property that makes retrieval work is the same property that aims a pool mislabel.

And there is a new dependency in the request path. The embedding service and the vector index are now on the critical path of every classification, with their own availability, their own version skew when the embedding model is upgraded and the pool is not re-embedded, and their own failure modes. That argues for the same thing the accuracy problem argues for, which is a fallback.

A selector with a fallback, and what triggers it#

The fixed block does not go away when you ship a selector. It becomes the default that the selector has to earn its way past on each request:

Three triggers, each tied to a failure named above. A low top similarity means the pool has nothing genuinely like this input, so the retrieved examples are eight arbitrary tickets and the curated block is strictly better; 0.62 is not a universal number and is found by plotting accuracy against top-1 similarity on your own eval and reading off where the retrieved block stops beating the fixed one. High purity is the boundary guard, and it is deliberately conservative — a unanimous block is either an easy case, where the fixed block also gets it right, or a boundary case, where the unanimous block is actively harmful, so refusing to use unanimous blocks costs little and removes the worst behaviour. The timeout keeps an embedding outage from becoming a classifier outage.

The rate at which those triggers fire is a number to watch rather than a number to minimise. A fallback rate near zero means the thresholds are doing nothing and you have shipped the unguarded selector with extra code; a fallback rate near one means you have shipped the fixed block with an embedding call attached to it and should stop paying for the index. Between those, the useful reading is which trigger is firing: mostly low similarity says the pool does not cover your traffic and should be grown, while mostly high purity says a large share of your inputs sit in dense single-label neighbourhoods, where the curated block was going to win anyway.

There are two refinements worth knowing before writing your own thresholds. Retrieving per class rather than globally — the two nearest refund_request cases, the two nearest billing_question cases, and so on — produces a balanced block by construction and removes the majority-label pressure entirely, at the cost of forcing in examples that may be far from the input. Maximal marginal relevance, which penalises a candidate for resembling candidates already selected, gives a more diverse block from the same pool and helps for the same reason.

When the selector is the right instrument anyway#

Everything above is a case against shipping a selector carelessly, and there are tasks where a fixed block cannot compete and the work of building the guards is obviously worth it. The clearest is a large label space. A taxonomy of forty categories cannot be covered by any block small enough to afford, and retrieval solves that directly: the eight retrieved examples are drawn from the handful of categories actually plausible for this input, so the block is dense in the region that matters instead of thin everywhere. The gain there is not three points, it is twenty, and no amount of curation substitutes for it.

That is a coverage argument, and it is the argument the earlier chapter on coverage-driven selection made about slices, run at a different scale. With forty categories and room for eight examples, a fixed block is thin everywhere by arithmetic, since one example per class is forty examples and forty examples is not a block anybody is going to pay for. Retrieval does not make the block bigger. It moves the eight slots to wherever the request happens to be, so the same tokens are spent on the small region the answer will come from. The boundary problem is still underneath all of that, and it is simply much smaller than twenty points.

The second is a long tail of input shapes that no curated set of four can represent — a document extractor handling two hundred distinct vendor invoice layouts, where the useful demonstration is one from the same vendor. The third is per-customer or per-tenant convention, where the correct answer genuinely depends on which account the request came from and the pool can be filtered to that tenant before the search. In all three the selection is doing work that is not available any other way, and the boundary problem is a cost to manage rather than a reason to decline.

A hybrid is often the best shape once you have both artefacts. Keep three or four curated examples as a fixed prefix — the ones that anchor output shape, the house convention and the malformed-input behaviour — and append two to four retrieved ones after them. The curated prefix stays cacheable, which recovers most of the cost the selector gave away, it holds the label distribution away from unanimity on its own, and the retrieved tail supplies the input-specific coverage. On the eval above this scored 86.5 on the boundary slice, above both pure approaches, because the fixed examples give the model a reason to disagree with the retrieved majority.

Why that recovers the boundary slice is what the mechanism above predicts. A boundary case fails under pure retrieval because every demonstration in front of the model agrees, and it agrees for a reason — local density — that has nothing to do with the taxonomy. Each retrieved case is correctly labelled for itself, so what the model is missing is contrary evidence rather than accurate evidence. Three or four curated examples chosen to spread across the labels put that contrary evidence back, and they do it without displacing the retrieved tail, which is still the part supplying coverage of this particular input.

Retrieval is worth separating from the other thing it resembles. Selecting demonstrations is not retrieval-augmented generation, or RAG, where documents are fetched and put in the prompt to supply facts the model does not have; the pool here is not a knowledge base and the retrieved items are not evidence the model should draw facts from. If your task fails because the model lacks information, retrieving eight similar past tickets will not supply it, and you want the document rather than the demonstration. Conflating the two produces a system that puts eight old tickets in the prompt and hopes the answer is in there, which is neither good selection nor good retrieval.

The distinction turns on what the retrieved text is for. A document fetched by a retrieval-augmented system is evidence, and the model is meant to read facts out of it and carry them into the answer. A demonstration fetched by a selector is a specification, and the model is meant to read the form of an answer off it and apply that form to a different input. Those are different kinds of thing, and no quantity of the second ever substitutes for the first.

The rule that comes out of this is narrow and worth stating on its own. Similarity is a good proxy for relevance everywhere except the place your errors live, and the place your errors live is the class boundary, where similarity and label part company. So instrument the purity of every block you retrieve, read the boundary slice before the aggregate, and keep the curated block in the codebase as the thing that runs when the pool has nothing useful to say.

Both designs — the curated block and the pool behind the selector — now rest on a body of labelled examples somebody assembled at some point, and neither chapter has said anything about how old that material is or whether it still describes the product.