Prompt portability across models

Moving a working prompt onto a different model is a port. Everything the prompt is made of travels — the instructions, the worked examples, the tool definitions, the output contract — and the destination is another provider's model, another family from the same provider, or a newer version of the one you were already calling. The prompt is meant to arrive doing the job it was doing before, and portability is the word for how much of that job survives the trip. In practice a port arrives as one line in a configuration file — a model identifier, occasionally an API base URL beside it — plus a re-run of whatever evaluation you already have, which is a diff that reviews in under a minute and is most of the reason a port gets under-estimated.

Prompts get moved because few of them stay on the model they were written against for the life of a product. Two models that can both do your task can differ by a factor of two to five in price per call, which is where most migrations start. A provider has an outage and the review afterwards asks for a second path; a procurement contract or a data-residency rule names a vendor you are not currently using; a newer model is released that is better at the thing the product is sold on. Doing nothing runs out as an option too, because dated model versions are retired on the provider's schedule rather than yours.

A one-line change can move behaviour at all because a prompt is not a program the model executes. It is input to a system whose response to input of that shape was settled during post-training: the stages that follow pre-training on bulk text, where a lab tunes the model on demonstrations of the behaviour it wants and on judgements of which of two candidate responses is better. Each lab runs those stages on its own data, with its own conventions for how a prompt is laid out, how much force a delimiter carries, and how far an instruction at the top outranks text that arrives later. Two models can be equally capable and still have been taught different things about what your sentence means.

The chain from a lab's training decisions to a change in your production behaviour is short enough to follow all the way down. A model, at every step, produces a score for every token in its vocabulary and picks among the high scorers. Post-training does not add rules on top of that machine; it moves the scores. Show it many thousands of transcripts in which a request phrased like a policy question is answered with a decline, and the scores on the tokens that begin a decline rise for prompts of that shape. Show it many thousands in which a bare question is answered from memory rather than by calling a tool, and the scores on the tokens that open a tool call fall. Each of those adjustments is a decision somebody made about a distribution you cannot inspect, and none of them surfaces as a parameter you can read off the API.

The wrong belief to get rid of early, then, is that a byte-identical prompt makes the swap a controlled experiment. The pull toward that belief is a reasonable one: everywhere else in software, holding the input fixed and changing one dependency is exactly how you isolate that dependency. What breaks here is that the prompt is not the input and the model is not the dependency. The input is the prompt after the serving stack has wrapped it in that model's own conversation format, and the function it is fed to is a different set of weights carrying a different set of judgements. Two of the three terms moved. The one held fixed is the one that mattered least.

What actually moves when only the model changes#

It is tempting to think of model portability as a question of wording tolerance — whether the second model "understands" the instructions as well as the first. Wording tolerance is real and it is the least of it. The behaviours that shift under a byte-identical prompt fall into four groups that move independently of each other, and a port can be clean on three and broken on the fourth.

The fourth is the one that survives an eval most easily, because most eval harnesses score the answer and throw away everything around it. If your prompt asks for a confidence float and downstream code routes anything below 0.7 to a human, then a model whose confidence distribution is shifted up by 0.15 has changed your human-review volume without changing your accuracy number at all. On a queue that was escalating 18% of tickets, that kind of shift took one team to 6% escalation, which read as an efficiency win for two weeks and then as a spike in bad resolutions.

That escalation move is worth walking slowly, because it is the shape every calibration regression takes. The router never reads the answer. It reads one float and compares it to a fixed line at 0.7. Adding a constant to every score slides the whole distribution across that fixed line, and the cases that cross are precisely the ones that had been sitting in the band immediately below it — a band exactly as wide as the shift. On this queue that narrow band held two thirds of everything being escalated, which is the whole of the drop from 18% to 6%. Nothing about what the model wrote changed in that move. The same cases, carrying the same text and the same error rate, now leave without a human looking at them. The eval scores the text, so the eval does not move either, and the only instrument that registers the change is a volume metric on which it looks like progress.

The independence of the four axes is the part worth internalising, because it decides how the measurement has to be built. They are not four symptoms of one underlying quality; they are four separate consequences of four separate sets of post-training decisions, and there is no ordering among them. A model can hold a schema perfectly and refuse a category you depend on. It can call every tool correctly and hedge on questions the previous model answered flatly. Averaging them into a headline score is what makes a broken port look clean, and the average is exactly what a harness returning one boolean per case computes.

Making the two sides a comparison#

Three things have to be true before the numbers on either side of a swap can be compared at all, and none of them is about which model is better. Before any of them, the arm being measured has to be the arm you are shipping: retries, repair passes and output-cleaning code all sit between the model and the score, and a harness that strips a code fence before parsing is measuring your repair layer rather than the model. Run the comparison with those disabled and record what they would have absorbed separately, because their cost is real and it belongs on its own line rather than folded into an accuracy number.

The evaluation set has to be frozen and it has to be the same set. That sounds obvious and it is routinely violated in a specific way: someone regenerates the set from production traffic between the two runs, because the newer sample is more representative. It is more representative and it is a different experiment. Freeze the set, hash it, and record the hash with the results, so that a run six months later is either comparable or visibly not.

Decoding settings have to be matched, which is harder than it looks because the parameters are not the same parameters. temperature is not a standardised quantity; it scales a logit distribution — the raw scores the model assigns to every token in its vocabulary before they are turned into probabilities — whose shape is a property of the model, so temperature 0.7 on two models from different families are two different amounts of randomness. The ranges differ too, and so do the defaults you inherit when you omit a field.

The mechanism under that is one step long and worth carrying, because it is what makes matching the number hopeless rather than merely awkward. The model hands back a raw score for every token it could emit next. Those scores are divided by the temperature, then exponentiated and rescaled so they sum to one, and it is those rescaled probabilities that get sampled. Dividing by a number below one magnifies the gaps between scores, so the leading token pulls further ahead; dividing by a number above one compresses them, so trailing tokens are handed a share. Temperature is therefore not a quantity of randomness at all. It is a multiplier applied to gaps whose size is the model's own habit — a model that routinely gives its top token a wide lead stays close to deterministic at a setting where a model whose scores bunch together is choosing freely.

The practical stance is to run both sides as close to greedy as each API allows, greedy being the setting where the model always takes its highest-scoring next token instead of sampling among the plausible ones, note that this is an approximation rather than a control, and record the exact request body you sent. Matching the intent — minimum randomness — is achievable. Matching the number is not meaningful.

The word "approximation" is doing work there, because the belief it is trying to remove is that the lowest temperature setting makes a model deterministic. At that setting the model still takes its top-scoring token, and that ranking comes out of summing a very large number of floating-point products on a GPU. Floating-point addition is not associative: adding the same terms in a different order can give a slightly different total. The order depends on how your request was batched alongside other traffic, which is not yours to set. When two candidate tokens are close to tied, that last bit of difference decides between them, and the two runs part company from that token onward. It is uncommon, it is not zero, and it is the reason the exact request body gets recorded rather than the run being assumed reproducible.

Third, each side needs enough repeats to establish its own spread, because a single run gives you a point with no error bar and the difference between two points with no error bars is not information. Five repeats of a 260-case set is 1,300 calls, which at typical prices for a mid-tier model is a few dollars and about twenty minutes wall-clock with modest concurrency. What that buys is the ability to say that model B's schema-violation rate is 2.4% ± 0.6 rather than "2.4%", and therefore whether the 0.0% you measured on model A is a real difference.

The 1,300 is the set size multiplied by the number of passes, and multiplication is the right operation because a repeat is a whole sweep of the same frozen set rather than a re-roll of individual cases: every case is asked five times, so every case contributes five calls. What the extra passes buy is a denominator for the word "different". Each pass samples the same underlying process with nothing else moving, so the variation between passes is the model's own noise, isolated. One pass gives a number and no way to separate a real shift from that noise. Five give a spread narrow enough to quote. When the interval around 2.4% does not reach down as far as zero, the 0.0% on the other model is a genuine difference; when it does reach zero, the two arms are two flips of one coin.

import hashlib, json, statistics
from collections import Counter

FROZEN = json.load(open("evalsets/triage-260.json"))
SET_HASH = hashlib.sha256(json.dumps(FROZEN, sort_keys=True).encode()).hexdigest()[:12]

AXES = ("tool_compliance", "schema_valid", "refused", "answer_correct")

def run_arm(client, model, params, repeats=5):
    """One arm of the comparison. Everything except `client`/`model` is held fixed."""
    per_repeat = []
    for _ in range(repeats):
        tally = Counter()
        for case in FROZEN:
            resp = client.call(model=model, messages=render(case), **params)
            scored = score(case, resp)          # returns a dict over AXES
            tally.update({a: scored[a] for a in AXES})
        per_repeat.append({a: tally[a] / len(FROZEN) for a in AXES})
    return {
        "model": model,
        "set_hash": SET_HASH,
        "params": params,
        "mean": {a: statistics.mean(r[a] for r in per_repeat) for a in AXES},
        "stdev": {a: statistics.stdev(r[a] for r in per_repeat) for a in AXES},
    }

The shape that matters is that score returns four numbers per case rather than one boolean. One is exactly the resolution at which a refusal regression and a schema regression both disappear into an aggregate that stayed at 91%.

Refusals need their own set, built on purpose#

A set sampled from real traffic contains each category in the proportion the traffic contains it, which is what makes it useless for measuring anything that happens at the edge — and refusal thresholds live entirely at the edge. A category that is 1.5% of traffic is four cases in a 260-case set, and four cases resolve nothing.

That count is the category's share multiplied by the set size — one and a half hundredths of two hundred and sixty, landing just under four — and multiplication is the right operation precisely because a sample drawn from traffic reproduces traffic's proportions by construction. The property that makes such a set an honest picture of the common path is the same property that makes it blind at the edge. Four cases can only report in quarters. A refusal rate moving from under one per cent to several per cent changes the expected number of refusals among four cases by a fraction of one case, so the most likely observation before the regression and the most likely observation after it are the same observation: none of the four refused.

The fix is a second set, small and deliberately constructed, that oversamples the categories where a shift in refusal behaviour would hurt: medical-adjacent questions if you serve a health product, self-harm-adjacent phrasings if you serve consumers, security questions if you serve engineers, anything your legal team has an opinion about. Thirty to sixty cases is enough, half of which should be in-policy inputs that a correctly behaving model answers, and half genuinely out-of-policy inputs it should decline. Both directions matter, because a port that reduces refusals on the second half is as much a regression as one that increases them on the first.

A measurement of your prompt, not of the model#

Nothing this produces is a benchmark, and the numbers do not generalise past your prompt. A model that scores 94% on your schema adherence with your 1,400-token system message and your three tool schemas may score 99% with a different one, and the published evals that say model B leads model A on tool use are answering a question about their harness rather than yours. Nothing in a leaderboard tells you whether your resolve_ticket schema, with its two enums and its nullable date, is one the second model handles well.

A leaderboard reads as transferable because it appears to be measuring the same named thing — tool use is tool use. What it measures is one harness's schemas against one set of tasks, and schema adherence does not behave like a general skill that scales up and down. It is a per-schema outcome. The failures in the table above are all failures of near-miss: a plausible neighbouring value where an enum member was required, a string where a number was declared, a field invented because it looked like it belonged. Whether your schema presents those opportunities — whether it has an enum whose members are near-synonyms, or a field that is legitimately absent half the time — is a fact about your schema, and the benchmark never saw it.

It is also not a decision. Measuring what broke tells you the size of the gap; it does not tell you whether the gap is worth the saving that motivated the port, which depends on repair costs, latency budget and the price difference, and which a later chapter in this topic works through as an explicit number you commit to in advance. Nor does it cover what you do about a divergence once you find one — the patching, and the way patching creates its own regressions, comes later.

And it is not a substitute for watching production. An offline set is a fixed sample of a distribution that moves, so a port that is clean on 260 frozen cases can still be wrong about the traffic that arrives in March. The nine days in the account above were nine days of a production signal nobody was reading, and the same instrumentation that eventually found the refusal shift would have found it on day one had refusal rate been broken out by ticket tag.

What the measurement does establish is that portability is a property you observe rather than one you assume, and that the observation needs four axes, a frozen set, matched intent on decoding, and repeats. The next question is what to conclude when the two sides genuinely differ. The triage team's first instinct, when they saw the schema violations, was that the new model was worse at following instructions — and that read, which is the natural one, is usually the wrong one.