Logits and the next-token distribution

Topic 1.5 established the decoding loop: one decoding step per output token, each step reading everything produced so far, each step ending with one token chosen from its output. It named that choice and left it closed. Opening it is what everything below does. Sampling is not a new mechanism arriving on top of the loop — it is the half of the loop that 1.5 pointed at and did not enter.

What a step's forward pass hands over is a row of logits. A logit is one raw, unnormalized score for one entry in the vocabulary, produced at that position before anything turns the row into probabilities and before anything picks from it. Every entry in the vocabulary gets one, at every step, whether or not it has any business appearing there.

A row that size is easier to read on a position where only a handful of entries could plausibly win. Take an invoice-processing service extracting a payment_status field as JSON. The schema allows paid, overdue and pending, and the model has just written "payment_status": ". The next token is the value. Six of the row's entries matter enough to look at.

token        logit
paid          6.0
Paid          5.4
PAID          4.8
overdue       3.0
pending       2.4
pineapple    -4.0

paid scores highest, its two casings sit close behind it, the other two legal values are well back, and a fruit the schema has never heard of is down at -4.0. Every number in that block is the output of the forward pass and the input to everything that follows. Sampling never changes these scores or what produced them; it only changes which token gets picked once they exist. A model that scored the wrong token highest to begin with has already made its mistake, and no setting in a request body reaches back far enough to unmake it.

A little bit of history#

The simplest rule anyone could write for a row like that is to take the single highest-scoring token, every step, without exception. It needs no parameters and no random numbers, and it puts paid in the JSON here.

On open-ended generation the same rule produces something a reader recognises immediately. Ask for a product description and the output is flat and serviceable and slightly dead. Ask twice and the second answer is the first answer, word for word, because the same prompt produces the same scores and the same top score gets taken at every position. Ask for a longer piece and it starts circling: a phrase that scored highest once will score highest again in the same neighbourhood, and there is nothing in the rule to stop it. Dull, repetitive, and identical on every rerun of the same prompt — those are not three problems, they are one rule's signature.

The way out of that signature is to stop reading the row at its peak and start reshaping the row itself before anything is picked from it. Lujun Li, Lama Sleem, Niccolò Gentile, Geoffrey Nichil and Radu State's Exploring the Impact of Temperature on Large Language Models: Hot or Cold? measures what that deliberate reshaping buys. The study swept temperature from 0.1 to 1.9 across six capability benchmarks and twelve open-source models grouped into small, medium and large size bands, and reports the maximum relative performance spread across those settings at 192.32% for machine translation and 186.81% for creativity in the small-model band. Where the best setting sits is not one place:

Read down the last column and no row says yes. That is the paper's own summary of its sweep: no single temperature is optimal across tasks, higher is not always better for creative writing, and zero is not always better for instruction following. Taking the top score is the bottom end of that sweep, and the bottom end is not reliably where the best reading was found.

Getting off that bottom end took more than one idea, and the ideas that came out of it do not do the same job as each other.

The controls between a score and a chosen token#

The first of those ideas is temperature, which reshapes every score in the row without discarding any of them. Top-k and top-p discard scores outright and keep only some. At their extreme those settings collapse into a deterministic rule that has its own name, which is worth giving only once both mechanisms are on the page to see it in.

Those three act at one moment in the generation; other controls reach further out, and it is worth naming them by how far they reach. Frequency and presence penalties act across an entire completion rather than at one step. Seeds and output variance describe what happens across repeated calls rather than within one.

Those are three different scopes, and the scope is what decides whether a parameter can possibly explain a behaviour. Temperature, top-k and top-p act on one decoding step's logits and know nothing about any other step. Frequency and presence penalties act on the history of a whole completion, so they need tokens already generated before they do anything at all. Seeds and output variance describe behaviour across separate requests, which is a scope no single request can be inspected to see.

Temperature#

Temperature acts directly on the logits vector.

To turn logits into probabilities, softmax exponentiates each logit and then divides by the sum of all the exponentials. Temperature divides every logit by the same number before that happens. Dividing by a number below one spreads the row out, which widens the gaps, which sends the softmax's output toward the single highest score. Dividing by a number above one squeezes the row together, which narrows the gaps, which pulls the softmax's output toward uniform. The order of the row never changes, because dividing every entry by the same positive number cannot reorder them.

The arithmetic runs in three moves. Divide each logit by T. Exponentiate each result. Divide each of those by the sum of all of them, so that what comes out sums to one. Run the payment_status row through that at three settings and the shape is visible without any claim about it being necessary.

token        logit     T=1.0      T=0.3       T=1.5
paid          6.0      51.9%      86.7%       42.6%
Paid          5.4      28.5%      11.7%       28.6%
PAID          4.8      15.6%       1.6%       19.1%
overdue       3.0       2.6%     0.0039%       5.8%
pending       2.4       1.4%     0.0005%       3.9%
pineapple    -4.0    0.0024%    2.9e-13%      0.055%

Read the T=0.3 column against the T=1.0 column. The 0.6 gap between paid and Paid in the logits becomes a gap of 2 after the division, exponentiation turns that into a factor of about seven, and paid walks off with 86.7% of the mass. Read the T=1.5 column the same way: the same 0.6 gap becomes 0.4, the factor drops to about one and a half, and paid is down to 42.6% while overdue has more than doubled its share.

Every one of those columns costs the same thing to produce: one division across a fixed-size vector, one exponential per entry, one sum, one division by that sum, done once per decoding step no matter how long the completion runs or how many completions are requested. Temperature does not lengthen the row, add passes, or change what the model computed to fill the row in. A request at T=1.5 and the same request at T=0.3 are the same amount of arithmetic.

The T=1.5 column also shows temperature's limit, and shows it in the bottom rows. overdue at 5.8%, pending at 3.9%, pineapple at 0.055% — none of them is zero, and none of them becomes zero at any temperature, because exponentiating a finite number cannot produce zero and dividing by a finite sum cannot either. Temperature reshapes but never discards, so a wrong or nonsensical token keeps some nonzero chance of being drawn however far down the row it sits. Run that extraction enough times and pineapple eventually appears in the JSON.

Top-k sampling#

Reshaping alone can never rule a token out, so ruling one out needs a different kind of rule: one that removes candidates from the row rather than re-proportioning them.

Top-k sampling keeps only the k tokens with the highest scores, discards everything else, and renormalizes what survives before a token is drawn.

Temperature and top-k are alike in where they act. Both take the same logits vector for the same single decoding step, both run before anything has been chosen, and both hand on a distribution for the draw to work from. They differ in what they do to it. Temperature reshapes every entry and keeps every entry; top-k leaves the surviving entries in exactly the proportions it found them and throws the rest away.

Take the T=1.5 column and set k=3.

k=3
  kept       paid 42.6    Paid 28.6    PAID 19.1        sum 90.3
  dropped    overdue 5.8    pending 3.9    pineapple 0.055
  paid       42.6 / 90.3 = 47.2%
  Paid       28.6 / 90.3 = 31.7%
  PAID       19.1 / 90.3 = 21.1%

k=1
  kept       paid 42.6                                  sum 42.6
  dropped    everything else
  paid       42.6 / 42.6 = 100%

The survivors keep their ratios to each other — paid was 1.49 times Paid before and is 1.49 times Paid after — and the 9.7 points of mass that went out with the discarded tokens are redistributed in those same ratios. That division is the whole of what renormalizing means here.

The k=1 block is the trivial rule from the history section arriving by a different route. One survivor divided by itself is one, the draw has a single candidate, and the highest-scoring token is taken every step. The name for that is a section away and worth the wait, because it is about to arrive from a second direction too.

Now a different position, in a different feature. A headline generator is asked for an opening word for a new-feature announcement, and the model has no strong opinion at all.

token          logit   probability
Introducing     5.0      20.2%
Meet            4.9      18.3%
Say             4.8      16.6%
Announcing      4.7      15.0%
Discover        4.6      13.6%
Presenting      4.5      12.3%
everything else   —       4.0%

In this position, six openings sit within half a logit of each other, and every one of them is a headline a copywriter would sign off. Set k=3 here.

k=3
  kept       Introducing 20.2    Meet 18.3    Say 16.6      sum 55.1
  dropped    Announcing 15.0    Discover 13.6    Presenting 12.3
  Introducing  20.2 / 55.1 = 36.7%
  Meet         18.3 / 55.1 = 33.2%
  Say          16.6 / 55.1 = 30.1%

Forty-one points of mass go in the bin, and what goes with them is Announcing, Discover and Presenting — three usable headlines whose only offence was sorting fourth, fifth and sixth in a field where the gap between first and sixth is half a logit. The same k=3 on the payment_status position did the opposite: it spent two of its three slots on Paid and PAID, casings of an answer the schema will reject, which nobody wanted kept.

A fixed count is the wrong size whenever confidence differs from one position to the next. Too loose where the model is sure and holding junk it should have dropped, too tight where the model is genuinely open and dropping candidates it should have held. There is no k that is right for the invoice extractor and the headline generator, and no k that is right for two consecutive positions in one completion either.

Top-p, or nucleus, sampling#

A fixed count cannot track how confident a distribution actually is, so tracking it needs a cutoff derived from the distribution rather than declared in advance.

Top-p sampling, also called nucleus sampling, keeps the smallest set of highest-scoring tokens whose cumulative probability reaches a threshold p, discards the rest, and renormalizes what survives. The number of tokens kept is whatever that threshold happens to require, which depends on the shape of the row rather than on anything set in the request.

Top-k and top-p are alike in what they do to the candidate set. Both discard outright, both leave survivors in their found proportions, and both renormalize before the draw. What varies between them is where the boundary comes from: a count fixed in advance against a count that falls out of the data. The threshold is on cumulative probability mass, not on a share of the vocabulary, which is why p=0.9 over a vocabulary of a hundred thousand entries routinely keeps a handful of them.

payment_status at T=1.5, p=0.9

token        probability   cumulative
paid            42.6%         42.6%
Paid            28.6%         71.2%
PAID            19.1%         90.3%   <- reaches 0.9; the nucleus closes here
overdue          5.8%         96.1%
pending          3.9%         99.9%
pineapple      0.055%        100.0%

Three tokens, and they are paid, Paid and PAID — the same three top-k=3 kept on this row, arrived at without anyone naming a three.

Now hold p at 0.9 and change the row instead.

headline openings at p=0.9

token          probability   cumulative
Introducing        20.2%        20.2%
Meet               18.3%        38.6%
Say                16.6%        55.1%
Announcing         15.0%        70.1%
Discover           13.6%        83.7%
Presenting         12.3%        96.0%   <- reaches 0.9; the nucleus closes here
everything else     4.0%       100.0%

Six tokens, from that same p=0.9, on a row where no candidate is far enough ahead to carry the threshold on its own. That pair is what makes the candidate count confidence-dependent rather than merely variable: p did not move, the distribution did, and the nucleus went from three wide to six wide on its own.

Now hold the headline row still and move p instead.

the same six openings, the same probabilities, p now 0.5

token          probability   cumulative
Introducing        20.2%        20.2%
Meet               18.3%        38.6%
Say                16.6%        55.1%   <- reaches 0.5; the nucleus closes here

  Introducing  20.2 / 55.1 = 36.7%
  Meet         18.3 / 55.1 = 33.2%
  Say          16.6 / 55.1 = 30.1%

Tightening p that far leaves half the candidates, and Announcing, Discover and Presenting are gone. That is the setting to reach for when a completion is mostly fine and only occasionally not, and the reason is in what "occasionally" means mechanically. An otherwise fluent output that goes incoherent once in a while is almost never the sign of a generally flat distribution — a generally flat distribution produces uniformly loose prose, not good prose with one bad patch. It is one draw at one step out of several hundred landing on a single low-probability tail token. Because every later step conditions on everything produced before it, that one token is not a local blemish: the rest of the sentence is generated to follow it, and the model will write coherent continuations of nonsense as readily as coherent continuations of sense.

The two exhibits above say which control removes that. Lowering temperature would squeeze the whole headline row — the 20.2 / 18.3 / 16.6 body as much as the 4.0 of tail — and the near-equal alternatives in that body are the entire reason the feature exists. Lowering p leaves Introducing through Presenting at the ratios the model gave them and cuts only the mass those bad draws come from. That is the case for tightening top_p and leaving temperature alone when a high-variety feature is occasionally incoherent and the variety is still wanted.

Top-p has its own limit, and it shows up on a row where the model is not uncertain at all. A different extraction, this time a case_status field whose schema allows open and closed.

case_status at p=0.9

token          probability   cumulative
in_review          78%           78%
"Closed "          14%           92%   <- reaches 0.9; the nucleus closes here
closed              5%           97%
open                3%          100%

The nucleus that closes there is two tokens wide. One of them is a status the schema does not contain and the model invented; the other is the right word with a trailing space, which will not match an enum comparison. The only two values that would pass validation are closed at 5% and open at 3%, and both of them are outside the cutoff.

That row is the whole limit in miniature: a threshold on cumulative mass says nothing about whether the tokens inside that mass are good ones. Where the model is confidently wrong, top-p tightens around the wrong answer, and it tightens harder the lower p goes: at p=0.8 the nucleus here is in_review alone, at 100%.

Greedy decoding, and how the dial bottoms out#

But bottoming out a dial is not unique to top-k. The rule described in the history section — take the single highest-scoring token, every step — and the k=1 collapse worked out under top-k are the same behaviour reached from different directions, and it has a name. Greedy decoding is the rule that always takes the single highest-scoring token. Temperature at 0 and top-k at 1 both converge on it.

Greedy decoding and a sampled draw are alike in what they deliver: each ends a decoding step by naming exactly one token, and the loop cannot tell which produced it. They differ in whether anything is left to chance. A draw consumes a random number and can return a token that was not the highest-scoring one; greedy decoding consumes nothing and cannot.

That a setting can be bottomed out from either direction raises the question of what happens to the settings around it. Seeing why an extreme setting silences another setting means seeing the order the settings run in.

1  logits          one raw score per vocabulary entry, from the forward pass
2  temperature     divide every logit by T
3  truncation      apply top-k, or top-p, or both; discard what falls outside
4  renormalise     divide each surviving probability by the surviving total
5  draw            take one token from what remains

Temperature runs at step 2 and truncation at step 3. Temperature is applied before truncation, not after it and not alongside it, which means truncation never sees the row the model produced — it sees whatever temperature handed on. Temperature at 0 is handled as a direct argmax rather than a division by zero.

Run the payment_status row through that order with temperature=0 and top_p=0.9 both set.

step 2, T=0, argmax over the logits
  paid       100%
  Paid         0%
  PAID         0%
  overdue      0%
  pending      0%
  pineapple    0%

step 3, top_p=0.9 over the distribution above

token        probability   cumulative
paid              100%         100%   <- reaches 0.9 on the first token
Paid                0%         100%
PAID                0%         100%
overdue             0%         100%
pending             0%         100%
pineapple           0%         100%

step 4   100 / 100 = 100%
step 5   paid

top_p=0.9 ran. It found the threshold already crossed by the first candidate, closed the nucleus there, and discarded five tokens carrying no mass between them. Removing zero probability from a distribution leaves the distribution where it was, so the renormalisation is a division of 100 by 100 and the draw has one candidate. The setting is in the request, the provider accepts it, and it changes nothing about the output.

That gives the rule for choosing among the three. Temperature sets how much the whole curve is reshaped. Top-k gives a hard, predictable ceiling on candidate count. Top-p gives a ceiling that tracks the model's own confidence. Because they compose in the fixed order above rather than acting as alternatives to each other, setting one of them to an extreme can silently make another do nothing — and nothing in the response says so.

Frequency and presence penalties#

Every control so far acts on one decoding step in isolation and is blind to every other step. A completion can still go wrong across many steps while each individual step's settings look reasonable, because what goes wrong is a pattern in the sequence rather than a bad choice at any one position.

To see how such a pattern forms, consider this case. A summarisation prompt is run over five line items belonging to one invoice, INV-4471, and each summary line is supposed to open with the invoice identifier. Repeating that identifier five times is the correct behaviour here, not a bug.

INV-4471: freight surcharge, 240.00
INV-4471: pallet handling, 85.00
INV-4471: fuel adjustment, 61.50
INV-4471: after-hours delivery, 120.00
INV-4471: redelivery fee, 45.00

The two controls reached for when a completion repeats itself like this are frequency penalty and presence penalty, and they are alike in almost everything. Both are score reductions applied to a token on the basis of how many times it has already appeared among the tokens generated so far in this completion — generated, not prompted, so text the caller supplied is invisible to them. Both are subtracted from the logit at step 2's input, before temperature and before truncation. Both keep applying at every subsequent step until generation ends, so a token penalised on line two is still penalised on line five.

They differ in how the subtraction grows.

occurrences so far   frequency_penalty=0.8   presence_penalty=0.8
        1                   -0.8                    -0.8
        2                   -1.6                    -0.8
        3                   -2.4                    -0.8
        4                   -3.2                    -0.8

Frequency penalty multiplies the coefficient by the count, so the subtraction keeps growing for as long as the token keeps appearing. Presence penalty charges the coefficient once and never more, so a token that has appeared four times is in exactly the position of a token that has appeared once. At a count of one the columns agree, which is why a docs table comparing them at a single occurrence makes them look interchangeable.

Set frequency_penalty=0.8 on the invoice summary. The identifier tokenises as INV, -, 44, 71, and the token worth following is 44, whose nearest competitor at that position is 47.

line   count of `44`   penalised `44`      `47`              chosen
 1           0         6.2 - 0.0 = 6.2     4.0               44
 2           1         6.2 - 0.8 = 5.4     4.0               44
 3           2         6.2 - 1.6 = 4.6     4.0               44
 4           3         6.2 - 2.4 = 3.8     4.0               47
 5           3         6.2 - 2.4 = 3.8     4.0 - 0.8 = 3.2   44

the same prompt, the same five line items, frequency_penalty now 0.8

line 4:  INV-4771: after-hours delivery, 120.00
every other line unchanged

Line four of that summary reports a different invoice. Nothing in the output looks broken, the repetition metric the penalty was added to improve did improve, and a line item has been filed against an invoice number that does not exist. Line five swings back because 47 picked up a penalty of its own, which is worse rather than better: the output now names two identifiers for one invoice and is internally inconsistent about which is right.

A penalty cannot tell a genuine loop from a legitimately repeated fact, because occurrence count is the only thing it ever looks at. It has no access to whether the repeated token is an entity name, an order number, a figure or a verbal tic, and it applies the same subtraction to all of them. A quality change made through a penalty is therefore never a one-dimensional change — whatever the repetition metric does, something on the accuracy side of the eval suite moved at the same time, and only one of them was being watched.

Degenerate repetition#

A penalty is a response to repeated output rather than a diagnosis of why the output repeated, and the failure it responds to is worth naming on its own terms. Degenerate repetition is the failure mode where a model's output gets stuck on the same token, phrase or pattern instead of progressing — a list that emits its second item nine times, a paragraph that restates its own last clause until the token limit stops it.

Degenerate repetition and the invoice summary's five identical openings look identical from the outside: a token appearing again and again in the completion. What separates them is whether the repetition is the content being correct or the generation failing to move, and only one of those is a defect.

The causes sit side by side and are told apart by looking at different things. A genuinely low temperature, or a top-k tight enough to be near-greedy, on a task where variety was needed: the model has nowhere else to go and takes the same high-scoring continuation each time it returns to the same neighbourhood. A prompt or system message that states the same instruction twice: the model treats it as two things to satisfy rather than one, and satisfies it twice. A system-plus-user message pair where an instruction was copy-pasted into both, which is the same fault arriving by accident rather than by drafting. Or source text that is itself repetitive, in which case the output is a faithful summary of something that repeats.

Only the first of those is a sampling problem. The second — the duplicated instruction — is worth putting on the page as a pair of prompts, one carrying the fault and one with it removed.

system: You summarise invoice line items. Summarise the line items below, one
        line each, opening each line with the invoice identifier.

user:   Summarise the line items below, one line each, opening each line with
        the invoice identifier.

        INV-4471   freight surcharge       240.00
        INV-4471   pallet handling          85.00
        INV-4471   fuel adjustment          61.50
        INV-4471   after-hours delivery    120.00
        INV-4471   redelivery fee           45.00

the same system message, the same five line items, the instruction removed
from the user turn

user:   INV-4471   freight surcharge       240.00
        INV-4471   pallet handling          85.00
        INV-4471   fuel adjustment          61.50
        INV-4471   after-hours delivery    120.00
        INV-4471   redelivery fee           45.00

Under the duplicated prompt, the model produces the five-line summary and then produces it again, because two instructions were issued and both were followed. Under the deduplicated one it produces it once. No sampling parameter was touched between them, and no sampling parameter would have distinguished them: a penalty applied to the first prompt would have suppressed the repeated tokens of a second summary the model was right to think it had been asked for, at the cost demonstrated by line four of the invoice output above.

So the order is fixed. Check the prompt and the context for exactly this kind of duplication or contradiction before reaching for a penalty at all. A penalty applied to a genuinely repetitive but correct prompt buys the factual damage and leaves the repetition, which a rewritten prompt would have removed for free.

Seeds and reproducibility#

Every control up to here, the penalties included, governs a single request. What happens when that same request is run again is a separate question, and it becomes urgent the first time a production incident has to be reproduced from a log line.

A random seed is a value passed to the sampler that pins the pseudo-random draws it uses when choosing among candidate tokens, so that rerunning with the same seed and the same inputs reproduces the same draws. A seed and greedy decoding are alike in the outcome they are reached for — a rerun that returns what the first run returned — and they differ in how they get there. Greedy decoding removes the draw; a seed fixes it. A seed only has something to pin where there is a draw to make, so at temperature 0 it has nothing to do at all.

The seed initialises the sampler's pseudo-random generator once, at the start of the request, and every draw in the completion comes off that single generator in sequence. It is not a per-token quantity that gets spent. Length alone therefore never breaks it: a five-hundred-token completion is exactly as reproducible as a five-token one, and two seeded runs that diverge have not diverged because the seed ran out partway through. Ruling that out is what sends an investigation to the right place, which is underneath the logits rather than at the sampler.

{
  "model": "extractor-v2",
  "seed": 4271,
  "temperature": 1.0,
  "logprobs": true,
  "top_logprobs": 2,
  "messages": [
    { "role": "user", "content": "Extract payment_status from the invoice below." }
  ]
}

served under batch size 8, kernel build 2024.11
top-2 logprobs for the token after "payment_status": "
  paid   -0.31
  Paid   -1.34

the same body, the same prompt, the same "seed": 4271

served under batch size 32, kernel build 2025.02
top-2 logprobs for the token after "payment_status": "
  paid   -0.33
  Paid   -1.31

The seed is identical, the prompt is identical, and the scores the sampler drew from are not. paid moved from -0.31 to -0.33 and Paid from -1.34 to -1.31, which closes the gap between them by 0.05. At a position where two candidates are already near a boundary, a gap that narrows by 0.05 is enough for the same pinned draw to land on the other side of it and return a different token.

A seed pins the draws the sampler makes over a given set of logits. It does not pin the arithmetic that produced those logits. Topic 1.17 covers why an identical seed and an identical prompt can still yield different logits on a shared serving stack; topic 1.7 covers logprobs, which is the instrument for checking whether a seeded rerun diverged in the scores before it diverged in the chosen tokens. Topic 2.22's model deprecation and version pinning is the other place a seeded reproduction quietly stops reproducing, because a model identifier that resolves to a different build resolves to different arithmetic.

Providers describe seeded reproducibility as best-effort rather than guaranteed. That is the accurate word rather than a hedge: nothing about a seed constrains anything upstream of the draw it controls, so a guarantee would be a guarantee about a system the seed has no contact with.

Output variance across repeated calls#

Because seeds control only the sampler's pseudo-random draws and not the model that produces the row, how much repeated calls differ from each other is left to the other controls. Run a request once and the settings decide one completion. Run it repeatedly and the settings decide a population. Output variance is the degree to which outputs differ from each other across repeated generations of the same prompt, and what drives it is mainly temperature and top-p — the controls that decide how much of the row survives to be drawn from and how flat that survivor set is.

Output variance and the distinctness of items within one completion are alike in what someone asks for when they reach for either: outputs that are not copies of each other. They differ in where the differing happens. Output variance is across separate requests, which share nothing but the prompt. Distinctness within one completion is inside a single sequence, where every item can see every item before it.

To see that difference in action, consider the headline generator. Five calls at a low temperature come back with openings that are near-copies of each other, because a sharpened row puts most of its mass on Introducing and the draw finds it five times out of five. Five calls at a higher temperature come back with five genuinely different openings, because the row at that setting spreads its mass across six near-equal candidates and five independent draws over that row will not agree.

Which is why the instinct to raise temperature when a feature needs ten distinct options is aimed at the wrong thing. Within one completion the model conditions on every token it has already produced in that same completion, which is 1.5's causal dependency doing ordinary work: by the time it is writing option seven, options one through six are in its input, and an instruction to produce several distinct options is an instruction it can already act on with what it can see. Repeating itself is a continuation it has been told not to write and can observe itself about to write. Raising temperature does not target that at all. It reaches every position of the completion alike — the choice of headline, and equally the comma after it, the capital letter that opens it, and the spelling of the product name.

The difference between those two moves is what they act on. Repeated, independent calls and an explicit distinctness instruction act on the thing that was actually asked for, which is options differing from each other. Temperature acts on every choice in the completion, most of which nobody wanted varied.

Choosing settings for a feature#

The invoice extractor and the headline generator have been running side by side through every control above, and they want opposite settings for reasons that are now all on the page.

{
  "temperature": 0,
  "top_p": 1,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "response_format": { "type": "json_schema" }
}

{
  "temperature": 1.1,
  "top_p": 0.85,
  "frequency_penalty": 0,
  "presence_penalty": 0
}

temperature is the only field that moves far, and everything else follows from what each feature's output is for. The extractor's top_p sits at 1 because at temperature 0 any other value would be a parameter that runs and changes nothing, and a value of 1 says so in the request body instead of hiding it. The headline generator's top_p sits below 1 because a widened row has a tail, and the tail is where a single incoherent draw comes from. Both penalties are off in both features, for different reasons: the extractor is expected to repeat field names and digits, and the headline generator's completions are too short for an occurrence count to reach a magnitude that matters.

Step 9 is the one that catches people after the fact. Raising temperature or loosening top_p on anything whose output is compared as a string moves the score without touching the model, and the movement shows up as a regression nobody made. Step 11 is the one that saves the most work: once temperature is at 0 and structured generation is on, syntactic failure is gone, and what remains is a smaller and different population — invented enum values, missing required fields, the in_review case — which a repair pass over valid JSON cannot fix.

That leaves the question of who is allowed to move any of it. A product manager can move temperature and top_p on a creative feature within a bounded range, because the worst case is headlines somebody dislikes and the range caps how far it goes. Anything feeding a parser or an evaluation stays behind code review, because a downstream contract depends on it and an eval will fail if it moves. The line between those paths is not about trust. One of them has a test that can fail and the other does not.

Whichever of the three scopes a parameter acts in — one decoding step, one completion's history, the pattern across separate requests — none of them reaches where the seed section stopped. The draw is the last operation in the order, everything a sampling parameter touches happens after the scores exist, and the two places that arithmetic can move on its own are the ones that section pointed at: 1.17 for a shared serving stack, 2.22 for a model version shifting underneath a pinned request. That is the boundary the opening drew, arriving from the other end — the same reason nothing in a request body could have rescued a row that scored pineapple highest.