Top-k and nucleus sampling

Both truncate the candidate set before a token is drawn, and they cut on different axes. Top-k keeps exactly k candidates whatever their probabilities, so it retains a long junk tail on a peaked distribution and discards most of the model's proposals on a flat one; nucleus sampling keeps however many candidates it takes to reach mass p, so the count moves with the distribution. This chapter runs one set of scores through both filters, records where they disagree, and shows where a fixed cutoff still earns a place.

Sampling — read the current version at https://daviddominguez.dev/learn/en/sampling/model-temperature

Back to top