The decoding loop

Trace a greedy decoder by hand: each step runs the model over the prefix so far, selects one token, appends it, and starts again. Nothing revisits an earlier choice, so changing one word near the beginning of an answer changes the state every later token was conditioned on, and the rest diverges completely — measurably, in 92% of cases within twelve tokens. The exchange rate that fixes everything later in this topic is set here: one token of output costs one full invocation of the model.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/the-decoding-loop

Back to top