1.12.4
Top-k Sampling
What is it?
Top-k sampling restricts the next-token candidates to the k highest-probability tokens. The remaining tokens are removed, and the probabilities of the retained tokens are renormalized before sampling.
Key points
Only the top k tokens remain eligible. Probability mass is redistributed among the retained tokens. Small k values restrict variation. Larger k values allow more alternatives. Unlike greedy decoding, the selection is sampled rather than always choosing the highest-probability token.
Top-k Sampling Example
After the phrase The cat sat on the..., suppose the most likely continuations are mat, floor and couch. With a suitable k value, the model samples from these leading candidates while ignoring extremely unlikely alternatives.
Real-world application
Useful for general-purpose text generation when some variation is desired but very unlikely token choices should be excluded.
Key takeaway
Top-k controls diversity by limiting the number of eligible next-token choices.