OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Text Generation: Decoding Methods
Lesson 70 of 415 5 min read
1.12.4

Top-k Sampling

What is it?

Top-k sampling restricts the next-token candidates to the k highest-probability tokens. The remaining tokens are removed, and the probabilities of the retained tokens are renormalized before sampling.

Key points

Only the top k tokens remain eligible. Probability mass is redistributed among the retained tokens. Small k values restrict variation. Larger k values allow more alternatives. Unlike greedy decoding, the selection is sampled rather than always choosing the highest-probability token.

Top-k Sampling Example

After the phrase The cat sat on the..., suppose the most likely continuations are mat, floor and couch. With a suitable k value, the model samples from these leading candidates while ignoring extremely unlikely alternatives.

Real-world application

Useful for general-purpose text generation when some variation is desired but very unlikely token choices should be excluded.
Key takeaway

Top-k controls diversity by limiting the number of eligible next-token choices.