1.12.5
Top-p (Nucleus) Sampling
What is it?
Top-p sampling, also called nucleus sampling, selects the smallest set of tokens whose cumulative probability reaches a chosen threshold p. The candidate set changes according to the model distribution at each step.
Key points
Candidates are ordered by probability. Tokens are included until cumulative probability reaches p. The retained probabilities are renormalized. The number of eligible tokens can change at every step. The method adapts to the model confidence.
Top-p Sampling Example
If p is set to 0.9, the model retains the smallest group of highest-probability tokens whose combined probability is at least 90%. It then samples the next token from that group.
Real-world application
Useful for open-ended generation where the number of plausible next tokens changes depending on the context.
Key takeaway
Top-p uses cumulative probability rather than a fixed number of candidate tokens.