1.12.9
Repetition Penalties
What is it?
Repetition penalties reduce the probability of tokens that have already appeared in the generated response. They modify token logits before the next-token probability distribution is calculated.
Key points
A multiplicative repetition penalty adjusts logits for previously generated tokens. The source describes typical penalty factors around 1.1–1.3. Frequency penalty increases with the number of occurrences. Presence penalty applies a flat penalty if a token has appeared at least once. These penalties can be used alongside sampling strategies.
Reducing Repeated Words
If a model repeatedly generates the same word or phrase, a repetition penalty reduces the likelihood of selecting those previously generated tokens again. A frequency penalty increases the adjustment when a token has appeared multiple times, while a presence penalty applies once it has appeared.
Real-world application
Useful in chatbots, long-form generation and other applications where repeated words or phrases reduce response quality.
Key takeaway
Repetition penalties discourage repeated tokens but should be tuned carefully to avoid unnatural wording.