OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Text Generation: Decoding Methods
Lesson 75 of 415 6 min read
1.12.9

Repetition Penalties

What is it?

Repetition penalties reduce the probability of tokens that have already appeared in the generated response. They modify token logits before the next-token probability distribution is calculated.

Key points

A multiplicative repetition penalty adjusts logits for previously generated tokens. The source describes typical penalty factors around 1.1–1.3. Frequency penalty increases with the number of occurrences. Presence penalty applies a flat penalty if a token has appeared at least once. These penalties can be used alongside sampling strategies.

Reducing Repeated Words

If a model repeatedly generates the same word or phrase, a repetition penalty reduces the likelihood of selecting those previously generated tokens again. A frequency penalty increases the adjustment when a token has appeared multiple times, while a presence penalty applies once it has appeared.

Real-world application

Useful in chatbots, long-form generation and other applications where repeated words or phrases reduce response quality.
Key takeaway

Repetition penalties discourage repeated tokens but should be tuned carefully to avoid unnatural wording.