5.3.3
Entropy Bonus and Exploration
Understand how entropy encourages exploration and prevents premature convergence.
What is it?
The entropy bonus encourages the policy to retain some uncertainty in its action distribution. It prevents premature convergence to a deterministic policy and supports exploration. The source specifies an entropy coefficient of 0.01.
Key points
1. Entropy measures uncertainty in the policy distribution.
2. Higher entropy indicates more distributed action probabilities.
3. Lower entropy indicates a more concentrated distribution.
4. The entropy bonus encourages exploration.
5. It prevents premature deterministic convergence.
6. The source specifies c2 = 0.01.
Example: Exploring Different Actions
Imagine an AI agent choosing between several possible actions.
A policy assigning probabilities of 0.9 and 0.1 is highly concentrated.
A policy assigning probabilities of 0.5 and 0.5 is more uncertain.
The entropy bonus encourages exploration rather than allowing the policy to become deterministic too early.
Real-world application
Entropy regularization helps maintain exploration while training an AI agent using PPO.
Key takeaway
The entropy bonus helps PPO maintain exploration and avoid premature convergence.