7.5.9.2
Direct Policy Divergence Constraints
Learn how DPPO constrains policy updates using direct divergence estimates between old and new policy distributions.
What is it?
DPPO defines a trust-region constraint directly between the new and old policy distributions. Its objective uses a divergence measure and applies the policy update only when the divergence remains within a threshold delta.
Key points
The constraint compares the new and old policy distributions.|The divergence measure is represented by D.|The threshold is delta.|The objective uses an indicator to enforce the constraint.
Constraining policy updates with divergence
DPPO uses a condition of the form D(πθ || πold) ≤ δ. When the chosen divergence stays within the threshold, the corresponding policy contribution is retained by the objective.
Real-world application
Understand the divergence threshold when designing a custom DPPO loss.
Key takeaway
DPPO constrains policy updates using a direct divergence condition.