7.5.9.3
Total Variation and KL Divergence
Compare the Total Variation and KL divergence measures used to define DPPO trust-region constraints.
What is it?
The source identifies Total Variation and KL divergence as the two measures used by DPPO to compare old and new policy distributions.
Key points
Total Variation is abbreviated TV.|KL means Kullback–Leibler divergence.|Both measure differences between policy distributions.|The selected measure defines the trust-region constraint.
Two divergence measures in DPPO
DPPO can use Total Variation or KL divergence to measure the difference between the new policy and the old policy. The selected measure determines how the divergence threshold is evaluated.
Real-world application
Compare TV-based and KL-based constraints when studying DPPO implementations.
Key takeaway
DPPO uses TV or KL divergence instead of relying only on probability-ratio clipping.