OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 303 of 415 5 min read
7.5.9.3

Total Variation and KL Divergence

Compare the Total Variation and KL divergence measures used to define DPPO trust-region constraints.

What is it?

The source identifies Total Variation and KL divergence as the two measures used by DPPO to compare old and new policy distributions.

Key points

Total Variation is abbreviated TV.|KL means Kullback–Leibler divergence.|Both measure differences between policy distributions.|The selected measure defines the trust-region constraint.

Two divergence measures in DPPO

DPPO can use Total Variation or KL divergence to measure the difference between the new policy and the old policy. The selected measure determines how the divergence threshold is evaluated.

Real-world application

Compare TV-based and KL-based constraints when studying DPPO implementations.
Key takeaway

DPPO uses TV or KL divergence instead of relying only on probability-ratio clipping.