7.5.9.1
The Problem with Ratio Clipping
Understand why PPO probability-ratio clipping can over-penalise low-probability tokens and under-penalise high-probability tokens.
What is it?
PPO ratio clipping is an indirect way to constrain the KL divergence between old and new policies. However, it can over-penalise low-probability tokens and under-penalise high-probability tokens. DPPO replaces this proxy with direct divergence estimates.
Key points
Ratio clipping is a proxy for KL control.|Low-probability tokens can be over-penalised.|High-probability tokens can be under-penalised.|DPPO uses direct divergence estimates.
Why probability ratios can misrepresent policy changes
A small absolute probability change for a low-probability token can create a large ratio change. A larger absolute change for a high-probability token can create a relatively small ratio change. Standard ratio clipping may therefore constrain these tokens unevenly.
Real-world application
Consider DPPO when ratio clipping does not adequately control policy changes across tokens with very different probability levels.
Key takeaway
DPPO addresses limitations of ratio clipping by measuring policy divergence directly.