OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / GRPO Variants and Extensions
Lesson 281 of 415 5 min read
7.5.9.1

The Problem with Ratio Clipping

Understand why PPO probability-ratio clipping can over-penalise low-probability tokens and under-penalise high-probability tokens.

What is it?

PPO ratio clipping is an indirect way to constrain the KL divergence between old and new policies. However, it can over-penalise low-probability tokens and under-penalise high-probability tokens. DPPO replaces this proxy with direct divergence estimates.

Key points

Ratio clipping is a proxy for KL control.|Low-probability tokens can be over-penalised.|High-probability tokens can be under-penalised.|DPPO uses direct divergence estimates.

Why probability ratios can misrepresent policy changes

A small absolute probability change for a low-probability token can create a large ratio change. A larger absolute change for a high-probability token can create a relatively small ratio change. Standard ratio clipping may therefore constrain these tokens unevenly.

Real-world application

Consider DPPO when ratio clipping does not adequately control policy changes across tokens with very different probability levels.
Key takeaway

DPPO addresses limitations of ratio clipping by measuring policy divergence directly.