OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Derivation of the PPO Gradient and Update Rule
Lesson 174 of 415 5 min read
5.4.4

Step 4: The Problem with Unconstrained Surrogates

Understand the risks of extreme importance ratios and unconstrained policy updates.

What is it?

The unconstrained surrogate objective is valid, but a single gradient step can push the probability ratio far from 1.0. This can create extreme importance weights, high variance and unstable policy behaviour. TRPO addresses this by imposing a KL divergence constraint.

Key points

1. The unconstrained surrogate is L_CPI. 2. A large update can push the probability ratio far from 1.0. 3. Extreme importance weights cause high variance. 4. The policy may enter untested regions. 5. The reward model may give unreliable scores in those regions. 6. Severe updates can cause catastrophic policy collapse. 7. TRPO constrains D_KL(πθold || πθ) ≤ δ. 8. TRPO requires expensive second-order optimization.

Example: An Excessive Policy Update

Suppose the old policy assigns probability 0.20 to an action, but an unconstrained update increases it to 0.90. Probability ratio = 0.90 / 0.20 = 4.5. This is a very large change. Such extreme ratios can produce unstable importance weights and unreliable updates.

Real-world application

When designing policy optimization, monitor policy changes and importance ratios. Unconstrained updates can destabilize training.
Key takeaway

An unconstrained surrogate can produce extreme policy updates. TRPO addresses this using a KL divergence constraint.