5.1.3
Why PPO Replaced TRPO in Practice
Understand PPO’s first-order clipped objective and practical advantages.
What is it?
Proximal Policy Optimization (PPO), introduced in 2017, achieves similar stability to TRPO using a simpler first-order clipped objective. It avoids expensive second-order optimization, is easier to implement and scales well to distributed training.
Key points
Introduced in 2017; Uses a clipped surrogate objective; Uses first-order optimization; Avoids expensive second-order methods; Simpler to implement than TRPO; Scales to distributed training.
TRPO vs PPO: Two Approaches to Stable Updates
TRPO controls policy movement through a KL divergence constraint and second-order optimization. PPO uses a clipped objective to limit excessive policy updates with first-order gradients. PPO offers a simpler implementation while retaining the practical goal of stable policy learning.
Real-world application
PPO is widely used in LLM reinforcement learning pipelines, including RLHF, where the policy needs to improve using reward feedback without making excessively large updates.
Key takeaway
PPO makes policy optimization simpler by using a first-order clipped objective instead of TRPO’s expensive second-order optimization.