OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Motivation and History
Lesson 164 of 415 5 min read
5.1.3

Why PPO Replaced TRPO in Practice

Understand PPO’s first-order clipped objective and practical advantages.

What is it?

Proximal Policy Optimization (PPO), introduced in 2017, achieves similar stability to TRPO using a simpler first-order clipped objective. It avoids expensive second-order optimization, is easier to implement and scales well to distributed training.

Key points

Introduced in 2017; Uses a clipped surrogate objective; Uses first-order optimization; Avoids expensive second-order methods; Simpler to implement than TRPO; Scales to distributed training.

TRPO vs PPO: Two Approaches to Stable Updates

TRPO controls policy movement through a KL divergence constraint and second-order optimization. PPO uses a clipped objective to limit excessive policy updates with first-order gradients. PPO offers a simpler implementation while retaining the practical goal of stable policy learning.

Real-world application

PPO is widely used in LLM reinforcement learning pipelines, including RLHF, where the policy needs to improve using reward feedback without making excessively large updates.
Key takeaway

PPO makes policy optimization simpler by using a first-order clipped objective instead of TRPO’s expensive second-order optimization.