4.4.1
PPO – Proximal Policy Optimization
Explore the clipped surrogate objective, GAE, critic network and full RLHF training loop.
What is it?
Proximal Policy Optimization (PPO) is the workhorse reinforcement learning algorithm for aligning large language models with human preferences. It uses a clipped surrogate objective to prevent excessively large policy updates.
Key points
1. PPO stands for Proximal Policy Optimization.
2. Uses a clipped surrogate objective.
3. Uses Generalized Advantage Estimation (GAE) for advantage estimation.
4. Includes a critic network to estimate state values.
5. Forms the basis of a complete RLHF training loop.
6. Designed for stable policy optimization.
7. The source describes PPO as the workhorse behind GPT-4 and Claude.
Example: Updating an assistant with PPO
An assistant generates responses to user prompts.
A reward model scores the responses. GAE estimates their advantages, while the critic estimates expected values.
PPO updates the policy using a clipped objective, limiting how far the new policy moves from the previous policy.
This helps improve responses while maintaining training stability.
Real-world application
PPO is useful when implementing RLHF pipelines that require reward optimization, advantage estimation and controlled policy updates.
Key takeaway
PPO combines clipped policy optimization, GAE and a critic network for stable RLHF training.