3.7.4
Evolution from REINFORCE to PPO and GRPO
Understand the progression from vanilla policy gradients to actor-critic and modern LLM reinforcement learning methods.
What is it?
Actor-Critic methods form part of the progression from REINFORCE toward more stable policy optimization. A2C and A3C use TD-based advantages. TRPO constrains policy changes using KL divergence. PPO clips the policy ratio and uses first-order optimization. GRPO removes the critic and uses group statistics as a baseline.
Key points
1. REINFORCE: High variance and no bootstrapping; impractical for LLMs according to the source.
2. A2C/A3C: Use TD-based advantage estimates and reduce variance, but have unbounded step sizes.
3. TRPO: Constrains KL divergence between policy updates; stable but computationally expensive due to second-order optimization.
4. PPO: Clips the policy ratio and uses first-order optimization.
5. PPO is described as the standard for LLM RL training.
6. GRPO removes the critic entirely.
7. GRPO uses group statistics as a baseline and is effective for verifiable rewards.
Example: Evolution of an LLM training approach
REINFORCE uses complete trajectory returns to update a policy, but its estimates have high variance.
Actor-Critic introduces a learned critic to provide TD-based advantage estimates.
TRPO constrains policy changes to improve stability.
PPO uses a clipped objective to achieve similar stability with first-order optimization.
GRPO removes the critic and uses group-level reward statistics as its baseline.
Real-world application
This progression explains why PPO and GRPO are important approaches in modern LLM reinforcement learning and alignment.
Key takeaway
Actor-Critic reduces variance through a learned critic. PPO adds clipped policy updates, while GRPO removes the critic and uses group statistics.