OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Actor-Critic Methods
Lesson 127 of 415 8 min read
3.7.4

Evolution from REINFORCE to PPO and GRPO

Understand the progression from vanilla policy gradients to actor-critic and modern LLM reinforcement learning methods.

What is it?

Actor-Critic methods form part of the progression from REINFORCE toward more stable policy optimization. A2C and A3C use TD-based advantages. TRPO constrains policy changes using KL divergence. PPO clips the policy ratio and uses first-order optimization. GRPO removes the critic and uses group statistics as a baseline.

Key points

1. REINFORCE: High variance and no bootstrapping; impractical for LLMs according to the source. 2. A2C/A3C: Use TD-based advantage estimates and reduce variance, but have unbounded step sizes. 3. TRPO: Constrains KL divergence between policy updates; stable but computationally expensive due to second-order optimization. 4. PPO: Clips the policy ratio and uses first-order optimization. 5. PPO is described as the standard for LLM RL training. 6. GRPO removes the critic entirely. 7. GRPO uses group statistics as a baseline and is effective for verifiable rewards.

Example: Evolution of an LLM training approach

REINFORCE uses complete trajectory returns to update a policy, but its estimates have high variance. Actor-Critic introduces a learned critic to provide TD-based advantage estimates. TRPO constrains policy changes to improve stability. PPO uses a clipped objective to achieve similar stability with first-order optimization. GRPO removes the critic and uses group-level reward statistics as its baseline.

Real-world application

This progression explains why PPO and GRPO are important approaches in modern LLM reinforcement learning and alignment.
Key takeaway

Actor-Critic reduces variance through a learned critic. PPO adds clipped policy updates, while GRPO removes the critic and uses group statistics.