OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Roadmap of This Part
Lesson 157 of 415 5 min read
4.4.3

GRPO – Group Relative Policy Optimization

Explore critic-free optimization using group-level reward normalization.

What is it?

Group Relative Policy Optimization (GRPO) is a critic-free reinforcement learning algorithm. It uses group-level reward normalization to estimate relative advantages, eliminating the need for a separate critic network.

Key points

1. GRPO stands for Group Relative Policy Optimization. 2. Uses groups of generated responses. 3. Compares rewards within each group. 4. Normalizes rewards to obtain relative learning signals. 5. Eliminates the critic network. 6. Associated with DeepSeek-R1. 7. The source describes it as a dominant choice for reasoning-model training.

Example: Comparing multiple reasoning attempts

An LLM receives a mathematics problem and generates several candidate solutions. Each solution is checked and receives a reward. GRPO compares the rewards within the group and uses relative performance to guide policy updates. It does not need a separate critic network to estimate the value of each state.

Real-world application

GRPO is useful for reasoning tasks where multiple candidate responses can be generated and evaluated using verifiable rewards.
Key takeaway

GRPO uses group-relative reward normalization instead of a separate critic network.