OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / On-Policy vs Off-Policy — Detailed Comparison
Lesson 135 of 415 7 min read
3.9.4

On-Policy and Off-Policy in RLHF

Understand how PPO, GRPO, DPO and Online DPO use training data.

What is it?

In LLM reinforcement learning, PPO and GRPO are on-policy methods because they generate responses using the current policy, compute advantages, update the model and generate fresh responses again. DPO is off-policy because it trains on a fixed preference dataset without generating responses during training.

Key points

1. PPO: Generates responses using the current policy. 2. GRPO: Generates fresh responses and uses group-based reward comparisons. 3. Both compute learning signals, update the policy and generate again. 4. DPO: Uses a fixed preference dataset. 5. DPO does not require response generation during training. 6. DPO is cheaper but can suffer from distribution shift as the policy changes. 7. Online DPO generates fresh responses from the current policy and applies DPO loss. 8. Online DPO combines fresh data generation with a preference-based optimization objective.

Example: Three approaches to training an AI assistant

PPO: Generate answers using the current model, score them, calculate advantages and update the model. GRPO: Generate multiple answers for a prompt, compare their rewards and update the model using group-relative signals. DPO: Use previously collected preferred and rejected answer pairs to train the model without generating new answers during training. Online DPO: Generate fresh answer pairs using the current model, score them and apply the DPO objective.

Real-world application

These approaches represent different trade-offs between fresh experience generation, training cost and preference-data reuse.
Key takeaway

PPO and GRPO use fresh policy-generated data; standard DPO uses a static preference dataset; Online DPO combines fresh generation with DPO optimization.