3.9.4
On-Policy and Off-Policy in RLHF
Understand how PPO, GRPO, DPO and Online DPO use training data.
What is it?
In LLM reinforcement learning, PPO and GRPO are on-policy methods because they generate responses using the current policy, compute advantages, update the model and generate fresh responses again. DPO is off-policy because it trains on a fixed preference dataset without generating responses during training.
Key points
1. PPO: Generates responses using the current policy.
2. GRPO: Generates fresh responses and uses group-based reward comparisons.
3. Both compute learning signals, update the policy and generate again.
4. DPO: Uses a fixed preference dataset.
5. DPO does not require response generation during training.
6. DPO is cheaper but can suffer from distribution shift as the policy changes.
7. Online DPO generates fresh responses from the current policy and applies DPO loss.
8. Online DPO combines fresh data generation with a preference-based optimization objective.
Example: Three approaches to training an AI assistant
PPO:
Generate answers using the current model, score them, calculate advantages and update the model.
GRPO:
Generate multiple answers for a prompt, compare their rewards and update the model using group-relative signals.
DPO:
Use previously collected preferred and rejected answer pairs to train the model without generating new answers during training.
Online DPO:
Generate fresh answer pairs using the current model, score them and apply the DPO objective.
Real-world application
These approaches represent different trade-offs between fresh experience generation, training cost and preference-data reuse.
Key takeaway
PPO and GRPO use fresh policy-generated data; standard DPO uses a static preference dataset; Online DPO combines fresh generation with DPO optimization.