3.9.3
On-Policy vs Off-Policy: Key Differences
Compare data sources, data reuse, sample efficiency and training stability.
What is it?
On-policy and off-policy methods differ mainly in where their training data comes from, whether old data can be reused, sample efficiency and stability.
Key points
1. On-policy data source: Current policy πθ only.
2. Off-policy data source: Any policy or stored experience.
3. On-policy old data: Must be regenerated after updates.
4. Off-policy old data: Remains reusable.
5. On-policy sample efficiency: Low; data is used once.
6. Off-policy sample efficiency: High; data can be reused many times.
7. On-policy stability: Generally more stable due to a consistent distribution.
8. Off-policy stability: Can diverge because of distribution mismatch.
9. On-policy examples: REINFORCE, PPO, A2C and GRPO.
10. Off-policy examples: Q-Learning, DQN, SAC and DPO.
Example: Comparing two training systems
System A — On-policy:
An assistant generates 1,000 responses, learns from them, updates its policy and generates another 1,000 responses.
System B — Off-policy:
An assistant stores historical training examples and reuses them across multiple updates.
System A continuously collects fresh experience. System B reuses data but must manage the risk that its training data no longer matches the current policy distribution.
Real-world application
This comparison helps select reinforcement learning algorithms based on experience-generation cost, data availability and training stability.
Key takeaway
On-policy methods prioritize fresh and consistent experience; off-policy methods prioritize reuse and sample efficiency.