3.3.5
On-Policy vs Off-Policy Learning
Understanding the differences in data collection, experience reuse, sample efficiency and stability.
What is it?
On-Policy methods learn using data generated by the current policy. Off-Policy methods can learn from data generated by other or older policies, allowing previously collected experience to be reused.
Key points
1. On-Policy uses experience generated by the current policy.
2. After policy updates, fresh data must generally be generated.
3. On-Policy methods include REINFORCE, PPO and A2C.
4. Off-Policy can use experience from different policies.
5. Previously collected data can be reused.
6. Off-Policy methods are generally more sample-efficient.
7. Off-Policy learning can be harder to stabilize due to distribution mismatch.
8. Q-Learning, DQN and SAC are Off-Policy examples.
Example: Learning from fresh and stored experience
On-Policy:
A robot collects new navigation experiences using its current policy. After updating the policy, it collects fresh experiences again.
Off-Policy:
The robot stores previous experiences and can reuse them to improve its learning, even when those experiences were generated by an older policy.
Real-world application
This distinction helps select training strategies when data collection is expensive or previously collected experience is available.
Key takeaway
On-Policy learns from the current policy; Off-Policy can reuse experience generated by other policies.