3.9.2
What Is Off-Policy Learning?
Understand learning from data generated by older or different policies.
What is it?
Off-policy learning can use experience generated by any policy, including older policy versions or other agents. Stored experience can be reused for multiple learning updates, improving sample efficiency but introducing possible distribution mismatch.
Key points
1. Data can come from any policy.
2. Older experience remains usable.
3. Replay buffers can store and reuse past transitions.
4. Sample efficiency is generally higher.
5. Distribution mismatch can make training unstable or cause divergence.
6. Examples: Q-Learning, DQN and SAC.
7. The source also classifies DPO as off-policy because it uses a static preference dataset.
Example: Learning from historical warehouse journeys
A warehouse robot stores journeys completed under previous policy versions.
Instead of discarding every journey after a policy update, the robot can sample stored experiences and learn from them again.
This reduces the need to collect new experience for every update, although older experiences may not perfectly represent the robot current behavior.
Real-world application
Off-policy learning is useful when collecting new experience is expensive and historical data can be reused.
Key takeaway
Off-policy learning reuses experience generated by older or different policies, improving sample efficiency but requiring attention to distribution mismatch.