3.3.2
Model-Free vs Model-Based Learning
Understanding direct policy or value learning compared with learning an environment model for planning.
What is it?
Model-Free methods learn a policy or value function directly from experience. Model-Based methods learn or use an environment model that predicts state transitions and supports planning.
Key points
1. Model-Free learns policy π and/or value V or Q directly.
2. It does not require an explicit environment model.
3. Model-Based uses a model of transitions P̂(s′|s,a).
4. Model-Based methods can simulate future trajectories.
5. Model-Based methods can be more sample-efficient.
6. Inaccurate environment models can cause errors to compound.
7. PPO, DQN and SAC are Model-Free examples.
8. MuZero, Dreamer and AlphaGo are Model-Based examples.
Example: A robot planning a route
Model-Free approach:
The robot learns which actions work well through experience.
Model-Based approach:
The robot uses a model to predict the result of possible movements and plans a route before acting.
The model-based approach can plan ahead, but its predictions must be sufficiently accurate.
Real-world application
Model-Free methods are useful when environment dynamics are difficult to model. Model-Based methods are useful when planning and sample efficiency are important.
Key takeaway
Model-Free learns directly from experience; Model-Based uses an environment model to support planning.