3.2.6
Bellman Equations and Optimal Policy
Understand the recursive Bellman relationships for value functions and how Bellman optimality identifies the best policy.
What is it?
Bellman equations express value functions recursively by relating current value to immediate reward and expected future value. Bellman optimality equations describe the best achievable value. An optimal policy selects an action that maximizes Q*(s,a).
Key points
1. Bellman equations express recursive value relationships.
2. State value includes immediate reward and discounted next-state value.
3. Action value includes reward and expected future value.
4. Bellman optimality uses the maximum over available actions.
5. Q*(s,a) represents optimal action value.
6. Optimal policy: π*(s) = arg maxₐ Q*(s,a).
Example: Choosing the highest-value action
Suppose an agent has three actions:
Action A: Q*(s,A) = 8.
Action B: Q*(s,B) = 12.
Action C: Q*(s,C) = 5.
The optimal action is B because it has the highest optimal action value.
π*(s) = arg maxₐ Q*(s,a).
Real-world application
Bellman equations are foundational to dynamic programming and Q-learning.
Key takeaway
Bellman equations connect immediate rewards with future values and provide the basis for finding optimal policies.