OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Core Concepts and Definitions
Lesson 109 of 415 7 min read
3.2.6

Bellman Equations and Optimal Policy

Understand the recursive Bellman relationships for value functions and how Bellman optimality identifies the best policy.

What is it?

Bellman equations express value functions recursively by relating current value to immediate reward and expected future value. Bellman optimality equations describe the best achievable value. An optimal policy selects an action that maximizes Q*(s,a).

Key points

1. Bellman equations express recursive value relationships. 2. State value includes immediate reward and discounted next-state value. 3. Action value includes reward and expected future value. 4. Bellman optimality uses the maximum over available actions. 5. Q*(s,a) represents optimal action value. 6. Optimal policy: π*(s) = arg maxₐ Q*(s,a).

Example: Choosing the highest-value action

Suppose an agent has three actions: Action A: Q*(s,A) = 8. Action B: Q*(s,B) = 12. Action C: Q*(s,C) = 5. The optimal action is B because it has the highest optimal action value. π*(s) = arg maxₐ Q*(s,a).

Real-world application

Bellman equations are foundational to dynamic programming and Q-learning.
Key takeaway

Bellman equations connect immediate rewards with future values and provide the basis for finding optimal policies.