3.2.4
The Action-Value Function
Understand how Q evaluates the expected return from taking a particular action in a state and then following a policy.
What is it?
The action-value function estimates expected return when an agent starts in state s, first takes action a and then follows policy π. It is represented by Qᵖⁱ(s,a) = Eᵖⁱ[Gₜ | sₜ = s, aₜ = a].
Key points
1. Qᵖⁱ(s,a) evaluates an action in a state.
2. It estimates expected cumulative discounted reward.
3. The first action is specified explicitly.
4. Subsequent decisions follow policy π.
5. Different actions can have different Q-values.
6. Q-values help compare available actions.
Example: Selecting a warehouse route
A robot has two choices.
Action A: Take a short route through a busy aisle.
Action B: Take a longer route through a clear aisle.
The action-value function estimates the expected return for each choice.
Real-world application
Action-value functions support Q-learning, action selection and reinforcement learning algorithms.
Key takeaway
The action-value function estimates expected return from taking a specific action in a state.