OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Core Concepts and Definitions
Lesson 107 of 415 5 min read
3.2.4

The Action-Value Function

Understand how Q evaluates the expected return from taking a particular action in a state and then following a policy.

What is it?

The action-value function estimates expected return when an agent starts in state s, first takes action a and then follows policy π. It is represented by Qᵖⁱ(s,a) = Eᵖⁱ[Gₜ | sₜ = s, aₜ = a].

Key points

1. Qᵖⁱ(s,a) evaluates an action in a state. 2. It estimates expected cumulative discounted reward. 3. The first action is specified explicitly. 4. Subsequent decisions follow policy π. 5. Different actions can have different Q-values. 6. Q-values help compare available actions.

Example: Selecting a warehouse route

A robot has two choices. Action A: Take a short route through a busy aisle. Action B: Take a longer route through a clear aisle. The action-value function estimates the expected return for each choice.

Real-world application

Action-value functions support Q-learning, action selection and reinforcement learning algorithms.
Key takeaway

The action-value function estimates expected return from taking a specific action in a state.