OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Q-Learning
Lesson 118 of 415 8 min read
3.5.1

Understanding Q-Learning

Learn the foundational off-policy value-based algorithm, its update rule, SARSA comparison, DQN and limitations for LLMs.

What is it?

Q-Learning is a foundational off-policy, value-based reinforcement learning algorithm. It learns the optimal action-value function Q* directly, regardless of the policy currently followed by the agent. Its update uses the immediate reward and the maximum estimated value of the next state.

Key points

1. Q-Learning is off-policy and value-based. 2. It learns the optimal action-value function Q*. 3. Update: Q(sₜ,aₜ) ← Q(sₜ,aₜ) + α[rₜ + γ maxₐ′ Q(sₜ₊₁,a′) − Q(sₜ,aₜ)]. 4. α is the learning rate and γ is the discount factor. 5. The maximum next-state Q-value forms the learning target. 6. ε-greedy exploration allows random actions while learning. 7. SARSA is an on-policy alternative that uses the action actually taken. 8. DQN replaces tabular Q-values with a neural network. 9. DQN uses experience replay and a target network for stability.

Example: Learning the best warehouse route

A robot is at a warehouse junction and can move left or right. Current Q-value for moving left = 5. Immediate reward = 2. Discount factor γ = 0.9. Maximum Q-value in the next state = 8. Learning rate α = 0.5. Target = 2 + (0.9 × 8) = 9.2. Updated Q-value = 5 + 0.5 × (9.2 − 5) = 7.1. The robot increases the estimated value of moving left because the observed outcome was better than its previous prediction.

Real-world application

Q-Learning is useful in discrete-action environments such as games, navigation and control tasks. DQN extends it to larger state spaces using neural networks.
Key takeaway

Q-Learning learns optimal action values using the maximum estimated value of the next state, independently of the action selected by the behavior policy.