3.5.1
Understanding Q-Learning
Learn the foundational off-policy value-based algorithm, its update rule, SARSA comparison, DQN and limitations for LLMs.
What is it?
Q-Learning is a foundational off-policy, value-based reinforcement learning algorithm. It learns the optimal action-value function Q* directly, regardless of the policy currently followed by the agent. Its update uses the immediate reward and the maximum estimated value of the next state.
Key points
1. Q-Learning is off-policy and value-based.
2. It learns the optimal action-value function Q*.
3. Update: Q(sₜ,aₜ) ← Q(sₜ,aₜ) + α[rₜ + γ maxₐ′ Q(sₜ₊₁,a′) − Q(sₜ,aₜ)].
4. α is the learning rate and γ is the discount factor.
5. The maximum next-state Q-value forms the learning target.
6. ε-greedy exploration allows random actions while learning.
7. SARSA is an on-policy alternative that uses the action actually taken.
8. DQN replaces tabular Q-values with a neural network.
9. DQN uses experience replay and a target network for stability.
Example: Learning the best warehouse route
A robot is at a warehouse junction and can move left or right.
Current Q-value for moving left = 5.
Immediate reward = 2.
Discount factor γ = 0.9.
Maximum Q-value in the next state = 8.
Learning rate α = 0.5.
Target = 2 + (0.9 × 8) = 9.2.
Updated Q-value = 5 + 0.5 × (9.2 − 5) = 7.1.
The robot increases the estimated value of moving left because the observed outcome was better than its previous prediction.
Real-world application
Q-Learning is useful in discrete-action environments such as games, navigation and control tasks. DQN extends it to larger state spaces using neural networks.
Key takeaway
Q-Learning learns optimal action values using the maximum estimated value of the next state, independently of the action selected by the behavior policy.