3.6.1
Policy Gradient Objective and Theorem
Understand expected-return optimization and the policy gradient theorem.
What is it?
Policy Gradient methods directly optimize the parameters θ of a policy πθ to maximize expected return. Instead of learning a value function and deriving a policy, they improve the probability distribution over actions.
Key points
1. Objective: J(θ) = Eτ~πθ[R(τ)].
2. The goal is to maximize expected cumulative reward.
3. Policy Gradient Theorem: ∇θJ(θ) = Eπθ[Σₜ ∇θ log πθ(aₜ|sₜ) · Gₜ].
4. θ represents policy parameters.
5. πθ(aₜ|sₜ) is the probability of selecting action aₜ in state sₜ.
6. Gₜ is the future return from time step t.
7. The gradient indicates how policy parameters should change to improve expected return.
Example: Improving a recommendation policy
An AI recommendation agent suggests products to customers.
State: Customer browsing history.
Action: Recommend a product.
Reward: Customer engagement or purchase.
The policy gradient method adjusts the policy parameters so actions associated with higher future returns become more likely.
The policy is optimized directly rather than selecting actions through a learned Q-value table.
Real-world application
Policy gradients are useful when the action space is large or continuous. They are particularly relevant to language models, where actions correspond to selecting tokens from a large vocabulary.
Key takeaway
Policy Gradient methods directly optimize policy parameters to maximize expected return.