3.6.2
The REINFORCE Algorithm
Understand trajectory sampling, return calculation and policy parameter updates.
What is it?
REINFORCE is a Monte Carlo policy-gradient algorithm introduced by Williams (1992). It samples complete trajectories using the current policy, calculates the return at each time step and updates policy parameters using the policy gradient estimate.
Key points
1. Sample a complete trajectory τ under πθ.
2. Calculate return Gₜ for each time step.
3. Compute the policy gradient contribution ∇θ log πθ(aₜ|sₜ) · Gₜ.
4. Update parameters: θ ← θ + α Σₜ ∇θ log πθ(aₜ|sₜ) · Gₜ.
5. α represents the learning rate.
6. REINFORCE uses complete episode returns.
7. It does not bootstrap from estimated future values.
8. The gradient does not require differentiating through environment dynamics.
Example: Training an agent to complete a game
An agent plays a complete game using its current policy.
Step 1: It records the states, actions and rewards.
Step 2: After the game ends, it calculates the return associated with each action.
Step 3: Actions from high-return trajectories receive positive updates.
Step 4: Policy parameters are adjusted to increase the probability of actions associated with better outcomes.
The process repeats over new trajectories.
Real-world application
REINFORCE provides a foundational method for training stochastic policies using sampled episodes and observed rewards.
Key takeaway
REINFORCE learns by sampling complete trajectories, calculating returns and updating policy parameters.