5.7.2
Phase 2: Optimization Loop (Mini-Batch Updates)
Understand how stored rollout data is used for repeated mini-batch optimization.
What is it?
After the rollout buffer is full, PPO performs multiple optimization epochs over mini-batches. It evaluates the frozen old policy and the live updating policy on stored states. The old-policy log-probability can be reused from the rollout buffer, while the live policy is evaluated again as its weights change.
Key points
1. PPO runs K epochs, typically 3–10.
2. Training operates on mini-batches from the rollout buffer.
3. The old policy remains frozen.
4. The stored old-policy log-probability can be reused.
5. The live policy produces new logits and log-probabilities.
6. Live policy weights change after every mini-batch gradient step.
7. Old-policy outputs remain static throughout the optimization cycle.
Example: Comparing Two Policy Evaluations
A stored state is evaluated during PPO training.
Old policy:
The old-policy log-probability is retrieved from the rollout buffer.
Live policy:
The current network performs a forward pass and calculates a new log-probability.
The two values are compared to calculate the PPO probability ratio.
Real-world application
Reusing stored old-policy log-probabilities avoids repeating a forward pass through the frozen network for every mini-batch, which is particularly useful for large models.
Key takeaway
During optimization, the old policy stays frozen while the live policy changes.