3.9.5
PPO Data Reuse and Controlled Off-Policy Updates
Understand how PPO uses clipping to perform multiple gradient steps on collected data.
What is it?
PPO uses the probability ratio between the updated policy and the policy that generated the data. Its clipped objective allows multiple gradient steps on one batch of on-policy data while limiting excessive policy changes. This makes data reuse controlled rather than unrestricted.
Key points
1. Probability ratio: r = πnew(a|s) / πold(a|s).
2. πnew is the updated policy.
3. πold is the policy that generated the training data.
4. PPO uses a clipped probability ratio.
5. The source describes four gradient-training epochs on one batch.
6. Clipping limits excessive policy changes.
7. PPO therefore makes controlled reuse of collected on-policy data.
8. It is described as slightly off-policy in a controlled way.
Example: Reusing one batch of generated responses
A language model generates a batch of responses using its current policy.
The training system calculates the old action probabilities and stores them with the batch.
PPO then performs multiple gradient-training epochs using the same batch.
The probability ratio compares the updated policy with the policy that generated the responses. Clipping limits how far the update can move from the original probabilities.
After the training cycle, fresh responses are generated for the next rollout.
Real-world application
PPO clipping helps reuse a rollout batch for several optimization epochs without allowing unrestricted policy changes.
Key takeaway
PPO uses probability-ratio clipping to make limited, controlled reuse of on-policy training data.