3.9.1
What Is On-Policy Learning?
Understand learning from data generated by the current policy.
What is it?
On-policy learning uses data generated by the current policy πθ. When the policy is updated, the old data is considered invalid for the next update cycle, so fresh experience must be generated.
Key points
1. Data source: Current policy πθ only.
2. The agent learns from its current behavior.
3. After an update, old data must be regenerated.
4. Sample efficiency is relatively low because data is used once.
5. Consistent data distribution generally provides greater stability.
6. Examples: REINFORCE, PPO, A2C and GRPO.
Example: Training a customer-support assistant
An assistant generates responses using its current policy.
Step 1: Generate responses to customer questions.
Step 2: Evaluate the responses and calculate rewards or advantages.
Step 3: Update the policy.
Step 4: Generate fresh responses using the updated policy.
The next training cycle uses the new responses rather than reusing the previous batch as fresh on-policy data.
Real-world application
PPO and GRPO use fresh policy-generated responses during LLM reinforcement learning. This makes response generation a major part of training compute.
Key takeaway
On-policy learning trains using experience generated by the current policy and requires fresh data after updates.