4.3.3
Stage 3: RL Optimization with PPO or GRPO
Understand how PPO or GRPO optimizes the policy against reward signals with a KL constraint.
What is it?
RL Optimization is the third stage. The reward model provides a training signal to optimize the language-model policy using PPO or GRPO. A KL constraint against the SFT policy helps prevent the updated model from drifting too far from its instruction-following starting point.
Key points
1. Use the reward model to score generated responses.
2. Optimize the language-model policy to increase reward.
3. PPO and GRPO are the optimization methods identified by the source.
4. Apply a KL constraint against πSFT.
5. The KL constraint limits policy drift.
6. The objective is to improve response quality while retaining the useful behaviour learned during SFT.
Example: Improving a trained assistant
An SFT model generates a response to a customer query.
The reward model scores the response based on learned human preferences.
PPO or GRPO uses this reward signal to update the model policy.
The KL constraint keeps the updated policy from moving too far away from the original SFT policy.
The process is repeated to improve the model.
Real-world application
This stage is the core reinforcement-learning optimization step in a classic RLHF training pipeline.
Key takeaway
PPO or GRPO optimizes the policy against reward feedback while using a KL constraint against πSFT.