11.1.1
Why PPO Requires Multiple Models
Understand why RLHF training coordinates policy, reference, reward and value components instead of a single model.
What is it?
PPO-based RLHF requires multiple model components to operate together. The policy model generates responses, the reference model provides a comparison policy, the reward model scores responses, and the value component supports PPO optimisation. Coordinating these components creates a much larger memory requirement than ordinary single-model SFT.
Key points
RLHF coordinates multiple model components.
Policy model: generates responses and is updated during training.
Reference model: provides the reference distribution for KL comparison.
Reward model: evaluates generated responses.
Value component: estimates values for PPO updates.
The components and training state must be coordinated during rollout, scoring and optimisation.
The Models in a PPO Training System
A simplified PPO training cycle:
1. The policy model generates a response.
2. The reward model scores the response.
3. The reference model provides probabilities for KL comparison.
4. The value component supports advantage estimation.
5. PPO updates the policy model.
These components create a multi-model memory and coordination challenge.
Real-world application
When estimating infrastructure for PPO-based RLHF, account for every model component and the training state rather than budgeting only for the policy model.
Key takeaway
PPO-based RLHF is a coordinated multi-model training system, not simply a single model update.