6.1.1
Why PPO-Based RLHF Is Difficult
Understand the memory requirements, infrastructure complexity and training instability associated with PPO-based RLHF.
What is it?
PPO-based RLHF requires four models in memory: policy, reference, reward model and value head. It also needs complex reinforcement-learning infrastructure.
Key points
1. Policy generates responses.
2. Reference model provides the KL baseline.
3. Reward model scores responses.
4. Value head estimates values.
5. PPO requires additional rollout and optimization infrastructure.
The Four Models in PPO
A PPO-based RLHF system uses four components:
Policy model: generates and improves responses.
Reference model: provides a reference distribution.
Reward model: evaluates response quality.
Value head: estimates expected returns.
Real-world application
Consider memory requirements and training infrastructure before selecting PPO for large language-model alignment.
Key takeaway
PPO-based RLHF is powerful but requires substantial model memory and infrastructure.