6.1.2
The Need for a Simpler Alignment Method
Understand why DPO explores a simpler alignment method using preference data.
What is it?
DPO asks whether the reinforcement-learning loop can be skipped while retaining the goal of aligning a model with human preferences.
Key points
1. PPO involves a complex RL loop.
2. DPO uses preference data directly.
3. It avoids a separate reward-model training stage.
4. It avoids PPO rollout optimization.
5. It derives a supervised objective from the RLHF solution.
Why Choose DPO?
PPO-based RLHF: preference data → reward model → RL optimization.
DPO: preference data → direct preference optimization.
DPO simplifies the alignment training process.
Real-world application
DPO is useful when preference data is available and a separate PPO training pipeline is undesirable.
Key takeaway
DPO turns preference alignment into a simpler supervised optimization problem.