8.1.4
Online DPO vs Offline DPO vs PPO
Compare data freshness, model requirements, training objectives and suitable use cases.
What is it?
Offline DPO, Online DPO and PPO differ in their training data, model requirements and optimisation objectives. Offline DPO uses static pairs, Online DPO generates fresh pairs from the current policy, and PPO uses fresh policy data with a clipped policy objective and a value head.
Key points
Offline DPO: static pairs, two models, DPO loss.|Online DPO: fresh policy-generated pairs, three models, DPO loss.|PPO: fresh policy data, four models, PPO clipped objective.|Online DPO occupies a middle ground in complexity.|The source positions it as useful when DPO plateaus and exploration is needed.
Comparing three preference optimisation methods
Offline DPO is suitable for quick alignment when static preference data and limited compute are available. Online DPO is useful when the model needs fresh exploration after Offline DPO plateaus. PPO is presented as an option for maximum quality and complex reasoning, with greater model and training complexity.
Real-world application
Choose the method based on data freshness, available compute and the complexity of the training objective.
Key takeaway
Online DPO combines fresh on-policy preference data with the simpler pair-based DPO loss.