8.1.1
Motivation: The Distribution Shift Problem
Understand why static preference pairs can become outdated as the policy changes during training.
What is it?
Standard DPO trains on preference pairs generated by an earlier or different model. As the policy changes during training, its outputs may become different from those static training examples. This creates distribution shift and can make the training objective less relevant. Online DPO addresses this by generating fresh preference pairs from the current policy at every step.
Key points
Offline DPO uses static preference pairs.|The data may come from an older checkpoint or another model family.|The current policy changes during training.|Its outputs may diverge from the original preference data.|Online DPO generates fresh pairs using the current policy.
How preference data becomes outdated
Imagine a model initially generates short answers. After several training updates, it starts producing longer, more detailed responses. If training continues using preference pairs generated by the earlier model, those pairs may no longer represent the current policy distribution.
Real-world application
Consider Online DPO when static preference data becomes less representative of the policy being trained.
Key takeaway
Online DPO reduces distribution shift by generating fresh preference pairs from the current policy.