6.6.1
Distribution Shift in Preference Data
Understand why preference data generated by an older model can become irrelevant when the current policy generates different responses.
What is it?
Distribution shift occurs when preference data comes from an older model but the current policy generates different responses. The loss may then optimize against examples that no longer represent current model behaviour.
Key points
1. Preference data may come from an older policy.
2. The current policy may generate different text.
3. Training examples may become less relevant.
4. The loss can optimize on outdated preference examples.
Outdated Preference Data
An older model generates responses about a customer query. Human evaluators rank those responses and create preference pairs.
After substantial model updates, the current policy generates very different responses. The old preference pairs may no longer represent the current policy distribution.
DPO can continue optimizing on examples that are less relevant to the current model.
Real-world application
Review the source and freshness of preference datasets when the policy model changes significantly.
Key takeaway
Preference data should remain relevant to the responses generated by the current policy.