6.6.2
Lack of Exploration
Understand why DPO cannot discover behaviours that are absent from its preference dataset and may become stuck in a local optimum.
What is it?
DPO learns from the preference pairs provided in its dataset. It cannot discover behaviours that are absent from those examples, which can leave the policy stuck in a local optimum.
Key points
1. DPO depends on the available preference dataset.
2. It does not independently explore missing behaviours.
3. Unrepresented responses receive no direct preference supervision.
4. The policy can remain in a local optimum.
A Behaviour Missing from the Dataset
Suppose a preference dataset contains only short customer-support answers.
The model learns to favour that response style. If the dataset contains no examples of detailed troubleshooting, DPO has no direct preference signal teaching that behaviour.
The model may remain limited to the response patterns represented in its data.
Real-world application
Include preference examples that represent the behaviours and response styles the model should learn.
Key takeaway
DPO cannot learn preference distinctions that its dataset does not represent.