8.3.1
Why DPO Can Overfit Preference Data
Understand how DPO can drive the chosen-rejected margin toward infinity and memorise preference examples.
What is it?
DPO can reach zero loss by increasing the margin between chosen and rejected responses indefinitely. This can push the chosen response probability toward 1 and the rejected response probability toward 0, causing the model to memorise training preferences and overfit.
Key points
DPO uses a log-sigmoid objective.|The margin can grow without a natural finite stopping point.|Chosen response probability can be pushed toward 1.|Rejected response probability can be pushed toward 0.|This can cause preference-data overfitting.
How DPO can overfit
Suppose a training pair contains a preferred answer and a rejected answer. DPO can continue increasing the probability gap between them to reduce its loss. If this process drives the model to memorise the training pairs, evaluation performance may degrade.
Real-world application
Watch for training loss approaching zero while evaluation performance becomes worse.
Key takeaway
IPO addresses the risk of an unbounded preference margin in DPO.