8.3.4
When to Choose IPO over DPO
Identify when noisy preference labels, overfitting or multiple training epochs make IPO a useful alternative.
What is it?
IPO is useful when preference data contains noisy labels, DPO shows signs of overfitting, or training needs multiple epochs. The source describes IPO as more conservative and robust in these situations.
Key points
Use IPO with noisy crowdsourced preference data.|It can help when AI-judged labels contain errors.|Consider it when training loss approaches zero but evaluation degrades.|It supports more stable multiple-epoch training according to the source.|DPO may degrade after the first epoch.
Choosing IPO over DPO
Imagine a model trained on crowdsourced preference pairs. Training loss keeps decreasing, but evaluation quality starts falling. IPO’s finite target margin can provide a more conservative objective and reduce the tendency to memorise noisy preference pairs.
Real-world application
Consider IPO when DPO overfits or when preference labels are noisy and multiple training epochs are required.
Key takeaway
Choose IPO for a more conservative preference objective when DPO overfitting or noisy labels are concerns.