8.3.2
IPO Loss Function and Target Margin
Learn how IPO uses squared loss to target a finite preference margin.
What is it?
IPO replaces the DPO log-sigmoid objective with a squared loss that targets a specific preference margin. The loss is minimised at a finite gap rather than encouraging the margin to grow indefinitely.
Key points
IPO uses a squared loss.|The target margin is 1 divided by 2β.|Margins that are too small are penalised.|Margins that are too large are also penalised.|The finite target provides regularisation.
Understanding the IPO target margin
The source defines the IPO loss as:
L_IPO = E[(log(πθ(yw|x)/πref(yw|x)) − log(πθ(yl|x)/πref(yl|x)) − 1/(2β))²]
The objective targets a finite margin of 1/(2β). Unlike DPO, it penalises deviations on both sides of this target.
Real-world application
Use IPO when you want preference optimisation to target a finite margin rather than continually increasing the chosen-rejected gap.
Key takeaway
IPO uses squared loss to target a finite preference margin.