OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / IPO - Identity Preference Optimization
Lesson 339 of 415 7 min read
8.3.2

IPO Loss Function and Target Margin

Learn how IPO uses squared loss to target a finite preference margin.

What is it?

IPO replaces the DPO log-sigmoid objective with a squared loss that targets a specific preference margin. The loss is minimised at a finite gap rather than encouraging the margin to grow indefinitely.

Key points

IPO uses a squared loss.|The target margin is 1 divided by 2β.|Margins that are too small are penalised.|Margins that are too large are also penalised.|The finite target provides regularisation.

Understanding the IPO target margin

The source defines the IPO loss as: L_IPO = E[(log(πθ(yw|x)/πref(yw|x)) − log(πθ(yl|x)/πref(yl|x)) − 1/(2β))²] The objective targets a finite margin of 1/(2β). Unlike DPO, it penalises deviations on both sides of this target.

Real-world application

Use IPO when you want preference optimisation to target a finite margin rather than continually increasing the chosen-rejected gap.
Key takeaway

IPO uses squared loss to target a finite preference margin.