OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / IPO - Identity Preference Optimization
Lesson 338 of 415 5 min read
8.3.1

Why DPO Can Overfit Preference Data

Understand how DPO can drive the chosen-rejected margin toward infinity and memorise preference examples.

What is it?

DPO can reach zero loss by increasing the margin between chosen and rejected responses indefinitely. This can push the chosen response probability toward 1 and the rejected response probability toward 0, causing the model to memorise training preferences and overfit.

Key points

DPO uses a log-sigmoid objective.|The margin can grow without a natural finite stopping point.|Chosen response probability can be pushed toward 1.|Rejected response probability can be pushed toward 0.|This can cause preference-data overfitting.

How DPO can overfit

Suppose a training pair contains a preferred answer and a rejected answer. DPO can continue increasing the probability gap between them to reduce its loss. If this process drives the model to memorise the training pairs, evaluation performance may degrade.

Real-world application

Watch for training loss approaching zero while evaluation performance becomes worse.
Key takeaway

IPO addresses the risk of an unbounded preference margin in DPO.