OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Variants and When Each Fails
Lesson 234 of 415 5 min read
6.6.3

Reference Model Collapse

Understand the risks of a reference policy that is either too strong or too weak during preference optimization.

What is it?

Reference collapse occurs when the reference model constrains the policy too strongly or provides too little regularization. Either extreme can undermine preference optimization.

Key points

1. The reference model provides a regularization baseline. 2. An overly strong reference can prevent the policy from moving. 3. An overly weak reference provides insufficient regularization. 4. Reference strength affects the policy update.

Two Reference Model Extremes

Too strong: The policy remains too close to the reference and cannot make the changes needed to improve preferences. Too weak: The reference does not provide enough regularization, allowing the policy to move without sufficient constraint. Both extremes can make DPO training ineffective.

Real-world application

Check the reference checkpoint and the degree of policy deviation when diagnosing unsuccessful DPO training.
Key takeaway

The reference model must provide useful regularization without preventing necessary policy improvement.