6.6.3
Reference Model Collapse
Understand the risks of a reference policy that is either too strong or too weak during preference optimization.
What is it?
Reference collapse occurs when the reference model constrains the policy too strongly or provides too little regularization. Either extreme can undermine preference optimization.
Key points
1. The reference model provides a regularization baseline.
2. An overly strong reference can prevent the policy from moving.
3. An overly weak reference provides insufficient regularization.
4. Reference strength affects the policy update.
Two Reference Model Extremes
Too strong: The policy remains too close to the reference and cannot make the changes needed to improve preferences.
Too weak: The reference does not provide enough regularization, allowing the policy to move without sufficient constraint.
Both extremes can make DPO training ineffective.
Real-world application
Check the reference checkpoint and the degree of policy deviation when diagnosing unsuccessful DPO training.
Key takeaway
The reference model must provide useful regularization without preventing necessary policy improvement.