6.9.3
TR-DPO – Trust Region DPO
Understand how TR-DPO periodically updates the reference model using an exponential moving average to address reference-model staleness.
What is it?
Standard DPO keeps the reference model fixed throughout training. As the policy improves, the KL penalty can grow and eventually dominate the loss, preventing further improvement. TR-DPO periodically updates the reference model using an exponential moving average of the current policy and the previous reference.
Key points
EMA update: π_ref^(t+1) ← α · π_θ^(t) + (1 − α) · π_ref^(t).
α is the mixup coefficient, where 0 < α < 1.
The update is applied every T_sync gradient steps.
Use cases include long training runs, early loss plateaus caused by KL penalty domination, and iterative pipelines using new preference data from the current policy.
α close to 1: faster reference updates.
α close to 0: slower reference updates.
TRL settings shown: sync_ref_model = True, ref_model_mixup_alpha = 0.6, ref_model_sync_steps = 512.
Updating a Stale Reference Model
Imagine a long DPO training run in which the policy has moved considerably from its initial reference model. The KL penalty may increasingly restrict further policy improvement. TR-DPO periodically blends the current policy into the reference model so that the reference tracks policy changes over time.
Real-world application
In TRL, the source demonstrates enabling sync_ref_model and configuring ref_model_mixup_alpha and ref_model_sync_steps. Monitor whether the loss plateaus early due to KL penalty domination.
Key takeaway
TR-DPO addresses reference-model staleness by periodically updating the reference using an exponential moving average.