OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 248 of 415 8 min read
6.9.3

TR-DPO – Trust Region DPO

Understand how TR-DPO periodically updates the reference model using an exponential moving average to address reference-model staleness.

What is it?

Standard DPO keeps the reference model fixed throughout training. As the policy improves, the KL penalty can grow and eventually dominate the loss, preventing further improvement. TR-DPO periodically updates the reference model using an exponential moving average of the current policy and the previous reference.

Key points

EMA update: π_ref^(t+1) ← α · π_θ^(t) + (1 − α) · π_ref^(t). α is the mixup coefficient, where 0 < α < 1. The update is applied every T_sync gradient steps. Use cases include long training runs, early loss plateaus caused by KL penalty domination, and iterative pipelines using new preference data from the current policy. α close to 1: faster reference updates. α close to 0: slower reference updates. TRL settings shown: sync_ref_model = True, ref_model_mixup_alpha = 0.6, ref_model_sync_steps = 512.

Updating a Stale Reference Model

Imagine a long DPO training run in which the policy has moved considerably from its initial reference model. The KL penalty may increasingly restrict further policy improvement. TR-DPO periodically blends the current policy into the reference model so that the reference tracks policy changes over time.

Real-world application

In TRL, the source demonstrates enabling sync_ref_model and configuring ref_model_mixup_alpha and ref_model_sync_steps. Monitor whether the loss plateaus early due to KL penalty domination.
Key takeaway

TR-DPO addresses reference-model staleness by periodically updating the reference using an exponential moving average.