OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Online DPO
Lesson 330 of 415 5 min read
8.1.1

Motivation: The Distribution Shift Problem

Understand why static preference pairs can become outdated as the policy changes during training.

What is it?

Standard DPO trains on preference pairs generated by an earlier or different model. As the policy changes during training, its outputs may become different from those static training examples. This creates distribution shift and can make the training objective less relevant. Online DPO addresses this by generating fresh preference pairs from the current policy at every step.

Key points

Offline DPO uses static preference pairs.|The data may come from an older checkpoint or another model family.|The current policy changes during training.|Its outputs may diverge from the original preference data.|Online DPO generates fresh pairs using the current policy.

How preference data becomes outdated

Imagine a model initially generates short answers. After several training updates, it starts producing longer, more detailed responses. If training continues using preference pairs generated by the earlier model, those pairs may no longer represent the current policy distribution.

Real-world application

Consider Online DPO when static preference data becomes less representative of the policy being trained.
Key takeaway

Online DPO reduces distribution shift by generating fresh preference pairs from the current policy.