OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Online DPO
Lesson 331 of 415 6 min read
8.1.2

Online DPO Algorithm: Generating Fresh Preferences

Learn how Online DPO generates responses, scores them, creates preference pairs and applies DPO loss.

What is it?

Online DPO repeatedly generates multiple responses from the current policy, scores them with a reward model, selects the highest-scoring and lowest-scoring responses as a preference pair, and applies the DPO loss. The process repeats with fresh generations.

Key points

Generate K responses for each prompt.|Score responses using reward model rϕ.|Select the highest-scoring response as chosen.|Select the lowest-scoring response as rejected.|Apply DPO loss and repeat with new generations.

The Online DPO training loop

For a prompt, the current policy generates four responses. The reward model scores all four. The highest-scoring response becomes the chosen example and the lowest-scoring response becomes the rejected example. Online DPO applies DPO loss to this fresh pair before generating new responses in the next step.

Real-world application

Use reward-model scoring to construct fresh chosen and rejected pairs during training.
Key takeaway

Online DPO combines current-policy generation with pair-based DPO optimisation.