8.1.2
Online DPO Algorithm: Generating Fresh Preferences
Learn how Online DPO generates responses, scores them, creates preference pairs and applies DPO loss.
What is it?
Online DPO repeatedly generates multiple responses from the current policy, scores them with a reward model, selects the highest-scoring and lowest-scoring responses as a preference pair, and applies the DPO loss. The process repeats with fresh generations.
Key points
Generate K responses for each prompt.|Score responses using reward model rϕ.|Select the highest-scoring response as chosen.|Select the lowest-scoring response as rejected.|Apply DPO loss and repeat with new generations.
The Online DPO training loop
For a prompt, the current policy generates four responses. The reward model scores all four. The highest-scoring response becomes the chosen example and the lowest-scoring response becomes the rejected example. Online DPO applies DPO loss to this fresh pair before generating new responses in the next step.
Real-world application
Use reward-model scoring to construct fresh chosen and rejected pairs during training.
Key takeaway
Online DPO combines current-policy generation with pair-based DPO optimisation.