OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 246 of 415 8 min read
6.9.1

f-DPO – Generalised f-Divergence DPO

Understand how f-DPO generalises standard DPO using different f-divergences, including reverse KL, forward KL, Jensen–Shannon and alpha divergence.

What is it?

Standard DPO uses reverse KL divergence as its regulariser. Reverse KL is mode-seeking, concentrating probability on a few high-reward responses. Forward KL is mode-covering, spreading probability across plausible responses. f-DPO generalises DPO by allowing different f-divergences to control this trade-off.

Key points

f-DPO replaces the standard log-ratio with the derivative of an f-divergence generator. reverse_kl: f′(u) = log u; standard DPO; mode-seeking. forward_kl: f′(u) = −1/u; mode-covering; supports diversity. js_divergence: f′(u) = log(2u/(u + 1)); balances mode-seeking and mode-covering. alpha_divergence: f′(u) = u^(α−1); interpolates between forward and reverse KL. Use JS for a balance, forward KL for creative diversity, reverse KL for tasks with a single correct answer, and alpha divergence to tune the trade-off.

Choosing an f-Divergence

Consider a creative writing model that should produce varied responses. Forward KL is a suitable choice because it is mode-covering. For a task with one correct answer, such as a constrained mathematical response, reverse KL may be more appropriate. JS divergence offers a balance between diversity and quality.

Real-world application

In TRL, select the divergence through DPOConfig using f_divergence_type. The source demonstrates js_divergence and alpha_divergence, with f_alpha_divergence_coef controlling the alpha parameter.
Key takeaway

f-DPO lets practitioners choose how strongly training favours concentrated high-reward responses versus broader response diversity.