6.9.1
f-DPO – Generalised f-Divergence DPO
Understand how f-DPO generalises standard DPO using different f-divergences, including reverse KL, forward KL, Jensen–Shannon and alpha divergence.
What is it?
Standard DPO uses reverse KL divergence as its regulariser. Reverse KL is mode-seeking, concentrating probability on a few high-reward responses. Forward KL is mode-covering, spreading probability across plausible responses. f-DPO generalises DPO by allowing different f-divergences to control this trade-off.
Key points
f-DPO replaces the standard log-ratio with the derivative of an f-divergence generator.
reverse_kl: f′(u) = log u; standard DPO; mode-seeking.
forward_kl: f′(u) = −1/u; mode-covering; supports diversity.
js_divergence: f′(u) = log(2u/(u + 1)); balances mode-seeking and mode-covering.
alpha_divergence: f′(u) = u^(α−1); interpolates between forward and reverse KL.
Use JS for a balance, forward KL for creative diversity, reverse KL for tasks with a single correct answer, and alpha divergence to tune the trade-off.
Choosing an f-Divergence
Consider a creative writing model that should produce varied responses. Forward KL is a suitable choice because it is mode-covering. For a task with one correct answer, such as a constrained mathematical response, reverse KL may be more appropriate. JS divergence offers a balance between diversity and quality.
Real-world application
In TRL, select the divergence through DPOConfig using f_divergence_type. The source demonstrates js_divergence and alpha_divergence, with f_alpha_divergence_coef controlling the alpha parameter.
Key takeaway
f-DPO lets practitioners choose how strongly training favours concentrated high-reward responses versus broader response diversity.