6.9.4
EXO – Exact Optimisation
Explore how EXO addresses the KL direction problem described in standard DPO using reverse KL probability matching.
What is it?
EXO addresses what the source describes as a KL direction problem in DPO. It argues that standard DPO derives from a reverse KL constraint but optimises a forward KL objective in reward space. EXO instead uses reverse KL probability matching between the model and the reward-optimal target distribution.
Key points
EXO minimises reverse KL between the model distribution and the reward-optimal target distribution.
p*(y|q) ∝ π_ref(y|q) exp(r(y,q)/β).
EXO approximates the objective using available preference pairs.
The approximate loss swaps the roles of π_θ and π_ref compared with standard DPO.
TRL example: loss_type = "exo_pair", beta = 0.1.
Understanding the EXO Objective
Standard DPO compares policy and reference log-probabilities in its preference loss. EXO uses the reverse arrangement in its pairwise approximation: reference log-probabilities are compared relative to policy log-probabilities. This reflects its different probability-matching objective.
Real-world application
In TRL, the source demonstrates configuring DPOConfig with loss_type = "exo_pair" and beta = 0.1, then passing the configuration to DPOTrainer.
Key takeaway
EXO changes the probability-matching objective and swaps the policy and reference roles in its pairwise loss approximation.