OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 249 of 415 7 min read
6.9.4

EXO – Exact Optimisation

Explore how EXO addresses the KL direction problem described in standard DPO using reverse KL probability matching.

What is it?

EXO addresses what the source describes as a KL direction problem in DPO. It argues that standard DPO derives from a reverse KL constraint but optimises a forward KL objective in reward space. EXO instead uses reverse KL probability matching between the model and the reward-optimal target distribution.

Key points

EXO minimises reverse KL between the model distribution and the reward-optimal target distribution. p*(y|q) ∝ π_ref(y|q) exp(r(y,q)/β). EXO approximates the objective using available preference pairs. The approximate loss swaps the roles of π_θ and π_ref compared with standard DPO. TRL example: loss_type = "exo_pair", beta = 0.1.

Understanding the EXO Objective

Standard DPO compares policy and reference log-probabilities in its preference loss. EXO uses the reverse arrangement in its pairwise approximation: reference log-probabilities are compared relative to policy log-probabilities. This reflects its different probability-matching objective.

Real-world application

In TRL, the source demonstrates configuring DPOConfig with loss_type = "exo_pair" and beta = 0.1, then passing the configuration to DPOTrainer.
Key takeaway

EXO changes the probability-matching objective and swaps the policy and reference roles in its pairwise loss approximation.