4.4.4
Preference Optimization Variants
Explore preference optimization methods and their applications.
What is it?
Preference optimization variants extend the basic preference-learning toolkit with different data-generation, optimization and decision-calibration strategies. The chapter covers Online DPO, KTO, Best-of-N, method selection, RLCR, RLCD and CLM.
Key points
1. Online DPO uses fresh policy-generated data for preference optimization.
2. KTO is a preference optimization method based on prospect-theoretic utility.
3. Best-of-N generates multiple candidates and selects the best according to a scoring method.
4. Method selection depends on the training objective and available data.
5. RLCR adds a Brier-score calibration reward to RLVR so agents express reliable confidence.
6. RLCD covers TypeSafe’s Jev model, optimizing typed probability distributions over outcomes.
7. CLM means Contrastive Language Modeling for agent decisions.
Example: Choosing a preference optimization method
Consider three training requirements.
Requirement A: A team has a fixed preference dataset and wants direct preference optimization.
Requirement B: A team wants to generate fresh responses from the current policy and optimize preferences online.
Requirement C: An agent must provide reliable confidence alongside correct answers.
Different methods from this chapter address these different requirements. The choice depends on the objective and available feedback.
Real-world application
These variants help researchers select training approaches for offline preferences, online preferences, candidate selection and calibrated agent decisions.
Key takeaway
Preference optimization variants extend basic DPO and RL methods to different data, optimization and confidence-calibration requirements.