OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / RL Foundations for Language Models / Roadmap of This Part
Lesson 158 of 415 6 min read
4.4.4

Preference Optimization Variants

Explore preference optimization methods and their applications.

What is it?

Preference optimization variants extend the basic preference-learning toolkit with different data-generation, optimization and decision-calibration strategies. The chapter covers Online DPO, KTO, Best-of-N, method selection, RLCR, RLCD and CLM.

Key points

1. Online DPO uses fresh policy-generated data for preference optimization. 2. KTO is a preference optimization method based on prospect-theoretic utility. 3. Best-of-N generates multiple candidates and selects the best according to a scoring method. 4. Method selection depends on the training objective and available data. 5. RLCR adds a Brier-score calibration reward to RLVR so agents express reliable confidence. 6. RLCD covers TypeSafe’s Jev model, optimizing typed probability distributions over outcomes. 7. CLM means Contrastive Language Modeling for agent decisions.

Example: Choosing a preference optimization method

Consider three training requirements. Requirement A: A team has a fixed preference dataset and wants direct preference optimization. Requirement B: A team wants to generate fresh responses from the current policy and optimize preferences online. Requirement C: An agent must provide reliable confidence alongside correct answers. Different methods from this chapter address these different requirements. The choice depends on the objective and available feedback.

Real-world application

These variants help researchers select training approaches for offline preferences, online preferences, candidate selection and calibrated agent decisions.
Key takeaway

Preference optimization variants extend basic DPO and RL methods to different data, optimization and confidence-calibration requirements.