OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Summary: Choosing an Alignment Method
Lesson 350 of 415 6 min read
8.7.1

Comparing Alignment Methods: Models, Data and Compute

Compare PPO, GRPO, DPO, Online DPO, KTO, IPO, ORPO, Best-of-N, RLCR, RLCD and CLM by model count, training data, compute and stability.

What is it?

Alignment methods differ in model requirements, training data, compute cost and stability. Comparing these dimensions helps identify a method that fits the task and available resources.

Key points

PPO: 4 models, online generation, very high compute, low stability. GRPO: 2 models without a critic, online generation, high compute, medium stability. DPO: 2 models, offline pairs, low compute, high stability. Online DPO: 3 models, online generation, medium compute, medium-high stability. KTO: 2 models, unpaired binary feedback, low compute, high stability. IPO: 2 models, offline pairs, low compute, very high stability. ORPO: 1 model, offline pairs, very low compute, high stability. Best-of-N: 1 model plus reward model, online generation, medium compute, perfect stability. RLCR: 1 model, online generation, medium compute, medium stability. RLCD: 1 model, outcome-matched data, medium compute, high stability. CLM: dual encoder, contrastive pairs, medium compute, high stability.

Comparing Alignment Methods

A team has limited compute and paired preference data. DPO is a practical starting point because it uses offline preference pairs, requires two models and has relatively low compute requirements. A team training on verifiable mathematical rewards may instead select GRPO.

Real-world application

Compare the number of models, data format, compute requirement and stability before selecting an alignment method.
Key takeaway

There is no single best alignment method for every task. Choose according to data, compute, stability and the required outcome.