8.7.1
Comparing Alignment Methods: Models, Data and Compute
Compare PPO, GRPO, DPO, Online DPO, KTO, IPO, ORPO, Best-of-N, RLCR, RLCD and CLM by model count, training data, compute and stability.
What is it?
Alignment methods differ in model requirements, training data, compute cost and stability. Comparing these dimensions helps identify a method that fits the task and available resources.
Key points
PPO: 4 models, online generation, very high compute, low stability. GRPO: 2 models without a critic, online generation, high compute, medium stability. DPO: 2 models, offline pairs, low compute, high stability. Online DPO: 3 models, online generation, medium compute, medium-high stability. KTO: 2 models, unpaired binary feedback, low compute, high stability. IPO: 2 models, offline pairs, low compute, very high stability. ORPO: 1 model, offline pairs, very low compute, high stability. Best-of-N: 1 model plus reward model, online generation, medium compute, perfect stability. RLCR: 1 model, online generation, medium compute, medium stability. RLCD: 1 model, outcome-matched data, medium compute, high stability. CLM: dual encoder, contrastive pairs, medium compute, high stability.
Comparing Alignment Methods
A team has limited compute and paired preference data. DPO is a practical starting point because it uses offline preference pairs, requires two models and has relatively low compute requirements. A team training on verifiable mathematical rewards may instead select GRPO.
Real-world application
Compare the number of models, data format, compute requirement and stability before selecting an alignment method.
Key takeaway
There is no single best alignment method for every task. Choose according to data, compute, stability and the required outcome.