OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Summary: Choosing an Alignment Method
Lesson 354 of 415 7 min read
8.7.5

The Alignment Method Decision Tree

Apply the source decision tree to choose an alignment method based on rewards, preferences, feedback, compute and application requirements.

What is it?

The chapter decision tree maps common training constraints to an alignment method. Start with the reward and feedback available, then consider quality, compute and the required output.

Key points

1. Verifiable rewards for math or code → GRPO. 2. Maximum quality on complex tasks → PPO. 3. Paired preferences → DPO, or IPO if noisy. 4. Unpaired binary feedback → KTO. 5. Memory-limited and starting from a base model → ORPO. 6. DPO plateauing and on-policy data needed → Online DPO. 7. Strong baseline quickly → Best-of-N or RFT. 8. Calibrated confidence with correctness → RLCR. 9. Typed, fast software decisions → RLCD or Jev. 10. Cached state/action embeddings for agent decisions → CLM.

Alignment Method Decision Tree

A team has paired preference data, but the labels are noisy. Follow the decision tree: paired preferences point to DPO, while noisy labels point to IPO. If the team later needs fresh on-policy preferences because DPO has plateaued, Online DPO is another option.

Real-world application

Use the decision tree as an initial selection guide, then validate the choice with task-specific experiments.
Key takeaway

Start with the data and task requirements. Select the simplest suitable method and compare results under a fair compute budget.