8.7.5
The Alignment Method Decision Tree
Apply the source decision tree to choose an alignment method based on rewards, preferences, feedback, compute and application requirements.
What is it?
The chapter decision tree maps common training constraints to an alignment method. Start with the reward and feedback available, then consider quality, compute and the required output.
Key points
1. Verifiable rewards for math or code → GRPO. 2. Maximum quality on complex tasks → PPO. 3. Paired preferences → DPO, or IPO if noisy. 4. Unpaired binary feedback → KTO. 5. Memory-limited and starting from a base model → ORPO. 6. DPO plateauing and on-policy data needed → Online DPO. 7. Strong baseline quickly → Best-of-N or RFT. 8. Calibrated confidence with correctness → RLCR. 9. Typed, fast software decisions → RLCD or Jev. 10. Cached state/action embeddings for agent decisions → CLM.
Alignment Method Decision Tree
A team has paired preference data, but the labels are noisy. Follow the decision tree: paired preferences point to DPO, while noisy labels point to IPO. If the team later needs fresh on-policy preferences because DPO has plateaued, Online DPO is another option.
Real-world application
Use the decision tree as an initial selection guide, then validate the choice with task-specific experiments.
Key takeaway
Start with the data and task requirements. Select the simplest suitable method and compare results under a fair compute budget.