OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Summary: Choosing an Alignment Method
Lesson 352 of 415 5 min read
8.7.3

Choosing an Alignment Method Based on Compute and Stability

Select methods according to GPU resources, memory limits, training complexity and stability requirements.

What is it?

Compute availability and training stability help narrow the choice after identifying a suitable data format. More complex online methods can require substantially more resources.

Key points

Very low compute and memory requirements: ORPO. Low compute: DPO, KTO and IPO. Medium compute: Online DPO, Best-of-N, RLCR, RLCD and CLM. High compute: GRPO. Very high compute: PPO. IPO has very high stability; Best-of-N is listed with perfect stability in the comparison table.

Choosing Under Compute Constraints

A team has limited GPU memory and wants to start from a base model using offline preference pairs. ORPO is a relevant option because it combines supervised fine-tuning and preference alignment without a separate reference model.

Real-world application

Estimate available GPU memory, training budget and tolerance for instability before committing to a method.
Key takeaway

A method must be practical to train and stable enough for the intended workflow.