OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Preference Optimization Variants / Summary: Choosing an Alignment Method
Lesson 351 of 415 5 min read
8.7.2

Choosing an Alignment Method Based on Your Data

Understand how verifiable rewards, paired preferences, unpaired feedback and generated responses influence method selection.

What is it?

The type of feedback available is one of the strongest factors in choosing an alignment method. Paired preferences, unpaired feedback and verifiable rewards support different training approaches.

Key points

Verifiable math or code rewards → GRPO. Paired preference data → DPO. Noisy paired preferences → IPO. Unpaired binary feedback → KTO. DPO distribution shift or plateauing → Online DPO. Limited memory when combining supervised fine-tuning and alignment → ORPO.

Choosing from Available Data

A product team collects thumbs-up and thumbs-down feedback for individual answers, but users do not compare two answers side by side. KTO fits this unpaired binary feedback format. If the team instead collects chosen and rejected answer pairs, DPO or IPO becomes relevant.

Real-world application

Inspect the structure and quality of the dataset before choosing the training objective.
Key takeaway

Choose the method that matches the feedback format you actually have.