8.7.2
Choosing an Alignment Method Based on Your Data
Understand how verifiable rewards, paired preferences, unpaired feedback and generated responses influence method selection.
What is it?
The type of feedback available is one of the strongest factors in choosing an alignment method. Paired preferences, unpaired feedback and verifiable rewards support different training approaches.
Key points
Verifiable math or code rewards → GRPO. Paired preference data → DPO. Noisy paired preferences → IPO. Unpaired binary feedback → KTO. DPO distribution shift or plateauing → Online DPO. Limited memory when combining supervised fine-tuning and alignment → ORPO.
Choosing from Available Data
A product team collects thumbs-up and thumbs-down feedback for individual answers, but users do not compare two answers side by side. KTO fits this unpaired binary feedback format. If the team instead collects chosen and rejected answer pairs, DPO or IPO becomes relevant.
Real-world application
Inspect the structure and quality of the dataset before choosing the training objective.
Key takeaway
Choose the method that matches the feedback format you actually have.