OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 252 of 415 7 min read
6.9.7

Iterative RPO – Reasoning Preference Optimisation

Understand how RPO combines DPO with a negative log-likelihood term on preferred responses to preserve generation ability.

What is it?

Standard DPO teaches a model to distinguish winning responses from losing responses. For reasoning tasks, the model must also learn to generate the preferred reasoning trace. RPO adds a negative log-likelihood term on the winning response and combines DPO with supervised fine-tuning.

Key points

RPO loss: L_RPO = λ₁ L_DPO(y_w, y_l) + λ₂ L_NLL(y_w). L_NLL(y_w) = −log π_θ(y_w|q). The DPO term teaches preference discrimination. The NLL term teaches the model to generate the winning response. TRL example: loss_type = "sigmoid", rpo_alpha = 1.0, beta = 0.1. The source recommends RPO for reasoning tasks and when DPO reduces fluency or generation quality.

Combining Preference Learning and Generation

For a mathematics problem, a preferred response may contain a correct step-by-step derivation. DPO encourages the model to rank this response above an incorrect one. RPO also applies an NLL objective to the preferred response, helping the model learn to generate the reasoning trace itself.

Real-world application

Consider RPO for mathematics, coding and logic tasks where the model must generate step-by-step solutions. It is also relevant when DPO training reduces fluency or generation quality.
Key takeaway

RPO combines preference discrimination with a generation objective on the preferred response.