6.9.7
Iterative RPO – Reasoning Preference Optimisation
Understand how RPO combines DPO with a negative log-likelihood term on preferred responses to preserve generation ability.
What is it?
Standard DPO teaches a model to distinguish winning responses from losing responses. For reasoning tasks, the model must also learn to generate the preferred reasoning trace. RPO adds a negative log-likelihood term on the winning response and combines DPO with supervised fine-tuning.
Key points
RPO loss: L_RPO = λ₁ L_DPO(y_w, y_l) + λ₂ L_NLL(y_w).
L_NLL(y_w) = −log π_θ(y_w|q).
The DPO term teaches preference discrimination.
The NLL term teaches the model to generate the winning response.
TRL example: loss_type = "sigmoid", rpo_alpha = 1.0, beta = 0.1.
The source recommends RPO for reasoning tasks and when DPO reduces fluency or generation quality.
Combining Preference Learning and Generation
For a mathematics problem, a preferred response may contain a correct step-by-step derivation. DPO encourages the model to rank this response above an incorrect one. RPO also applies an NLL objective to the preferred response, helping the model learn to generate the reasoning trace itself.
Real-world application
Consider RPO for mathematics, coding and logic tasks where the model must generate step-by-step solutions. It is also relevant when DPO training reduces fluency or generation quality.
Key takeaway
RPO combines preference discrimination with a generation objective on the preferred response.