6.9.2
Robust DPO
Learn how Robust DPO analytically corrects preference-training loss for noisy or incorrectly flipped preference labels.
What is it?
Human preference annotations can contain disagreements, mistakes and flipped chosen/rejected labels. Standard DPO treats these labels as ground truth and can overfit to noise. Robust DPO analytically debiases the loss under an assumed label-flip probability.
Key points
Let ε represent the probability that a preference label is flipped.
L_robust = [(1 − ε)L_DPO(yw, yl) − εL_DPO(yl, yw)] / (1 − 2ε).
When ε = 0, the loss reduces to standard DPO.
When ε = 0.5, the denominator becomes zero and learning is not possible.
The source gives ε ∈ [0.05, 0.2] as a typical annotation-noise range.
TRL example: loss_type = "robust", label_smoothing = 0.1, beta = 0.1.
Handling Noisy Preference Labels
Suppose annotators sometimes mark the rejected response as preferred. With an assumed noise rate ε = 0.1, Robust DPO adjusts the loss using both the original and label-flipped preference pair. This reduces the bias caused by incorrect labels under the assumed noise model.
Real-world application
When preference annotations are known to contain noise, consider a robust loss configuration. Estimate the label-noise level carefully rather than assuming all preference labels are correct.
Key takeaway
Robust DPO corrects preference-training loss for label noise using an assumed probability of flipped labels.