9.3.3
Margin Losses
Using a margin to enforce meaningful reward separation between preferred and dispreferred responses.
What is it?
Margin loss adds a required margin between the reward assigned to a preferred response and a dispreferred response. It encourages the reward model to produce meaningfully separated scores.
Key points
Let r_w be the reward for the preferred response.
Let r_l be the reward for the dispreferred response.
Let m be the required margin.
The margin loss is L_margin = max(0, m - (r_w - r_l)).
The loss is zero when the reward difference reaches the required margin.
Enforcing a Reward Margin
Suppose the preferred response receives a reward of 0.8 and the dispreferred response receives 0.6. Their reward difference is 0.2. If the required margin is 0.5, the margin loss is 0.3. The model is encouraged to increase the separation between the two scores.
Real-world application
Use a margin in reward model training when merely ranking the preferred response above the dispreferred response is not enough and a meaningful score difference is required.
Key takeaway
Margin loss encourages a minimum reward gap between preferred and dispreferred responses.