OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Reward Model Training Tricks
Lesson 365 of 415 5 min read
9.3.3

Margin Losses

Using a margin to enforce meaningful reward separation between preferred and dispreferred responses.

What is it?

Margin loss adds a required margin between the reward assigned to a preferred response and a dispreferred response. It encourages the reward model to produce meaningfully separated scores.

Key points

Let r_w be the reward for the preferred response. Let r_l be the reward for the dispreferred response. Let m be the required margin. The margin loss is L_margin = max(0, m - (r_w - r_l)). The loss is zero when the reward difference reaches the required margin.

Enforcing a Reward Margin

Suppose the preferred response receives a reward of 0.8 and the dispreferred response receives 0.6. Their reward difference is 0.2. If the required margin is 0.5, the margin loss is 0.3. The model is encouraged to increase the separation between the two scores.

Real-world application

Use a margin in reward model training when merely ranking the preferred response above the dispreferred response is not enough and a meaningful score difference is required.
Key takeaway

Margin loss encourages a minimum reward gap between preferred and dispreferred responses.