OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 229 of 415 5 min read
6.5.6

Label Masking: What Gets Gradients

Identify which prompt, chosen-response, rejected-response and padding tokens receive gradients.

What is it?

DPO computes its loss over response tokens only. Prompt tokens and padding tokens are excluded, while every chosen and rejected response token contributes.

Key points

1. Prompt tokens provide context but do not contribute to response log-probability. 2. Chosen response tokens receive gradients. 3. Rejected response tokens receive gradients. 4. Padding tokens are excluded using attention masks. 5. Prompt labels are masked with −100. 6. Response labels remain available for log-probability calculation.

Which Tokens Receive Gradients?

Prompt: NO gradient contribution to the DPO response loss. Chosen response: ALL response tokens contribute. Their probabilities are pushed upward. Rejected response: ALL response tokens contribute. Their probabilities are pushed downward. Padding: NO contribution. Padding is excluded using the attention mask. In the implementation, labels equal to −100 are excluded from the sequence log-probability sum.

Real-world application

Check label construction and attention masks when preparing preference datasets for DPO training.
Key takeaway

Correct masking ensures that DPO optimizes responses rather than prompt or padding tokens.