6.5.6
Label Masking: What Gets Gradients
Identify which prompt, chosen-response, rejected-response and padding tokens receive gradients.
What is it?
DPO computes its loss over response tokens only. Prompt tokens and padding tokens are excluded, while every chosen and rejected response token contributes.
Key points
1. Prompt tokens provide context but do not contribute to response log-probability.
2. Chosen response tokens receive gradients.
3. Rejected response tokens receive gradients.
4. Padding tokens are excluded using attention masks.
5. Prompt labels are masked with −100.
6. Response labels remain available for log-probability calculation.
Which Tokens Receive Gradients?
Prompt: NO gradient contribution to the DPO response loss.
Chosen response: ALL response tokens contribute. Their probabilities are pushed upward.
Rejected response: ALL response tokens contribute. Their probabilities are pushed downward.
Padding: NO contribution. Padding is excluded using the attention mask.
In the implementation, labels equal to −100 are excluded from the sequence log-probability sum.
Real-world application
Check label construction and attention masks when preparing preference datasets for DPO training.
Key takeaway
Correct masking ensures that DPO optimizes responses rather than prompt or padding tokens.