6.2.5
Cancelling the Normalization Term
Understand why the normalization term cancels when comparing responses to the same prompt.
What is it?
When the reward equation is substituted into Bradley-Terry, both responses share the same prompt. Their normalization terms therefore cancel.
Key points
1. Both responses belong to prompt x.
2. Each reward contains β log Z(x).
3. Subtract the rejected reward from the chosen reward.
4. The identical normalization terms cancel.
5. The remaining expression uses policy probabilities.
Why Z(x) Cancels
Chosen reward contains: + β log Z(x).
Rejected reward contains: + β log Z(x).
Subtracting the two rewards removes the shared term.
Real-world application
This cancellation removes the need to calculate the normalization term during DPO preference comparison.
Key takeaway
The same prompt causes the normalization term to cancel.