OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Mathematical Derivation
Lesson 212 of 415 5 min read
6.2.5

Cancelling the Normalization Term

Understand why the normalization term cancels when comparing responses to the same prompt.

What is it?

When the reward equation is substituted into Bradley-Terry, both responses share the same prompt. Their normalization terms therefore cancel.

Key points

1. Both responses belong to prompt x. 2. Each reward contains β log Z(x). 3. Subtract the rejected reward from the chosen reward. 4. The identical normalization terms cancel. 5. The remaining expression uses policy probabilities.

Why Z(x) Cancels

Chosen reward contains: + β log Z(x). Rejected reward contains: + β log Z(x). Subtracting the two rewards removes the shared term.

Real-world application

This cancellation removes the need to calculate the normalization term during DPO preference comparison.
Key takeaway

The same prompt causes the normalization term to cancel.