OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Mathematical Derivation
Lesson 210 of 415 6 min read
6.2.3

Expressing Reward Through Policy Probabilities

Understand how reward can be expressed using policy and reference-model probabilities.

What is it?

Taking the logarithm of the optimal policy equation and rearranging it expresses reward through the policy-to-reference probability ratio.

Key points

1. Divide the optimal policy by the reference policy. 2. Take the natural logarithm. 3. Multiply by β. 4. Include the normalization term. 5. Reward is now expressed using policy probabilities.

Reward Transformation

r(x,y) = β log[π*(y|x) / π_ref(y|x)] + β log Z(x) The reward is represented through the relationship between the optimal policy and reference policy.

Real-world application

This transformation allows DPO to avoid explicitly training a separate reward model.
Key takeaway

Reward can be represented through policy and reference-model probabilities.