6.2.3
Expressing Reward Through Policy Probabilities
Understand how reward can be expressed using policy and reference-model probabilities.
What is it?
Taking the logarithm of the optimal policy equation and rearranging it expresses reward through the policy-to-reference probability ratio.
Key points
1. Divide the optimal policy by the reference policy.
2. Take the natural logarithm.
3. Multiply by β.
4. Include the normalization term.
5. Reward is now expressed using policy probabilities.
Reward Transformation
r(x,y) = β log[π*(y|x) / π_ref(y|x)] + β log Z(x)
The reward is represented through the relationship between the optimal policy and reference policy.
Real-world application
This transformation allows DPO to avoid explicitly training a separate reward model.
Key takeaway
Reward can be represented through policy and reference-model probabilities.