6.2.2
Deriving the Optimal Policy
Understand the closed-form optimal policy derived from the KL-regularized RLHF objective.
What is it?
Solving the KL-regularized objective produces a closed-form expression for the optimal policy.
Key points
1. π* represents the optimal policy.
2. π_ref is the reference policy.
3. Reward changes response probabilities.
4. β controls reward influence.
5. Z(x) is a prompt-dependent normalization term.
Optimal Policy Equation
π*(y|x) = [π_ref(y|x) × exp(r(x,y)/β)] / Z(x)
The optimal policy adjusts the reference distribution according to the reward.
Real-world application
Use this equation to understand the connection between reward and policy probabilities.
Key takeaway
The optimal policy can be expressed using the reference policy and reward.