6.2.1
The RLHF Objective
Understand reward maximization with KL divergence regularization.
What is it?
The RLHF objective maximizes expected reward while penalizing divergence from a reference policy.
Key points
1. Maximize expected reward.
2. Penalize KL divergence.
3. π is the policy being optimized.
4. π_ref is the reference policy.
5. β controls the KL penalty strength.
RLHF Objective
max_π E[r(x,y)] − β D_KL[π || π_ref]
The first term maximizes reward. The second term discourages excessive deviation from the reference policy.
Real-world application
Understand this objective before studying the mathematical derivation of DPO.
Key takeaway
RLHF balances reward improvement with reference-policy regularization.