OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Mathematical Derivation
Lesson 208 of 415 5 min read
6.2.1

The RLHF Objective

Understand reward maximization with KL divergence regularization.

What is it?

The RLHF objective maximizes expected reward while penalizing divergence from a reference policy.

Key points

1. Maximize expected reward. 2. Penalize KL divergence. 3. π is the policy being optimized. 4. π_ref is the reference policy. 5. β controls the KL penalty strength.

RLHF Objective

max_π E[r(x,y)] − β D_KL[π || π_ref] The first term maximizes reward. The second term discourages excessive deviation from the reference policy.

Real-world application

Understand this objective before studying the mathematical derivation of DPO.
Key takeaway

RLHF balances reward improvement with reference-policy regularization.