OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Mathematical Derivation
Lesson 209 of 415 6 min read
6.2.2

Deriving the Optimal Policy

Understand the closed-form optimal policy derived from the KL-regularized RLHF objective.

What is it?

Solving the KL-regularized objective produces a closed-form expression for the optimal policy.

Key points

1. π* represents the optimal policy. 2. π_ref is the reference policy. 3. Reward changes response probabilities. 4. β controls reward influence. 5. Z(x) is a prompt-dependent normalization term.

Optimal Policy Equation

π*(y|x) = [π_ref(y|x) × exp(r(x,y)/β)] / Z(x) The optimal policy adjusts the reference distribution according to the reward.

Real-world application

Use this equation to understand the connection between reward and policy probabilities.
Key takeaway

The optimal policy can be expressed using the reference policy and reward.