OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / Gradient Analysis
Lesson 216 of 415 5 min read
6.3.3

Increasing Chosen and Decreasing Rejected Probabilities

Understand how gradient descent changes chosen and rejected response probabilities.

What is it?

The DPO gradient increases the chosen response probability and decreases the rejected response probability.

Key points

1. Chosen response log-probability is pushed upward. 2. Rejected response log-probability is pushed downward. 3. Both responses contribute to the gradient. 4. The objective increases their relative preference margin.

Two Competing Responses

Prompt: Explain a scientific concept. Chosen response: Clear and suitable explanation. Rejected response: Less suitable explanation. Training increases preference for the chosen response and reduces preference for the rejected response.

Real-world application

This mechanism helps align language-model responses with human preferences.
Key takeaway

DPO learns both which response to prefer and which response to avoid.