OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 224 of 415 6 min read
6.5.1

Sequence-Level Log-Probabilities

Understand how DPO calculates the log-probability of an entire response by summing token-level log-probabilities.

What is it?

DPO calculates the log-probability of an entire response by summing the log-probabilities of its individual tokens, conditioned on the prompt and preceding tokens.

Key points

1. A response consists of tokens y1, y2, ... yT. 2. Each token probability depends on the prompt and preceding tokens. 3. Apply log-softmax to obtain each token log-probability. 4. Sum all response-token log-probabilities. 5. The standard DPO calculation uses a sum rather than an average. 6. Every response token contributes to the sequence log-probability.

Sequence Log-Probability

The sequence probability is factorized autoregressively: πθ(y|x) = Π(t=1 to T) πθ(yt|x, y

Real-world application

Calculate sequence-level log-probabilities for chosen and rejected responses before computing the DPO loss.
Key takeaway

DPO uses the sum of token-level log-probabilities to represent the probability of an entire response.