OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 226 of 415 7 min read
6.5.3

Forward Pass: Step by Step

Follow sequence construction, model forward passes, log-probability extraction, reference comparison and loss calculation.

What is it?

A DPO training forward pass constructs chosen and rejected sequences, extracts response-token log-probabilities, compares them with reference values and calculates the loss.

Key points

1. Concatenate prompt and chosen response. 2. Concatenate prompt and rejected response. 3. Pad sequences within the batch. 4. Run both through the policy model. 5. Extract actual response-token log-probabilities. 6. Sum the token log-probabilities. 7. Subtract reference log-probabilities. 8. Calculate DPO loss and perform backpropagation.

DPO Forward Pass

For one training example (x, yw, yl): Step 1: Construct [x; yw] and [x; yl]. Step 2: Run both sequences through policy πθ. Step 3: Apply log-softmax and select the log-probability of each actual response token. Step 4: Sum response-token log-probabilities: logp_chosen = Σ log πθ(yw_t | x, yw_

Real-world application

Use this sequence of operations to understand or implement the DPO forward and backward training pipeline.
Key takeaway

DPO compares policy and reference probabilities for both responses before updating the policy.