OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / How DPO Works: Full Mechanics
Lesson 230 of 415 8 min read
6.5.7

Pseudocode: DPO Training Step

Understand the PyTorch-style DPO forward and backward training step, including sequence log-probability calculation.

What is it?

A DPO training step runs policy and reference forward passes, calculates sequence log-probabilities, computes implicit rewards and minimizes the preference loss.

Key points

1. Obtain chosen and rejected policy log-probabilities. 2. Obtain reference log-probabilities without gradients. 3. Calculate chosen and rejected rewards. 4. Calculate the reward margin. 5. Apply log-sigmoid to obtain the loss. 6. Sum only response-token log-probabilities. 7. Shift logits and labels for next-token prediction. 8. Exclude labels set to −100.

PyTorch-Style DPO Training Step

def get_sequence_logprob(model, sequences): outputs = model( sequences["input_ids"], attention_mask=sequences["mask"] ) logits = outputs.logits[:, :-1, :] labels = sequences["labels"][:, 1:] log_probs = F.log_softmax(logits, dim=-1) token_logps = log_probs.gather( -1, labels.unsqueeze(-1) ).squeeze(-1) mask = (labels != -100).float() return (token_logps * mask).sum(dim=-1) def dpo_loss(model, ref_model, batch, beta=0.1): logps_chosen = get_sequence_logprob( model, batch["chosen"] ) logps_rejected = get_sequence_logprob( model, batch["rejected"] ) with torch.no_grad(): ref_chosen = get_sequence_logprob( ref_model, batch["chosen"] ) ref_rejected = get_sequence_logprob( ref_model, batch["rejected"] ) chosen_rewards = beta * (logps_chosen - ref_chosen) rejected_rewards = beta * (logps_rejected - ref_rejected) loss = -F.logsigmoid( chosen_rewards - rejected_rewards ).mean() return loss

Real-world application

Study the pseudocode to understand the complete DPO computation before using a training framework such as Hugging Face TRL.
Key takeaway

The training step combines response-only log-probability sums, reference comparisons and a sigmoid preference loss.