6.5.7
Pseudocode: DPO Training Step
Understand the PyTorch-style DPO forward and backward training step, including sequence log-probability calculation.
What is it?
A DPO training step runs policy and reference forward passes, calculates sequence log-probabilities, computes implicit rewards and minimizes the preference loss.
Key points
1. Obtain chosen and rejected policy log-probabilities.
2. Obtain reference log-probabilities without gradients.
3. Calculate chosen and rejected rewards.
4. Calculate the reward margin.
5. Apply log-sigmoid to obtain the loss.
6. Sum only response-token log-probabilities.
7. Shift logits and labels for next-token prediction.
8. Exclude labels set to −100.
PyTorch-Style DPO Training Step
def get_sequence_logprob(model, sequences):
outputs = model(
sequences["input_ids"],
attention_mask=sequences["mask"]
)
logits = outputs.logits[:, :-1, :]
labels = sequences["labels"][:, 1:]
log_probs = F.log_softmax(logits, dim=-1)
token_logps = log_probs.gather(
-1, labels.unsqueeze(-1)
).squeeze(-1)
mask = (labels != -100).float()
return (token_logps * mask).sum(dim=-1)
def dpo_loss(model, ref_model, batch, beta=0.1):
logps_chosen = get_sequence_logprob(
model, batch["chosen"]
)
logps_rejected = get_sequence_logprob(
model, batch["rejected"]
)
with torch.no_grad():
ref_chosen = get_sequence_logprob(
ref_model, batch["chosen"]
)
ref_rejected = get_sequence_logprob(
ref_model, batch["rejected"]
)
chosen_rewards = beta * (logps_chosen - ref_chosen)
rejected_rewards = beta * (logps_rejected - ref_rejected)
loss = -F.logsigmoid(
chosen_rewards - rejected_rewards
).mean()
return loss
Real-world application
Study the pseudocode to understand the complete DPO computation before using a training framework such as Hugging Face TRL.
Key takeaway
The training step combines response-only log-probability sums, reference comparisons and a sigmoid preference loss.