6.5.3
Forward Pass: Step by Step
Follow sequence construction, model forward passes, log-probability extraction, reference comparison and loss calculation.
What is it?
A DPO training forward pass constructs chosen and rejected sequences, extracts response-token log-probabilities, compares them with reference values and calculates the loss.
Key points
1. Concatenate prompt and chosen response.
2. Concatenate prompt and rejected response.
3. Pad sequences within the batch.
4. Run both through the policy model.
5. Extract actual response-token log-probabilities.
6. Sum the token log-probabilities.
7. Subtract reference log-probabilities.
8. Calculate DPO loss and perform backpropagation.
DPO Forward Pass
For one training example (x, yw, yl):
Step 1: Construct [x; yw] and [x; yl].
Step 2: Run both sequences through policy πθ.
Step 3: Apply log-softmax and select the log-probability of each actual response token.
Step 4: Sum response-token log-probabilities:
logp_chosen = Σ log πθ(yw_t | x, yw_
Real-world application
Use this sequence of operations to understand or implement the DPO forward and backward training pipeline.
Key takeaway
DPO compares policy and reference probabilities for both responses before updating the policy.