6.5.1
Sequence-Level Log-Probabilities
Understand how DPO calculates the log-probability of an entire response by summing token-level log-probabilities.
What is it?
DPO calculates the log-probability of an entire response by summing the log-probabilities of its individual tokens, conditioned on the prompt and preceding tokens.
Key points
1. A response consists of tokens y1, y2, ... yT.
2. Each token probability depends on the prompt and preceding tokens.
3. Apply log-softmax to obtain each token log-probability.
4. Sum all response-token log-probabilities.
5. The standard DPO calculation uses a sum rather than an average.
6. Every response token contributes to the sequence log-probability.
Sequence Log-Probability
The sequence probability is factorized autoregressively:
πθ(y|x) = Π(t=1 to T) πθ(yt|x, y
Real-world application
Calculate sequence-level log-probabilities for chosen and rejected responses before computing the DPO loss.
Key takeaway
DPO uses the sum of token-level log-probabilities to represent the probability of an entire response.