6.9.5
NCA – Noise Contrastive Alignment
Understand how NCA addresses likelihood collapse by adding absolute likelihood terms to preference alignment.
What is it?
A failure mode of DPO is likelihood collapse: the model decreases the probability of the rejected response but also decreases the probability of the preferred response. NCA adds absolute likelihood terms to address this problem and reframes alignment as noise-contrastive estimation.
Key points
NCA uses three loss terms.
The first term encourages high implicit reward for the winning response.
The second and third terms penalise high implicit reward for both winning and losing responses.
Implicit reward: r_y = β log(π_θ(y|q) / π_ref(y|q)).
TRL setting: loss_type = "nca_pair".
The source demonstrates β = 0.01 to give the absolute likelihood term more weight.
Preventing Likelihood Collapse
During DPO training, a model learns to reduce the probability of a rejected answer. However, it also starts reducing the probability of the preferred answer. NCA adds absolute likelihood terms so training considers more than the relative difference between the two responses.
Real-world application
Consider NCA when the winning response probability decreases during DPO training or when absolute response quality matters, not only relative ranking. The source recommends a small β, such as 0.01, to give the absolute likelihood term more weight.
Key takeaway
NCA adds absolute likelihood terms to help prevent the model from lowering the probability of both preferred and rejected responses.