OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / DPO - Direct Preference Optimization / DPO Extensions and Variants
Lesson 250 of 415 7 min read
6.9.5

NCA – Noise Contrastive Alignment

Understand how NCA addresses likelihood collapse by adding absolute likelihood terms to preference alignment.

What is it?

A failure mode of DPO is likelihood collapse: the model decreases the probability of the rejected response but also decreases the probability of the preferred response. NCA adds absolute likelihood terms to address this problem and reframes alignment as noise-contrastive estimation.

Key points

NCA uses three loss terms. The first term encourages high implicit reward for the winning response. The second and third terms penalise high implicit reward for both winning and losing responses. Implicit reward: r_y = β log(π_θ(y|q) / π_ref(y|q)). TRL setting: loss_type = "nca_pair". The source demonstrates β = 0.01 to give the absolute likelihood term more weight.

Preventing Likelihood Collapse

During DPO training, a model learns to reduce the probability of a rejected answer. However, it also starts reducing the probability of the preferred answer. NCA adds absolute likelihood terms so training considers more than the relative difference between the two responses.

Real-world application

Consider NCA when the winning response probability decreases during DPO training or when absolute response quality matters, not only relative ranking. The source recommends a small β, such as 0.01, to give the absolute likelihood term more weight.
Key takeaway

NCA adds absolute likelihood terms to help prevent the model from lowering the probability of both preferred and rejected responses.