OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / LLM Architecture and Optimization Methods / Text Generation: Decoding Methods
Lesson 74 of 415 7 min read
1.12.8

Contrastive Decoding

What is it?

Contrastive decoding compares the probability predictions of a stronger expert model and a weaker amateur model. It favours tokens that the expert considers more likely than the amateur does, helping suppress generic continuations.

Key points

Uses two models: an expert and an amateur. Compares their log probabilities. Applies a plausibility constraint based on the expert model. Can reduce repetitive and generic phrasing. Requires running two models and depends on the choice of amateur model and threshold.

Expert and Amateur Models

The expert model assigns a high probability to a distinctive, useful continuation, while the amateur model also favours a generic phrase. Contrastive decoding subtracts the amateur log probability from the expert log probability, favouring the continuation that better reflects the expert model.

Real-world application

Useful for factual and coherent long-form generation when reducing generic or repetitive language is important and the additional compute is acceptable.
Key takeaway

Contrastive decoding tries to isolate the expert model’s distinctive signal by subtracting the weaker model’s predictions.