1.12.8
Contrastive Decoding
What is it?
Contrastive decoding compares the probability predictions of a stronger expert model and a weaker amateur model. It favours tokens that the expert considers more likely than the amateur does, helping suppress generic continuations.
Key points
Uses two models: an expert and an amateur. Compares their log probabilities. Applies a plausibility constraint based on the expert model. Can reduce repetitive and generic phrasing. Requires running two models and depends on the choice of amateur model and threshold.
Expert and Amateur Models
The expert model assigns a high probability to a distinctive, useful continuation, while the amateur model also favours a generic phrase. Contrastive decoding subtracts the amateur log probability from the expert log probability, favouring the continuation that better reflects the expert model.
Real-world application
Useful for factual and coherent long-form generation when reducing generic or repetitive language is important and the additional compute is acceptable.
Key takeaway
Contrastive decoding tries to isolate the expert model’s distinctive signal by subtracting the weaker model’s predictions.