9.1.2
Preference Probability and the Sigmoid Function
Explore the sigmoid transformation and how reward differences become pairwise preference probabilities.
What is it?
The sigmoid function converts the difference between two reward scores into a probability between zero and one. This makes the reward difference interpretable as the likelihood of a preference.
Key points
The sigmoid function is σ(z) = 1 / (1 + e^(−z)). If both responses have equal rewards, the preference probability is 0.5. A positive reward difference raises the probability above 0.5. A negative difference lowers it below 0.5. The Bradley-Terry probability can also be written using exponentiated rewards: exp(r1) / (exp(r1) + exp(r2)).
From Reward Scores to Preference Probability
Suppose Response A has reward 2 and Response B has reward 1. The difference is 1, so the probability of preferring A is σ(1), approximately 0.731. Reversing the scores gives approximately 0.269.
Real-world application
Interpret reward model outputs as relative preference probabilities rather than direct measures of absolute response quality.
Key takeaway
The sigmoid maps reward differences to preference probabilities.