OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Bradley-Terry Model: Full Derivation
Lesson 356 of 415 5 min read
9.1.2

Preference Probability and the Sigmoid Function

Explore the sigmoid transformation and how reward differences become pairwise preference probabilities.

What is it?

The sigmoid function converts the difference between two reward scores into a probability between zero and one. This makes the reward difference interpretable as the likelihood of a preference.

Key points

The sigmoid function is σ(z) = 1 / (1 + e^(−z)). If both responses have equal rewards, the preference probability is 0.5. A positive reward difference raises the probability above 0.5. A negative difference lowers it below 0.5. The Bradley-Terry probability can also be written using exponentiated rewards: exp(r1) / (exp(r1) + exp(r2)).

From Reward Scores to Preference Probability

Suppose Response A has reward 2 and Response B has reward 1. The difference is 1, so the probability of preferring A is σ(1), approximately 0.731. Reversing the scores gives approximately 0.269.

Real-world application

Interpret reward model outputs as relative preference probabilities rather than direct measures of absolute response quality.
Key takeaway

The sigmoid maps reward differences to preference probabilities.