5.7.3
From Logits to Probability Ratio
Understand how logits and softmax probabilities produce the PPO probability ratio.
What is it?
The PPO probability ratio compares the probability of an action under the live policy with its probability under the old policy. To avoid numerical underflow or overflow, PPO calculates the ratio using log-probabilities and exponentiation. The resulting ratio is used in the clipped PPO objective.
Key points
1. The ratio is r_t(θ) = π_θ(a_t|s_t) / π_θold(a_t|s_t).
2. LogSoftmax converts logits into log-probabilities.
3. The live and old policy log-probabilities are calculated separately.
4. The ratio is exp(log π_θ − log π_θold).
5. The ratio is used in the clipped PPO objective.
6. Positive advantages are capped at 1 + ε.
7. Negative advantages are bounded at 1 − ε.
8. The minimum operator selects the more conservative objective value.
Example: Calculating a Probability Ratio
Suppose:
Live policy log-probability = −0.4
Old policy log-probability = −0.6
Difference = −0.4 − (−0.6) = 0.2
Probability ratio = exp(0.2) ≈ 1.22
The selected action is approximately 1.22 times as likely under the live policy as under the old policy.
Real-world application
The log-space calculation provides a numerically stable way to compute PPO probability ratios for large vocabulary models.
Key takeaway
PPO calculates probability ratios from log-probabilities and uses clipping to control policy updates.