OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Detailed Mechanics: Logits and Policy Updates
Lesson 188 of 415 7 min read
5.7.3

From Logits to Probability Ratio

Understand how logits and softmax probabilities produce the PPO probability ratio.

What is it?

The PPO probability ratio compares the probability of an action under the live policy with its probability under the old policy. To avoid numerical underflow or overflow, PPO calculates the ratio using log-probabilities and exponentiation. The resulting ratio is used in the clipped PPO objective.

Key points

1. The ratio is r_t(θ) = π_θ(a_t|s_t) / π_θold(a_t|s_t). 2. LogSoftmax converts logits into log-probabilities. 3. The live and old policy log-probabilities are calculated separately. 4. The ratio is exp(log π_θ − log π_θold). 5. The ratio is used in the clipped PPO objective. 6. Positive advantages are capped at 1 + ε. 7. Negative advantages are bounded at 1 − ε. 8. The minimum operator selects the more conservative objective value.

Example: Calculating a Probability Ratio

Suppose: Live policy log-probability = −0.4 Old policy log-probability = −0.6 Difference = −0.4 − (−0.6) = 0.2 Probability ratio = exp(0.2) ≈ 1.22 The selected action is approximately 1.22 times as likely under the live policy as under the old policy.

Real-world application

The log-space calculation provides a numerically stable way to compute PPO probability ratios for large vocabulary models.
Key takeaway

PPO calculates probability ratios from log-probabilities and uses clipping to control policy updates.