OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / The Clipped Objective
Lesson 165 of 415 5 min read
5.2.1

The Probability Ratio

Understand the ratio between new-policy and old-policy action probabilities.

What is it?

The probability ratio measures how likely an action is under the new policy compared with the old policy. It is calculated by dividing the new policy probability by the old policy probability for the same action and state.

Key points

1. Formula: r_t(θ) = π_θ(a_t|s_t) / π_θold(a_t|s_t). 2. r = 1 means both policies assign the same probability. 3. r greater than 1 means the new policy increases the probability. 4. r less than 1 means the new policy decreases the probability. 5. PPO uses this ratio to measure policy changes. 6. The ratio is combined with the estimated advantage.

Example: Comparing old and new policy probabilities

Suppose the old policy assigns a probability of 0.40 to an action, while the new policy assigns 0.48. Probability ratio = 0.48 / 0.40 = 1.20. The new policy makes this action 1.2 times as likely as the old policy. If the new probability were 0.32 instead, the ratio would be 0.32 / 0.40 = 0.80.

Real-world application

During PPO training, calculate the probability ratio for sampled actions to understand how much the updated policy differs from the policy that generated the training data.
Key takeaway

The probability ratio measures the relative change in an action probability between the new and old policies.