5.2.1
The Probability Ratio
Understand the ratio between new-policy and old-policy action probabilities.
What is it?
The probability ratio measures how likely an action is under the new policy compared with the old policy. It is calculated by dividing the new policy probability by the old policy probability for the same action and state.
Key points
1. Formula: r_t(θ) = π_θ(a_t|s_t) / π_θold(a_t|s_t).
2. r = 1 means both policies assign the same probability.
3. r greater than 1 means the new policy increases the probability.
4. r less than 1 means the new policy decreases the probability.
5. PPO uses this ratio to measure policy changes.
6. The ratio is combined with the estimated advantage.
Example: Comparing old and new policy probabilities
Suppose the old policy assigns a probability of 0.40 to an action, while the new policy assigns 0.48.
Probability ratio = 0.48 / 0.40 = 1.20.
The new policy makes this action 1.2 times as likely as the old policy.
If the new probability were 0.32 instead, the ratio would be 0.32 / 0.40 = 0.80.
Real-world application
During PPO training, calculate the probability ratio for sampled actions to understand how much the updated policy differs from the policy that generated the training data.
Key takeaway
The probability ratio measures the relative change in an action probability between the new and old policies.