OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Derivation of the PPO Gradient and Update Rule
Lesson 173 of 415 6 min read
5.4.3

Step 3: Importance Sampling for Off-Policy Data

Understand how importance sampling corrects for differences between old and new policies.

What is it?

PPO collects training data using the old policy πθold but updates the current policy πθ. Importance sampling corrects this distribution mismatch by weighting samples using the probability ratio between the new and old policies.

Key points

1. Training samples are generated by πθold. 2. The updated policy is πθ. 3. Importance sampling corrects the distribution mismatch. 4. Probability ratio: r_t(θ) = πθ(a_t|s_t) / πθold(a_t|s_t). 5. Equation (5.5) applies this ratio to the policy gradient. 6. Using ∇θ log f = ∇θf / f gives Equation (5.6). 7. The surrogate objective is L_CPI(θ) = E_t[r_t(θ) · Â_t], Equation (5.7).

Example: Calculating the Importance Ratio

Suppose the old policy assigns an action probability of 0.40 and the updated policy assigns 0.50. Probability ratio = 0.50 / 0.40 = 1.25. The ratio indicates that the new policy assigns 25% more probability to the sampled action than the old policy.

Real-world application

Importance sampling allows PPO to reuse data generated by the old policy during multiple optimization epochs while accounting for the policy distribution difference.
Key takeaway

The probability ratio connects old-policy training data with the updated policy.