5.4.3
Step 3: Importance Sampling for Off-Policy Data
Understand how importance sampling corrects for differences between old and new policies.
What is it?
PPO collects training data using the old policy πθold but updates the current policy πθ. Importance sampling corrects this distribution mismatch by weighting samples using the probability ratio between the new and old policies.
Key points
1. Training samples are generated by πθold.
2. The updated policy is πθ.
3. Importance sampling corrects the distribution mismatch.
4. Probability ratio: r_t(θ) = πθ(a_t|s_t) / πθold(a_t|s_t).
5. Equation (5.5) applies this ratio to the policy gradient.
6. Using ∇θ log f = ∇θf / f gives Equation (5.6).
7. The surrogate objective is L_CPI(θ) = E_t[r_t(θ) · Â_t], Equation (5.7).
Example: Calculating the Importance Ratio
Suppose the old policy assigns an action probability of 0.40 and the updated policy assigns 0.50.
Probability ratio = 0.50 / 0.40 = 1.25.
The ratio indicates that the new policy assigns 25% more probability to the sampled action than the old policy.
Real-world application
Importance sampling allows PPO to reuse data generated by the old policy during multiple optimization epochs while accounting for the policy distribution difference.
Key takeaway
The probability ratio connects old-policy training data with the updated policy.