5.4.4
Step 4: The Problem with Unconstrained Surrogates
Understand the risks of extreme importance ratios and unconstrained policy updates.
What is it?
The unconstrained surrogate objective is valid, but a single gradient step can push the probability ratio far from 1.0. This can create extreme importance weights, high variance and unstable policy behaviour. TRPO addresses this by imposing a KL divergence constraint.
Key points
1. The unconstrained surrogate is L_CPI.
2. A large update can push the probability ratio far from 1.0.
3. Extreme importance weights cause high variance.
4. The policy may enter untested regions.
5. The reward model may give unreliable scores in those regions.
6. Severe updates can cause catastrophic policy collapse.
7. TRPO constrains D_KL(πθold || πθ) ≤ δ.
8. TRPO requires expensive second-order optimization.
Example: An Excessive Policy Update
Suppose the old policy assigns probability 0.20 to an action, but an unconstrained update increases it to 0.90.
Probability ratio = 0.90 / 0.20 = 4.5.
This is a very large change. Such extreme ratios can produce unstable importance weights and unreliable updates.
Real-world application
When designing policy optimization, monitor policy changes and importance ratios. Unconstrained updates can destabilize training.
Key takeaway
An unconstrained surrogate can produce extreme policy updates. TRPO addresses this using a KL divergence constraint.