5.1.2
TRPO: Constraining Policy Updates
Understand how Trust Region Policy Optimization uses a KL constraint to limit policy changes.
What is it?
Trust Region Policy Optimization (TRPO), introduced in 2015, controls policy changes by constraining the KL divergence between the old and new policies. This helps prevent excessively large updates. Its limitation is the computational complexity of second-order optimization.
Key points
Introduced in 2015; Uses a KL divergence constraint; Limits changes between old and new policies; Uses Fisher information matrices; Uses conjugate gradients; Requires expensive second-order optimization.
Illustration: Keeping Policy Updates Within a Trust Region
Suppose an AI agent has an existing policy that performs reliably. TRPO restricts how far the updated policy can move from the previous policy using a KL divergence constraint. This reduces the risk of a destructive update, although the optimization process is computationally expensive.
Real-world application
TRPO provides an important reference point when studying algorithms designed to make reinforcement learning updates more stable.
Key takeaway
TRPO improves update stability through a KL constraint, but requires expensive second-order optimization.