3.11.3
Theoretical Guarantees
Understand policy invariance, loop immunity and convergence acceleration provided by potential-based reward shaping.
What is it?
The PBRS Policy Invariance Theorem provides three guarantees: the optimal policy remains unchanged, cyclic trajectories cannot exploit net potential rewards, and denser learning signals can accelerate convergence in sparse-reward environments.
Key points
1. Policy invariance: The optimal policy π* under R′ is identical to the optimal policy under R.
2. Shaping cannot introduce sub-optimal behavior.
3. Loop immunity: A cycle returning to its starting state has zero net potential change.
4. Potential change around a complete cycle: Φ(s) − Φ(s) = 0.
5. Convergence acceleration: Denser gradient signals help learning.
6. The source reports 5–50× faster convergence in sparse-reward environments.
7. The optimal policy remains unchanged despite the additional feedback.
Example: Preventing a robot from exploiting reward loops
Imagine a robot repeatedly visiting the same intermediate checkpoint.
With arbitrary reward shaping, it might collect additional rewards indefinitely without completing its task.
With PBRS, a cycle that starts and ends at the same state has zero net potential change.
The robot cannot accumulate net shaping reward simply by repeating that loop.
The optimal policy remains the same, while useful intermediate signals can help learning progress faster.
Real-world application
PBRS is useful when training agents in sparse-reward environments where intermediate feedback is needed without changing the optimal objective.
Key takeaway
PBRS provides policy invariance, loop immunity and convergence acceleration. The source reports 5–50× faster convergence in sparse-reward environments.