3.11.2
Potential-Based Reward Shaping (PBRS)
Understand how potential differences between states create shaping rewards while preserving the optimal policy.
What is it?
Potential-Based Reward Shaping (PBRS) constrains the shaping function to the difference between a scalar potential function evaluated at the next and current states. This structure mathematically guarantees that reward shaping does not change the optimal policy.
Key points
1. Potential function: Φ : S → R.
2. Φ evaluates the desirable proximity of a state to the goal.
3. γ is the discount factor.
4. Shaping function: F(s,a,s′) = γΦ(s′) − Φ(s).
5. Complete shaped reward: R′(s,a,s′) = R(s,a,s′) + γΦ(s′) − Φ(s).
6. The potential difference supplies intermediate feedback.
7. PBRS constrains the reward modification rather than allowing an arbitrary shaping function.
Example: A robot moving toward a destination
Suppose a navigation robot has a potential function that assigns higher values to states closer to its destination.
Current state potential: Φ(s) = 2
Next state potential: Φ(s′) = 5
Discount factor: γ = 0.9
Original reward: R = 1
Shaping reward:
F = (0.9 × 5) − 2 = 2.5
Complete shaped reward:
R′ = 1 + 2.5 = 3.5
The robot receives an additional signal for moving to a more desirable state.
Real-world application
PBRS can provide intermediate progress feedback in navigation, reasoning and other sparse-reward learning tasks while preserving the optimal policy under the stated conditions.
Key takeaway
PBRS defines shaping rewards through potential differences: γΦ(s′) − Φ(s).