OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Reward Shaping
Lesson 138 of 415 8 min read
3.11.2

Potential-Based Reward Shaping (PBRS)

Understand how potential differences between states create shaping rewards while preserving the optimal policy.

What is it?

Potential-Based Reward Shaping (PBRS) constrains the shaping function to the difference between a scalar potential function evaluated at the next and current states. This structure mathematically guarantees that reward shaping does not change the optimal policy.

Key points

1. Potential function: Φ : S → R. 2. Φ evaluates the desirable proximity of a state to the goal. 3. γ is the discount factor. 4. Shaping function: F(s,a,s′) = γΦ(s′) − Φ(s). 5. Complete shaped reward: R′(s,a,s′) = R(s,a,s′) + γΦ(s′) − Φ(s). 6. The potential difference supplies intermediate feedback. 7. PBRS constrains the reward modification rather than allowing an arbitrary shaping function.

Example: A robot moving toward a destination

Suppose a navigation robot has a potential function that assigns higher values to states closer to its destination. Current state potential: Φ(s) = 2 Next state potential: Φ(s′) = 5 Discount factor: γ = 0.9 Original reward: R = 1 Shaping reward: F = (0.9 × 5) − 2 = 2.5 Complete shaped reward: R′ = 1 + 2.5 = 3.5 The robot receives an additional signal for moving to a more desirable state.

Real-world application

PBRS can provide intermediate progress feedback in navigation, reasoning and other sparse-reward learning tasks while preserving the optimal policy under the stated conditions.
Key takeaway

PBRS defines shaping rewards through potential differences: γΦ(s′) − Φ(s).