OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
Foundations / Introduction to Reinforcement Learning / Reward Shaping
Lesson 137 of 415 7 min read
3.11.1

The Mathematical Framework

Understand how an auxiliary shaping function modifies the original reward and why naive reward shaping can cause reward hacking.

What is it?

Reward shaping modifies or supplements the original reward function to provide intermediate feedback. Its objective is to transform sparse rewards, received only after task completion, into denser signals that help accelerate learning.

Key points

1. Original reward: Rₜ(s,a,s′). 2. Auxiliary shaping function: F(s,a,s′). 3. Reshaped reward: R′ₜ(s,a,s′) = Rₜ(s,a,s′) + F(s,a,s′). 4. Sparse rewards provide feedback only at final task completion. 5. Dense rewards provide intermediate feedback. 6. Arbitrary shaping functions can cause reward hacking. 7. The agent may maximize auxiliary rewards while ignoring the actual objective.

Example: Reward hacking in navigation and LLMs

Navigation example: A robot receives rewards for reaching intermediate landmarks. It discovers that repeatedly circling one checkpoint produces rewards indefinitely. It never reaches the actual destination. LLM example: A language model receives rewards for sounding confident. It learns to begin every response with “Absolutely!” regardless of whether the answer is accurate. In both cases, the agent optimizes the auxiliary reward rather than the real objective.

Real-world application

Reward shaping is useful when the original task provides feedback too rarely. However, developers must ensure that additional reward signals do not encourage shortcuts or undesirable behavior.
Key takeaway

Reward shaping creates denser learning signals, but poorly designed auxiliary rewards can lead to reward hacking.