9.4.3
Potential-Based Reward Shaping in LLMs
Understand how partial-credit signals can provide incremental feedback while preserving optimal-policy guarantees.
What is it?
Potential-Based Reward Shaping (PBRS) provides incremental learning signals using a potential function that estimates partial progress. In the source, it is used to make sparse final-answer rewards more informative for multi-step reasoning.
Key points
Original reward: 1 for a correct final answer and 0 otherwise.
Potential function Φ(s): partial credit from a verifier, such as the fraction of logically valid intermediate steps.
Shaped rewards provide incremental feedback for valid reasoning steps.
The source states that PBRS preserves the guarantee that the optimal policy still maximises final-answer correctness.
Applications include PRMs, intermediate code compilation checks and partial-match scores.
Providing Partial Credit During Reasoning
A code-generation task may receive a binary reward only after the complete program passes its tests. Intermediate compilation checks can provide additional signals when individual stages are valid. This makes feedback less sparse while retaining the source-described optimal-policy guarantee.
Real-world application
Use step-level PRM scores, intermediate compilation checks or partial-match scores to provide denser feedback in multi-step tasks.
Key takeaway
Reward shaping can provide incremental learning signals without changing the optimal policy objective described by PBRS.