9.3.1
Reward Centering
Centering reward outputs to stabilise reinforcement learning.
What is it?
Reward centering subtracts the expected reward under the current policy from the raw reward. This helps stabilise reinforcement learning when reward model outputs have arbitrary scales and offsets.
Key points
Raw reward outputs can have arbitrary scale and offset.
Centering subtracts the expected reward under the current policy.
Formula: r_centered(y,q) = r_phi(y,q) - E[r_phi(y',q)].
TRL provides center_rewards_coefficient.
The coefficient adds a regularisation term that penalises non-zero mean rewards.
Centering Reward Scores
Suppose a reward model produces scores with a consistently high positive offset. Reward centering adjusts each score by subtracting the expected reward under the current policy. The resulting values are centred around the policy expectation rather than retaining the arbitrary offset.
Real-world application
Use reward centering when raw reward outputs have arbitrary offsets that may destabilise RL training. In TRL, configure center_rewards_coefficient to apply the centering regularisation.
Key takeaway
Reward centering helps make reward signals more stable by removing their expected baseline.