OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / Reward Model Training / Reward Model Training Tricks
Lesson 363 of 415 5 min read
9.3.1

Reward Centering

Centering reward outputs to stabilise reinforcement learning.

What is it?

Reward centering subtracts the expected reward under the current policy from the raw reward. This helps stabilise reinforcement learning when reward model outputs have arbitrary scales and offsets.

Key points

Raw reward outputs can have arbitrary scale and offset. Centering subtracts the expected reward under the current policy. Formula: r_centered(y,q) = r_phi(y,q) - E[r_phi(y',q)]. TRL provides center_rewards_coefficient. The coefficient adds a regularisation term that penalises non-zero mean rewards.

Centering Reward Scores

Suppose a reward model produces scores with a consistently high positive offset. Reward centering adjusts each score by subtracting the expected reward under the current policy. The resulting values are centred around the policy expectation rather than retaining the arbitrary offset.

Real-world application

Use reward centering when raw reward outputs have arbitrary offsets that may destabilise RL training. In TRL, configure center_rewards_coefficient to apply the centering regularisation.
Key takeaway

Reward centering helps make reward signals more stable by removing their expected baseline.