7.1.4
Critic Training and Early RL Instability
Learn how noisy early value predictions can destabilize policy learning.
What is it?
The value head needs many samples to converge. During early reinforcement learning, its predictions can be noisy, which can destabilize policy learning.
Key points
The value head requires many training samples.
Convergence takes time.
Early RL predictions can be noisy.
Noisy value predictions can destabilize policy learning.
Why Early RL Training Can Be Unstable
At the beginning of reinforcement learning, the critic may not yet have enough experience to estimate expected rewards accurately. Its noisy predictions can produce unreliable advantages and unstable policy updates.
Real-world application
Monitor value estimates and advantage stability during early PPO training. Recognise that critic convergence is an additional training challenge.
Key takeaway
A critic that has not converged can introduce noisy signals into early policy learning.