OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / Motivation
Lesson 257 of 415 4 min read
7.1.4

Critic Training and Early RL Instability

Learn how noisy early value predictions can destabilize policy learning.

What is it?

The value head needs many samples to converge. During early reinforcement learning, its predictions can be noisy, which can destabilize policy learning.

Key points

The value head requires many training samples. Convergence takes time. Early RL predictions can be noisy. Noisy value predictions can destabilize policy learning.

Why Early RL Training Can Be Unstable

At the beginning of reinforcement learning, the critic may not yet have enough experience to estimate expected rewards accurately. Its noisy predictions can produce unreliable advantages and unstable policy updates.

Real-world application

Monitor value estimates and advantage stability during early PPO training. Recognise that critic convergence is an additional training challenge.
Key takeaway

A critic that has not converged can introduce noisy signals into early policy learning.