OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / GRPO - Group Relative Policy Optimization / Motivation
Lesson 255 of 415 4 min read
7.1.2

The Memory Cost of the Critic

Explore the memory overhead of a value head sharing the policy backbone or using a separate model.

What is it?

A value head adds a substantial memory requirement to PPO training. The source notes that a value head sharing the policy backbone requires 140GB for a 70B model, while a separate model doubles memory usage.

Key points

The value head shares the policy backbone in the described setup. The source gives 140GB for a 70B model. Using a separate model doubles memory requirements. GRPO removes the need for this separate value network.

Memory Requirements for a 70B Model

For a 70B model, the source cites 140GB for the value head sharing the policy backbone. If the value model is separate, memory requirements double. Removing the critic can therefore make a meaningful difference to the hardware needed for training.

Real-world application

Consider critic memory when estimating the hardware requirements of PPO training. GRPO is an alternative when removing the value network is important.
Key takeaway

Removing the critic reduces the memory burden associated with PPO-style value estimation.