7.1.2
The Memory Cost of the Critic
Explore the memory overhead of a value head sharing the policy backbone or using a separate model.
What is it?
A value head adds a substantial memory requirement to PPO training. The source notes that a value head sharing the policy backbone requires 140GB for a 70B model, while a separate model doubles memory usage.
Key points
The value head shares the policy backbone in the described setup.
The source gives 140GB for a 70B model.
Using a separate model doubles memory requirements.
GRPO removes the need for this separate value network.
Memory Requirements for a 70B Model
For a 70B model, the source cites 140GB for the value head sharing the policy backbone. If the value model is separate, memory requirements double. Removing the critic can therefore make a meaningful difference to the hardware needed for training.
Real-world application
Consider critic memory when estimating the hardware requirements of PPO training. GRPO is an alternative when removing the value network is important.
Key takeaway
Removing the critic reduces the memory burden associated with PPO-style value estimation.