OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / System Architecture and Infrastructure at Scale / The 4-Model Memory Challenge
Lesson 408 of 415 7 min read
11.1.3

Reference Model, Reward Model and Generation Memory

Understand the additional memory consumed by the reference model, reward model, activations and generation KV cache.

What is it?

The policy is not the only memory consumer. The reference model and reward model each add memory, while activations and the generation KV cache add further requirements. The source lists 140 GB each for reference and reward models, with INT8 alternatives of 70 GB each.

Key points

Reference model: 140 GB, or 70 GB in INT8. Reward model: 140 GB, or 70 GB in INT8. Activations for batch 128 and sequence length 2048: 50–100 GB. Generation KV cache: 20–60 GB. Memory use depends on model precision, batch size, sequence length and generation requirements.

Additional Memory Beyond the Policy

Additional components in the source memory budget: Reference model: 140 GB Reward model: 140 GB Activations: 50–100 GB Generation KV cache: 20–60 GB The reference and reward models can each be reduced to 70 GB in the source’s INT8 estimate.

Real-world application

Include reference and reward model memory when planning an RLHF cluster. Also reserve capacity for activations and generation KV cache based on the intended training workload.
Key takeaway

A realistic RLHF memory estimate must include supporting models, activations and generation cache.