11.1.3
Reference Model, Reward Model and Generation Memory
Understand the additional memory consumed by the reference model, reward model, activations and generation KV cache.
What is it?
The policy is not the only memory consumer. The reference model and reward model each add memory, while activations and the generation KV cache add further requirements. The source lists 140 GB each for reference and reward models, with INT8 alternatives of 70 GB each.
Key points
Reference model: 140 GB, or 70 GB in INT8.
Reward model: 140 GB, or 70 GB in INT8.
Activations for batch 128 and sequence length 2048: 50–100 GB.
Generation KV cache: 20–60 GB.
Memory use depends on model precision, batch size, sequence length and generation requirements.
Additional Memory Beyond the Policy
Additional components in the source memory budget:
Reference model: 140 GB
Reward model: 140 GB
Activations: 50–100 GB
Generation KV cache: 20–60 GB
The reference and reward models can each be reduced to 70 GB in the source’s INT8 estimate.
Real-world application
Include reference and reward model memory when planning an RLHF cluster. Also reserve capacity for activations and generation KV cache based on the intended training workload.
Key takeaway
A realistic RLHF memory estimate must include supporting models, activations and generation cache.