11.1.2
Policy Model Memory Requirements
Break down policy weights, FP32 master weights, Adam optimizer states and gradients for a 70B BF16 model.
What is it?
The policy model itself requires memory for its BF16 weights, FP32 master weights, Adam optimiser states and gradients. For a 70B BF16 model, the source lists 140 GB for policy weights, 280 GB for FP32 master weights, 560 GB for Adam states and 140 GB for gradients.
Key points
Policy weights (BF16): 140 GB.
FP32 master weights: 280 GB.
Adam optimiser states (m + v, FP32): 560 GB.
Gradients (BF16): 140 GB.
These components are part of the policy training memory budget.
Policy Training Memory for a 70B Model
Policy memory components from the source:
BF16 policy weights = 140 GB
FP32 master weights = 280 GB
Adam states = 560 GB
BF16 gradients = 140 GB
Total policy training state = 1,120 GB.
Real-world application
Use the component-level breakdown when estimating memory for policy training. Optimiser states and master weights can consume substantially more memory than the model weights alone.
Key takeaway
The policy model’s training state is much larger than its BF16 weights alone.