11.1.4
Calculating the Total GPU Memory Budget
Calculate the combined memory requirement and estimate the minimum number of 80GB A100 GPUs before parallelism overhead.
What is it?
The source totals the 70B BF16 PPO memory budget at approximately 1,470–1,560 GB. Dividing by 80 GB per A100 gives a naive minimum of 19–20 GPUs before parallelism overhead. The source notes that ZeRO-3 allows the setup to fit in 8 nodes.
Key points
Total estimated memory: 1,470–1,560 GB.
A100 GPU capacity in the calculation: 80 GB.
Naive minimum: 19–20 A100 GPUs.
The estimate excludes parallelism overhead.
The source states that ZeRO-3 allows the setup to fit in 8 nodes.
Distributed training strategies are needed to manage this memory footprint.
Estimating the Number of A100 GPUs
Using the source estimate:
Total memory = 1,470–1,560 GB
GPU memory = 80 GB
1,470 ÷ 80 ≈ 18.4
1,560 ÷ 80 = 19.5
The source rounds the naive minimum to 19–20 A100 80GB GPUs, before parallelism overhead. It also notes that ZeRO-3 can fit the setup in 8 nodes.
Real-world application
Use the total memory estimate as an initial capacity calculation, then account for distributed-training overhead, workload configuration and memory optimisation before finalising a cluster design.
Key takeaway
Large-scale PPO requires distributed memory management; counting policy weights alone severely underestimates the hardware requirement.