OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / System Architecture and Infrastructure at Scale / The 4-Model Memory Challenge
Lesson 409 of 415 7 min read
11.1.4

Calculating the Total GPU Memory Budget

Calculate the combined memory requirement and estimate the minimum number of 80GB A100 GPUs before parallelism overhead.

What is it?

The source totals the 70B BF16 PPO memory budget at approximately 1,470–1,560 GB. Dividing by 80 GB per A100 gives a naive minimum of 19–20 GPUs before parallelism overhead. The source notes that ZeRO-3 allows the setup to fit in 8 nodes.

Key points

Total estimated memory: 1,470–1,560 GB. A100 GPU capacity in the calculation: 80 GB. Naive minimum: 19–20 A100 GPUs. The estimate excludes parallelism overhead. The source states that ZeRO-3 allows the setup to fit in 8 nodes. Distributed training strategies are needed to manage this memory footprint.

Estimating the Number of A100 GPUs

Using the source estimate: Total memory = 1,470–1,560 GB GPU memory = 80 GB 1,470 ÷ 80 ≈ 18.4 1,560 ÷ 80 = 19.5 The source rounds the naive minimum to 19–20 A100 80GB GPUs, before parallelism overhead. It also notes that ZeRO-3 can fit the setup in 8 nodes.

Real-world application

Use the total memory estimate as an initial capacity calculation, then account for distributed-training overhead, workload configuration and memory optimisation before finalising a cluster design.
Key takeaway

Large-scale PPO requires distributed memory management; counting policy weights alone severely underestimates the hardware requirement.