11.2.6
3D Parallelism: Combining Strategies
Learn how production systems combine tensor, pipeline and data or fully sharded parallelism, and how to select a configuration based on model size and hardware.
What is it?
3D Parallelism combines multiple parallelism dimensions in one distributed training system. Production systems can combine Tensor Parallelism, Pipeline Parallelism and Data Parallelism or FSDP to use different levels of the hardware communication hierarchy.
Key points
TP splits weight matrices, typically within a node using NVLink.
PP splits Transformer layers into stages.
DP or FSDP distributes training across replicas or shards across nodes.
The source gives a 70B example using 64 A100 80GB GPUs across eight nodes.
Its recipe uses TP=8 for generation within a node and FSDP for training and inter-node scaling.
Pipeline Parallelism is a last resort in the source decision flow, particularly for models above 100B that still do not fit with TP and ZeRO.
Production systems typically combine two or three parallelism strategies.
Selecting a Parallelism Configuration
Follow the source decision flow:
1. Does the model fit on one GPU? Use DDP.
2. Does it fit on one node with FSDP? Use FSDP (ZeRO-3).
3. Does it fit on one node with TP and FSDP? Use TP within the node and FSDP across nodes.
4. Does it still not fit? Add Pipeline Parallelism across nodes as a last resort.
The source’s 70B production recipe uses TP=8 within nodes and FSDP across eight nodes, with approximately 70 GB held per GPU.
Real-world application
Design parallelism according to model size, GPU memory, intra-node bandwidth and inter-node network capacity. Start with the simplest configuration that fits, then add dimensions only when required.
Key takeaway
3D Parallelism combines complementary strategies: TP for weight matrices, PP for layers and DP/FSDP for distributed training and sharding.