11.2.5
Fully Sharded Data Parallelism (FSDP / ZeRO-3)
Understand how FSDP and ZeRO-3 shard parameters, gradients and optimizer states across GPUs, including their execution flow and memory-communication trade-off.
What is it?
Fully Sharded Data Parallelism (FSDP) and DeepSpeed ZeRO-3 reduce DDP memory duplication by distributing parameters, gradients and optimizer states across GPUs. Each GPU owns a shard and reconstructs full parameters temporarily when computation requires them.
Key points
Each GPU owns approximately 1/N of the model state.
Forward: AllGather parameters, compute, then discard non-owned parameter shards.
Backward: AllGather parameters again, compute gradients, then ReduceScatter gradient shards.
Optimizer step: Each GPU updates its own parameter and optimizer-state shard.
The source’s 70B model comparison across eight GPUs estimates 1,120 GB/GPU for unsharded DDP, 385 GB for ZeRO-1, 368 GB for ZeRO-2 and 140 GB for ZeRO-3/FSDP.
FSDP communicates more data than DDP: two AllGather operations and one ReduceScatter per step in the source comparison.
FSDP is useful when the model does not fit with DDP or communication can overlap with computation.
How ZeRO-3 Shards Model State
Imagine eight GPUs sharing a model’s training state.
Each GPU stores one eighth of the parameters, gradients and optimizer states.
Before a layer executes, the required parameters are gathered. After computation, each GPU discards the parameter portions it does not own. During backward computation, gradients are reduced and scattered so each GPU receives its own gradient shard.
The optimizer then updates only the locally owned shard.
Real-world application
Consider FSDP or ZeRO-3 when model state cannot fit on a GPU using ordinary DDP. Configure mixed precision and layer wrapping carefully, and account for the additional communication.
Key takeaway
FSDP and ZeRO-3 trade increased communication for substantial reductions in per-GPU model-state memory.