OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / System Architecture and Infrastructure at Scale / Parallelism Strategies in Detail
Lesson 410 of 415 8 min read
11.2.1

Data Parallelism (DP) and Distributed Data Parallelism (DDP)

Understand replicated model training, gradient synchronization, ring AllReduce, DDP communication and the limitations of legacy PyTorch DataParallel.

What is it?

Data Parallelism gives every GPU a complete model replica and assigns a different mini-batch to each GPU. Gradients are synchronised after backward computation. Distributed Data Parallelism (DDP) uses a separate process per GPU and ring AllReduce to synchronise gradients while backward computation continues.

Key points

Each GPU holds a complete model replica. Each GPU processes a different mini-batch. Vanilla PyTorch DataParallel uses one process and a master GPU for scattering, gathering and gradient broadcasting. DDP uses one process per GPU. DDP synchronises gradients using ring AllReduce. Gradient buckets (25 MB by default) allow communication to overlap with backward computation. DDP scales near-linearly up to approximately 64 GPUs; communication becomes more significant beyond that. For LLMs above 7B, the source recommends FSDP or ZeRO rather than ordinary DDP.

Understanding DDP Gradient Synchronisation

Imagine four GPUs training the same model: GPU 1 processes mini-batch A. GPU 2 processes mini-batch B. GPU 3 processes mini-batch C. GPU 4 processes mini-batch D. Each GPU calculates gradients locally. Ring AllReduce synchronises and averages the gradients so that every replica applies a consistent update. The source warns against legacy PyTorch DataParallel for LLM training because of the Python GIL, GPU 0 bottleneck, 2–3× slower performance than DDP, and inability to scale beyond one machine.

Real-world application

Use DDP when the complete model and its training state fit on each GPU. For larger models, evaluate FSDP or ZeRO to avoid duplicating all model state on every GPU.
Key takeaway

DDP distributes batches across replicated models and synchronises gradients efficiently. Its main limitation is that every GPU still stores the complete training state.