11.2.4
Pipeline Parallelism (PP)
Learn how Transformer layers are divided into pipeline stages, why pipeline bubbles occur, and how GPipe, 1F1B and other schedules manage execution.
What is it?
Pipeline Parallelism divides a model vertically by assigning consecutive groups of Transformer layers to different devices. Activations move forward through pipeline stages, while gradients move backward. Scheduling is needed to reduce idle pipeline time, known as bubbles.
Key points
The model is divided into P pipeline stages.
Micro-batches keep stages working on different portions of a batch.
Bubble fraction = (P − 1)/(P + M − 1), where M is the number of micro-batches.
For bubble overhead below 10%, the source recommends M ≥ 10 × (P − 1).
GPipe performs all forward passes before backward passes and stores activations for all micro-batches.
1F1B alternates forward and backward work and bounds steady-state activation memory to P micro-batches.
Pipeline communication uses point-to-point activation transfers between adjacent stages.
Stage workloads should be balanced across devices.
Understanding the 1F1B Pipeline Schedule
Consider a pipeline with four stages and multiple micro-batches.
Warmup: Forward passes fill the pipeline across the stages.
Steady state: Each stage alternates one forward pass and one backward pass (1F1B).
Cooldown: Remaining backward passes drain the pipeline.
Unlike GPipe, which stores activations for all M micro-batches, 1F1B limits steady-state activation storage to P sets. This is useful when the number of micro-batches is much larger than the number of stages.
Real-world application
Use pipeline scheduling when a model is split across devices by layers. Choose enough micro-batches to reduce idle time and balance the number of computationally demanding layers assigned to each stage.
Key takeaway
Pipeline Parallelism distributes layers across devices. Micro-batch scheduling reduces idle bubbles and controls activation memory.