OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / System Architecture and Infrastructure at Scale / Parallelism Strategies in Detail
Lesson 413 of 415 9 min read
11.2.4

Pipeline Parallelism (PP)

Learn how Transformer layers are divided into pipeline stages, why pipeline bubbles occur, and how GPipe, 1F1B and other schedules manage execution.

What is it?

Pipeline Parallelism divides a model vertically by assigning consecutive groups of Transformer layers to different devices. Activations move forward through pipeline stages, while gradients move backward. Scheduling is needed to reduce idle pipeline time, known as bubbles.

Key points

The model is divided into P pipeline stages. Micro-batches keep stages working on different portions of a batch. Bubble fraction = (P − 1)/(P + M − 1), where M is the number of micro-batches. For bubble overhead below 10%, the source recommends M ≥ 10 × (P − 1). GPipe performs all forward passes before backward passes and stores activations for all micro-batches. 1F1B alternates forward and backward work and bounds steady-state activation memory to P micro-batches. Pipeline communication uses point-to-point activation transfers between adjacent stages. Stage workloads should be balanced across devices.

Understanding the 1F1B Pipeline Schedule

Consider a pipeline with four stages and multiple micro-batches. Warmup: Forward passes fill the pipeline across the stages. Steady state: Each stage alternates one forward pass and one backward pass (1F1B). Cooldown: Remaining backward passes drain the pipeline. Unlike GPipe, which stores activations for all M micro-batches, 1F1B limits steady-state activation storage to P sets. This is useful when the number of micro-batches is much larger than the number of stages.

Real-world application

Use pipeline scheduling when a model is split across devices by layers. Choose enough micro-batches to reduce idle time and balance the number of computationally demanding layers assigned to each stage.
Key takeaway

Pipeline Parallelism distributes layers across devices. Micro-batch scheduling reduces idle bubbles and controls activation memory.