OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / System Architecture and Infrastructure at Scale / Parallelism Strategies in Detail
Lesson 412 of 415 7 min read
11.2.3

Sequence Parallelism (SP)

Understand how sequence parallelism reduces activation memory in LayerNorm, Dropout and residual operations by distributing sequence slices across GPUs.

What is it?

Sequence Parallelism reduces activation memory that remains replicated when Tensor Parallelism is used. It divides the sequence dimension for operations such as LayerNorm, Dropout and residual connections, while gathering the full sequence where required.

Key points

Tensor Parallelism splits weights but does not automatically reduce all activation memory. LayerNorm and Dropout activations can remain replicated across GPUs. Sequence Parallelism assigns each GPU a 1/T slice of the sequence for suitable operations. ReduceScatter distributes sequence slices; AllGather reconstructs the full sequence when needed. The source describes the total communication volume as identical to standard TP. It recommends enabling SP whenever TP is used. For a 70B model with TP=8, batch=4 and sequence length 2048, the source estimates approximately 59 GB of activation savings per GPU.

Reducing Activation Memory with Sequence Parallelism

Consider a sequence divided across eight GPUs. Instead of every GPU retaining the full sequence activation for LayerNorm and Dropout, each GPU processes one eighth of the sequence. ReduceScatter distributes the sequence slices. After the operations that can be performed locally, AllGather reconstructs the full sequence for the next operations that require it. The source estimates approximately 59 GB per GPU saved for its 70B, TP=8 example.

Real-world application

When using Tensor Parallelism, enable Sequence Parallelism to reduce activation memory without increasing the total communication volume described by the source.
Key takeaway

Sequence Parallelism complements Tensor Parallelism by reducing replicated activation memory through sequence slicing.