11.2.2
Tensor Parallelism (TP)
Learn how Transformer weight matrices are split across GPUs using column-parallel and row-parallel layers, and how communication affects TP configuration.
What is it?
Tensor Parallelism splits individual weight matrices across multiple GPUs. Each GPU performs part of a matrix operation, and collective communication combines partial results when required. Megatron-LM uses column-parallel and row-parallel linear layers inside Transformer blocks.
Key points
Column parallelism splits W column-wise: Wi has shape d × h/T. Each GPU computes its output independently.
Row parallelism splits W row-wise: Wi has shape d/T × h. Inputs are also split, and partial results are combined using AllReduce.
Transformer MLP: column-parallel first layer and row-parallel second layer.
Attention: Q, K and V projections are column-parallel; output projection is row-parallel.
Two AllReduce operations are required per Transformer layer.
The source recommends keeping TP within a node, typically TP ≤ 8.
The number of attention heads must be divisible by the TP degree.
Splitting a Transformer Weight Matrix
For a column-parallel layer, suppose a weight matrix has shape d × h and is split across four GPUs.
Each GPU receives one quarter of the columns:
Wi shape = d × h/4
Each GPU calculates Yi = XWi independently. The outputs remain split along the hidden dimension.
For a row-parallel layer, the input is split as well. Each GPU computes a partial result, and AllReduce combines the partial sums.
Real-world application
Choose TP based on model size, attention-head divisibility and the communication bandwidth inside the node. The source gives TP=4 as a standard configuration for 34–70B inference and TP=8 for 70B+ training.
Key takeaway
Tensor Parallelism divides the weight matrices themselves. It reduces per-GPU model computation and memory, but introduces frequent communication between GPUs.