OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / System Architecture and Infrastructure at Scale / Parallelism Strategies in Detail
Lesson 411 of 415 9 min read
11.2.2

Tensor Parallelism (TP)

Learn how Transformer weight matrices are split across GPUs using column-parallel and row-parallel layers, and how communication affects TP configuration.

What is it?

Tensor Parallelism splits individual weight matrices across multiple GPUs. Each GPU performs part of a matrix operation, and collective communication combines partial results when required. Megatron-LM uses column-parallel and row-parallel linear layers inside Transformer blocks.

Key points

Column parallelism splits W column-wise: Wi has shape d × h/T. Each GPU computes its output independently. Row parallelism splits W row-wise: Wi has shape d/T × h. Inputs are also split, and partial results are combined using AllReduce. Transformer MLP: column-parallel first layer and row-parallel second layer. Attention: Q, K and V projections are column-parallel; output projection is row-parallel. Two AllReduce operations are required per Transformer layer. The source recommends keeping TP within a node, typically TP ≤ 8. The number of attention heads must be divisible by the TP degree.

Splitting a Transformer Weight Matrix

For a column-parallel layer, suppose a weight matrix has shape d × h and is split across four GPUs. Each GPU receives one quarter of the columns: Wi shape = d × h/4 Each GPU calculates Yi = XWi independently. The outputs remain split along the hidden dimension. For a row-parallel layer, the input is split as well. Each GPU computes a partial result, and AllReduce combines the partial sums.

Real-world application

Choose TP based on model size, attention-head divisibility and the communication bandwidth inside the node. The source gives TP=4 as a standard configuration for 34–70B inference and TP=8 for 70B+ training.
Key takeaway

Tensor Parallelism divides the weight matrices themselves. It reduces per-GPU model computation and memory, but introduces frequent communication between GPUs.