/Learn
Modules · Model serving · Protocol

Tensor Parallelism & Ring-AllReduce Communication

LearnProtocolTier 2 · MechanismsUnderstand12 minModel serving

You will be able to: split a linear layer across GPUs both ways, and estimate the communication each step costs.

Prove it: PyTorch Fully Sharded Data Parallel (FSDP)

Source: Shoeybi et al., 2019 (Megatron-LM); Patarasuk & Yuan, 2009

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A 70B model does not fit on one GPU, so tensor parallelism splits each layer's matrices across the GPUs in a server. GPUs work on their share, then exchange results twice per transformer layer. That exchange decides whether scaling pays off.

Why it matters

Every transformer layer pays for two collectives on NVLink, which explains why tensor parallelism stays inside one server.

1.2Key concepts

One idea per card, in plain words.

01Column-parallel

Gives each GPU some columns of the weight matrix. Each computes a slice of the output, and all-gather joins the slices.

02Row-parallel

Gives each GPU some rows and a slice of the input. Partial results must add up, so all-reduce sums them.

03All-reduce

Leaves every GPU holding the element-wise sum of everyone's vector.

04Ring algorithm

Passes chunks around a ring of GPUs: reduce-scatter first, then all-gather. It suits large messages.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Llama 3.1 70B needs about 140 GB for FP16 weights. You have servers with 8 × 24 GB GPUs. How do you split it, what crosses the wire at each layer, and why must all 8 GPUs sit in one server?

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Split by columns

    The first MLP matrix goes column-parallel. Each GPU computes its slice of the output and keeps it local.

  2. Split by rows

    The second matrix goes row-parallel and takes those slices as input, so no communication happens between the two.

  3. Sum at the end

    Partial outputs need adding, so one all-reduce closes the MLP. Attention follows the same shape, giving two all-reduces per transformer layer.

  4. Run the ring

    Reduce-scatter sends chunks round the ring, summing as they go. All-gather then passes finished chunks round. Together they take 2(p − 1) small steps.

  5. Count the bytes

    Each GPU sends 2(p − 1)/p of the vector, so bandwidth use stays flat as p grows. Latency adds up with the step count.

3.2One decode step, bandwidth only

Hidden size 8,192, batch of 64 tokens and FP16 give a 1 MiB vector. Two all-reduces across 80 layers on 8 GPUs send about 280 MiB per GPU.

LinkBandwidthTime for 280 MiB
NVLink (H100, one direction)450 GB/s0.65 ms
InfiniBand NDR20025 GB/s12 ms
100 Gb Ethernet12.5 GB/s23 ms

3.3Code

Working code for the core of it.

Ring all-reduce

Every GPU must end with the sum of all gradients. A ring keeps each link equally busy, which suits large vectors.

ring-allreduce.tsTypeScript
1/**2 * Ring all-reduce: reduce-scatter, then all-gather, in 2(p - 1) steps.3 * Every rank sends one chunk per step, so each sends 2(p - 1)/p of the vector in total.4 */5export function ringAllReduce(vectors: number[][]): { result: number[][]; sentPerRank: number } {6  const p = vectors.length;7  const n = vectors[0].length;8  const size = Math.ceil(n / p);9  const chunks = vectors.map((v) => Array.from({ length: p }, (_, c) => v.slice(c * size, (c + 1) * size)));10  const mod = (x: number) => ((x % p) + p) % p;11  let sent = 0;12 13  // Reduce-scatter: each chunk travels round the ring, picking up every rank's values.14  for (let step = 0; step < p - 1; step++) {15    const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r - step), data: [...rank[mod(r - step)]] }));16    sent += msgs[0].data.length;17    for (const m of msgs) m.data.forEach((x, i) => (chunks[m.to][m.c][i] += x));18  }19  // Rank r now holds the complete chunk r + 1. All-gather passes the finished chunks on.20  for (let step = 0; step < p - 1; step++) {21    const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r + 1 - step), data: [...rank[mod(r + 1 - step)]] }));22    sent += msgs[0].data.length;23    for (const m of msgs) chunks[m.to][m.c] = m.data;24  }25  return { result: chunks.map((rank) => rank.flat()), sentPerRank: sent };26}
26 linesUTF-8TypeScript
2(p - 1) steps
Reduce-scatter, then all-gather.
chunks[m.to][m.c][i] += x
Each chunk travels the ring, adding each rank's values.
sentPerRank
Each rank sends about twice the vector in total, however many GPUs there are. That's why it scales well.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why not pipeline parallelism for serving?
Pipeline parallelism gives each GPU some layers, so a token passes through every stage. It needs less bandwidth but adds latency, so teams often use it across servers and tensor parallelism inside one.
Why a ring and not a tree?
A ring sends small chunks and keeps every link busy, which wins for large vectors. Trees take fewer steps and win for small messages. NCCL picks per message size.
What limits scaling?
Step count grows with GPUs, and the slowest GPU sets the pace of every collective.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Across servers
Bandwidth falls more than tenfold outside the NVLink domain. Keep tensor parallelism inside one server.
Stragglers
One thermally throttled GPU slows the whole group. Monitor per-GPU clocks, not just the average.
Counterpoint
Quantising the model onto fewer GPUs often beats adding tensor parallelism, because fewer GPUs communicate less.
AI Engineer · Tensor Parallelism & Ring-AllReduce Communication