/Learn
Modules · Infrastructure · Protocol

NCCL Inter-Node Transport Protocols & Infiniband

LearnProtocolTier 2 · MechanismsUnderstand10 minInfrastructure

You will be able to: identify the transport NCCL chose, and fix a silent fallback to sockets.

Prove it: Async Zero-Redundancy Optimizer Checkpointing

Source: NVIDIA NCCL documentation

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

NCCL moves tensors between GPUs. It bootstraps over TCP, detects the topology, then picks the fastest path: NVLink inside a server and InfiniBand with RDMA between servers. If that detection fails, it falls back to sockets, and training slows with no error.

Why it matters

Reads an NCCL debug log and finds the moment training silently falls back to TCP.

1.2Key concepts

One idea per card, in plain words.

01Bootstrap

Uses TCP sockets to find peers and exchange addresses, before any fast path exists.

02GPUDirect RDMA

Lets the network card read GPU memory directly, skipping the CPU.

03Socket fallback

Sends tensors over TCP through host memory, at a fraction of the speed.

04NCCL_DEBUG

Set to INFO to log the chosen transport and topology.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Moving from 1 to 2 nodes made training 10× slower per step, with no errors. Find out why.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Turn on logging

    Set NCCL_DEBUG=INFO, and restart.

  2. Read the channels

    Look for 'via NET/IB/…/GDRDMA' lines. 'via NET/Socket' means fallback.

  3. Check the interface

    NCCL_SOCKET_IFNAME and NCCL_IB_HCA select the bootstrap interface and the InfiniBand adapters.

  4. Check GPUDirect

    Confirm that the peer-memory driver is loaded, and that no IOMMU blocks it.

  5. Measure

    Run a benchmark such as nccl-tests' all_reduce_perf, and compare against the fabric's rating.

3.2What to look for

Log lineMeaning
via P2P/direct pointerSame process: fastest
via P2P/IPCSame server, over NVLink or PCIe
via NET/IB/GDRDMAAcross servers with RDMA: good
via NET/IB without GDRDMAInfiniBand through host memory: slower
via NET/SocketTCP fallback: fix this

3.3Code

A working implementation, then what each part does and why.

Ring all-reduce

Every GPU must end with the sum of all gradients. A ring keeps each link equally busy, which suits large vectors.

ring-allreduce.tsTypeScript
1/**2 * Ring all-reduce: reduce-scatter, then all-gather, in 2(p - 1) steps.3 * Every rank sends one chunk per step, so each sends 2(p - 1)/p of the vector in total.4 */5export function ringAllReduce(vectors: number[][]): { result: number[][]; sentPerRank: number } {6  const p = vectors.length;7  const n = vectors[0].length;8  const size = Math.ceil(n / p);9  const chunks = vectors.map((v) => Array.from({ length: p }, (_, c) => v.slice(c * size, (c + 1) * size)));10  const mod = (x: number) => ((x % p) + p) % p;11  let sent = 0;12 13  // Reduce-scatter: each chunk travels round the ring, picking up every rank's values.14  for (let step = 0; step < p - 1; step++) {15    const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r - step), data: [...rank[mod(r - step)]] }));16    sent += msgs[0].data.length;17    for (const m of msgs) m.data.forEach((x, i) => (chunks[m.to][m.c][i] += x));18  }19  // Rank r now holds the complete chunk r + 1. All-gather passes the finished chunks on.20  for (let step = 0; step < p - 1; step++) {21    const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r + 1 - step), data: [...rank[mod(r + 1 - step)]] }));22    sent += msgs[0].data.length;23    for (const m of msgs) chunks[m.to][m.c] = m.data;24  }25  return { result: chunks.map((rank) => rank.flat()), sentPerRank: sent };26}
26 linesUTF-8TypeScript
2(p - 1) steps
Reduce-scatter, then all-gather.
chunks[m.to][m.c][i] += x
Each chunk travels the ring, adding each rank's values.
sentPerRank
Each rank sends about twice the vector in total, however many GPUs there are. That's why it scales well.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why no error on fallback?
Sockets still work, just slowly. Only logs and timing reveal it.
Why does a hang point to bootstrap?
If nodes can't reach each other over TCP, collectives never start. Check interfaces and firewalls.
What belongs in the training image?
Matching NCCL, the InfiniBand plugin and drivers. Containers often miss the plugin.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Algorithm choice
NCCL picks ring or tree per message size. NCCL_ALGO overrides it, mainly for debugging.
Counterpoint
Not every workload needs RDMA. Infrequent gradient sharing tolerates TCP better than tensor parallelism does.
AI Engineer · NCCL Inter-Node Transport Protocols & Infiniband