Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
NCCL moves tensors between GPUs. It bootstraps over TCP, detects the topology, then picks the fastest path: NVLink inside a server and InfiniBand with RDMA between servers. If that detection fails, it falls back to sockets, and training slows with no error.
Reads an NCCL debug log and finds the moment training silently falls back to TCP.
1.2Key concepts
One idea per card, in plain words.
Uses TCP sockets to find peers and exchange addresses, before any fast path exists.
Lets the network card read GPU memory directly, skipping the CPU.
Sends tensors over TCP through host memory, at a fraction of the speed.
Set to INFO to log the chosen transport and topology.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Moving from 1 to 2 nodes made training 10× slower per step, with no errors. Find out why.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Turn on logging
Set NCCL_DEBUG=INFO, and restart.
- Read the channels
Look for 'via NET/IB/…/GDRDMA' lines. 'via NET/Socket' means fallback.
- Check the interface
NCCL_SOCKET_IFNAME and NCCL_IB_HCA select the bootstrap interface and the InfiniBand adapters.
- Check GPUDirect
Confirm that the peer-memory driver is loaded, and that no IOMMU blocks it.
- Measure
Run a benchmark such as nccl-tests' all_reduce_perf, and compare against the fabric's rating.
3.2What to look for
| Log line | Meaning |
|---|---|
| via P2P/direct pointer | Same process: fastest |
| via P2P/IPC | Same server, over NVLink or PCIe |
| via NET/IB/GDRDMA | Across servers with RDMA: good |
| via NET/IB without GDRDMA | InfiniBand through host memory: slower |
| via NET/Socket | TCP fallback: fix this |
3.3Code
A working implementation, then what each part does and why.
Ring all-reduce
Every GPU must end with the sum of all gradients. A ring keeps each link equally busy, which suits large vectors.
1/**2 * Ring all-reduce: reduce-scatter, then all-gather, in 2(p - 1) steps.3 * Every rank sends one chunk per step, so each sends 2(p - 1)/p of the vector in total.4 */5export function ringAllReduce(vectors: number[][]): { result: number[][]; sentPerRank: number } {6 const p = vectors.length;7 const n = vectors[0].length;8 const size = Math.ceil(n / p);9 const chunks = vectors.map((v) => Array.from({ length: p }, (_, c) => v.slice(c * size, (c + 1) * size)));10 const mod = (x: number) => ((x % p) + p) % p;11 let sent = 0;12 13 // Reduce-scatter: each chunk travels round the ring, picking up every rank's values.14 for (let step = 0; step < p - 1; step++) {15 const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r - step), data: [...rank[mod(r - step)]] }));16 sent += msgs[0].data.length;17 for (const m of msgs) m.data.forEach((x, i) => (chunks[m.to][m.c][i] += x));18 }19 // Rank r now holds the complete chunk r + 1. All-gather passes the finished chunks on.20 for (let step = 0; step < p - 1; step++) {21 const msgs = chunks.map((rank, r) => ({ to: mod(r + 1), c: mod(r + 1 - step), data: [...rank[mod(r + 1 - step)]] }));22 sent += msgs[0].data.length;23 for (const m of msgs) chunks[m.to][m.c] = m.data;24 }25 return { result: chunks.map((rank) => rank.flat()), sentPerRank: sent };26}2(p - 1) steps- Reduce-scatter, then all-gather.
chunks[m.to][m.c][i] += x- Each chunk travels the ring, adding each rank's values.
sentPerRank- Each rank sends about twice the vector in total, however many GPUs there are. That's why it scales well.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why no error on fallback?
Why does a hang point to bootstrap?
What belongs in the training image?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Algorithm choice
- NCCL picks ring or tree per message size. NCCL_ALGO overrides it, mainly for debugging.
- Counterpoint
- Not every workload needs RDMA. Infrequent gradient sharing tolerates TCP better than tensor parallelism does.