Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
FSDP splits each wrapped unit's weights across ranks. Just before a unit runs, ranks gather its weights; afterwards they discard them. Wrapping per transformer block keeps the gather small and the memory low.
PyTorch's native sharding: wrap per block, and each block gathers its weights only while it runs.
1.2Key concepts
One idea per card, in plain words.
Decides what counts as a unit. Per block works best for transformers.
Rebuilds a unit's weights for the forward pass, then discards them.
Sums gradients and leaves each rank its own share.
Each rank saves its shard, avoiding one huge file on rank 0.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Convert a single-GPU training loop for a transformer to FSDP on 8 GPUs. Show the wrapping, one training step, and what changes in the loop.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Choose the block class
Pass your transformer block class to the auto-wrap policy.
- Wrap before the optimiser
Build the optimiser after wrapping, or it holds the wrong parameters.
- Keep the loop plain
Forward, backward and step look as before; collectives happen inside the wrapper.
- Clip with FSDP's method
Gradients live across ranks, so use model.clip_grad_norm_.
- Save by shard
Save per rank or use torch.distributed.checkpoint, not a rank-0 state dict.
3.2Code
Working code for the core of it.
Sharded training with FSDP
PyTorch's native alternative to ZeRO stage 3.
1import functools2 3import torch4from torch.distributed.fsdp import FullyShardedDataParallel as FSDP5from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy6 7 8def wrap(model: torch.nn.Module, block_cls: type) -> FSDP:9 # Shard per transformer block: each block gathers its weights only while it runs.10 policy = functools.partial(transformer_auto_wrap_policy, transformer_layer_cls={block_cls})11 return FSDP(model, auto_wrap_policy=policy, device_id=torch.cuda.current_device())12 13 14def train_step(model: FSDP, batch: dict, optimizer: torch.optim.Optimizer) -> float:15 loss = model(**batch).loss16 loss.backward()17 model.clip_grad_norm_(1.0) # FSDP's own version: gradients live on different ranks18 optimizer.step()19 optimizer.zero_grad(set_to_none=True)20 return loss.item()transformer_auto_wrap_policy- Shards per transformer block, so a block gathers its weights only while it runs.
model.clip_grad_norm_(1.0)- Use FSDP's version. Gradients live on different ranks, so the usual helper gets the norm wrong.
Syntax-checked. The GPU libraries it needs don't run in our tests.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why wrap per block rather than the whole model?
Why build the optimiser after wrapping?
FSDP or DeepSpeed?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- New API
- FSDP2's fully_shard uses per-parameter sharding and simplifies state dicts. Prefer it on recent PyTorch.
- Counterpoint
- If the model fits one GPU, plain DDP stays simpler and faster.