/Learn
Modules · Fine-tuning · Machine coding

PyTorch Fully Sharded Data Parallel (FSDP)

PracticeMachine codingTier 1 · Whole pictureCreate35 minFine-tuning

You will be able to: wrap a transformer in FSDP, run a training step, and save a sharded checkpoint.

Source: Zhao et al., 2023 (PyTorch FSDP, VLDB)

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

FSDP splits each wrapped unit's weights across ranks. Just before a unit runs, ranks gather its weights; afterwards they discard them. Wrapping per transformer block keeps the gather small and the memory low.

Why it matters

PyTorch's native sharding: wrap per block, and each block gathers its weights only while it runs.

1.2Key concepts

One idea per card, in plain words.

01Wrapping policy

Decides what counts as a unit. Per block works best for transformers.

02All-gather and free

Rebuilds a unit's weights for the forward pass, then discards them.

03Reduce-scatter

Sums gradients and leaves each rank its own share.

04Sharded checkpoint

Each rank saves its shard, avoiding one huge file on rank 0.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Convert a single-GPU training loop for a transformer to FSDP on 8 GPUs. Show the wrapping, one training step, and what changes in the loop.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Choose the block class

    Pass your transformer block class to the auto-wrap policy.

  2. Wrap before the optimiser

    Build the optimiser after wrapping, or it holds the wrong parameters.

  3. Keep the loop plain

    Forward, backward and step look as before; collectives happen inside the wrapper.

  4. Clip with FSDP's method

    Gradients live across ranks, so use model.clip_grad_norm_.

  5. Save by shard

    Save per rank or use torch.distributed.checkpoint, not a rank-0 state dict.

3.2Code

Working code for the core of it.

Sharded training with FSDP

PyTorch's native alternative to ZeRO stage 3.

fsdp_train.pyPython
1import functools2 3import torch4from torch.distributed.fsdp import FullyShardedDataParallel as FSDP5from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy6 7 8def wrap(model: torch.nn.Module, block_cls: type) -> FSDP:9    # Shard per transformer block: each block gathers its weights only while it runs.10    policy = functools.partial(transformer_auto_wrap_policy, transformer_layer_cls={block_cls})11    return FSDP(model, auto_wrap_policy=policy, device_id=torch.cuda.current_device())12 13 14def train_step(model: FSDP, batch: dict, optimizer: torch.optim.Optimizer) -> float:15    loss = model(**batch).loss16    loss.backward()17    model.clip_grad_norm_(1.0)     # FSDP's own version: gradients live on different ranks18    optimizer.step()19    optimizer.zero_grad(set_to_none=True)20    return loss.item()
20 linesUTF-8Python
transformer_auto_wrap_policy
Shards per transformer block, so a block gathers its weights only while it runs.
model.clip_grad_norm_(1.0)
Use FSDP's version. Gradients live on different ranks, so the usual helper gets the norm wrong.

Syntax-checked. The GPU libraries it needs don't run in our tests.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why wrap per block rather than the whole model?
One wrapper would gather all weights at once, which defeats the purpose.
Why build the optimiser after wrapping?
Wrapping replaces parameters with their sharded versions.
FSDP or DeepSpeed?
FSDP stays native to PyTorch. DeepSpeed offers more features and offload options. Choose by your team's stack.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
New API
FSDP2's fully_shard uses per-parameter sharding and simplifies state dicts. Prefer it on recent PyTorch.
Counterpoint
If the model fits one GPU, plain DDP stays simpler and faster.
AI Engineer · PyTorch Fully Sharded Data Parallel (FSDP)