Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Blocking saves stop every GPU for minutes. Async checkpointing copies state to CPU memory quickly, lets training continue, and writes in the background. A commit marker, written last, separates complete saves from broken ones.
Saves sharded state in the background, so checkpoints stop stalling GPUs.
1.2Key concepts
One idea per card, in plain words.
Copies state before returning, so training can't change it mid-write.
Each rank writes its own shard, so no rank needs the whole model in memory.
Written after all shards. No marker means incomplete.
A new save waits if the last still runs, so saves never pile up.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Saving a sharded 70B model stalls training for four minutes every hour. Build a checkpointer that blocks only for the copy, survives a crash mid-write, and always loads the newest complete save.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Enough CPU memory for a copy? | Yes, but plan for it: one shard per rank. |
| Shared storage? | Yes, with slow, unreliable writes. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Copy the state
Take a snapshot synchronously. This is the only part that blocks training.
- Write in the background
Queue the write, so training continues at once.
- Serialise saves
Chain writes, so at most one runs at a time.
- Commit last
Write the marker after the shards. In multi-rank runs, rank 0 writes it after all ranks confirm.
- Load the newest committed
On restart, ignore any directory without a marker.
- Clean up
Keep the last few checkpoints, and delete older ones only after a newer commit.
3.2Code
Working code for the core of it.
Saving without stopping training
Writing a large checkpoint can take minutes. Training should only pause for the copy.
1export interface Storage {2 write(path: string, data: string): Promise<void>;3 list(): Promise<string[]>;4}5 6/** Copy the state fast, write it in the background, and write the commit marker last. */7export class AsyncCheckpointer<S> {8 private storage: Storage;9 private inFlight: Promise<void> = Promise.resolve();10 11 constructor(storage: Storage) {12 this.storage = storage;13 }14 15 /** Blocks training only for the copy. A second save waits for the first to finish writing. */16 save(step: number, rank: number, state: S): Promise<void> {17 const snapshot = JSON.stringify(state);18 this.inFlight = this.inFlight.then(async () => {19 await this.storage.write(`ckpt-${step}/rank-${rank}.json`, snapshot);20 await this.storage.write(`ckpt-${step}/COMMIT`, "ok"); // absent means incomplete21 });22 return this.inFlight;23 }24 25 /** The newest checkpoint with a commit marker. Half-written ones are ignored. */26 async latest(): Promise<number | null> {27 const files = await this.storage.list();28 const steps = files.filter((f) => f.endsWith("/COMMIT")).map((f) => Number(f.match(/ckpt-(\d+)/)?.[1]));29 return steps.length ? Math.max(...steps) : null;30 }31}JSON.stringify(state)- The copy. It stands for moving tensors to host memory, and this is the only part that blocks.
this.inFlight.then(...)- Writes one at a time, so a slow disk can't pile up saves.
COMMIT- Written last. A checkpoint without it is half-written and gets ignored on restart.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why not write straight from GPU memory?
What if a save fails?
How do you test it?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Resharding
- Restarting with a different GPU count needs a format that records tensor layout. torch.distributed.checkpoint stores this.
- Counterpoint
- Small models save in seconds. Async adds failure modes that a short blocking save doesn't have.