Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
SLURM queues jobs, assigns whole nodes and starts your processes. Long training runs hit time limits and bad nodes, so a good job plans for interruption: save on a warning signal, requeue itself, and resume automatically.
Requeues multi-node training on failure, and resumes from the last checkpoint with no one watching.
1.2Key concepts
One idea per card, in plain words.
Partitions group nodes; QoS sets limits and priority.
Requests generic resources such as GPUs: --gres=gpu:8 per node.
Sends your script a warning before the time limit, so it can save.
Lets the job return to the queue after a failure or preemption.
A program SLURM runs on each node. A failing GPU drains the node before it wastes jobs.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
A 4-node training job dies at SLURM's 24-hour limit and sometimes on a bad GPU. Write a job that resumes on its own, and explain each step.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Request resources
Ask for nodes, GPUs per node and CPUs. One task per node, and torchrun starts the rest.
- Find the rendezvous
Use the first hostname as the master address.
- Catch the warning
--signal=B:USR1@300 tells the batch script five minutes before the limit.
- Save and requeue
The trap requeues the job and asks training to stop, which saves on SIGTERM.
- Resume automatically
train.py finds the newest complete checkpoint and continues from there.
- Drain bad nodes
A node health check marks failing GPUs, so the requeued job avoids them.
3.2Code
Working code for the core of it.
A training job that survives preemption
Long jobs outlive their time limit. This one saves and goes back in the queue.
1#!/bin/bash2#SBATCH --job-name=train-70b3#SBATCH --nodes=44#SBATCH --ntasks-per-node=15#SBATCH --gres=gpu:86#SBATCH --cpus-per-task=647#SBATCH --time=24:00:008#SBATCH --requeue9#SBATCH --signal=B:USR1@300 # warn this script 5 minutes before the time limit10#SBATCH --output=logs/%x-%j.out11 12# On the warning: stop cleanly (training saves on SIGTERM), then go back in the queue.13trap 'scontrol requeue "$SLURM_JOB_ID"; kill -TERM "$TRAIN_PID"' USR114 15export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)16export MASTER_PORT=2950017 18srun torchrun --nnodes="$SLURM_NNODES" --nproc-per-node=8 \19 --rdzv-backend=c10d --rdzv-endpoint="$MASTER_ADDR:$MASTER_PORT" \20 train.py --resume-from-latest &21TRAIN_PID=$!22wait "$TRAIN_PID"--signal=B:USR1@300- Warns the script 5 minutes before the time limit.
trap 'scontrol requeue ...'- On the warning, it requeues, then stops training cleanly.
--resume-from-latest- The next run continues from the last complete checkpoint.
Syntax-checked only.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why the B: in --signal?
What if a node dies with no warning?
SLURM or Kubernetes?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Partial checkpoints
- A kill during a save must never look complete. Write a commit marker last.
- Counterpoint
- For small teams, a managed training service costs more per hour and saves a cluster administrator's salary.