Skip to main content

Node failures and checkpoints

What happens when a compute node fails​

TimeWhat happens
About 2 to 5 minutesThe compute node shows * in sinfo.
About 7 minutesSlurm marks the compute node down. Slurm requeues the job.
About 3 minutes laterThe job starts again from the beginning on a free compute node, if there is one.
Warning:

Without checkpoints, a job loses all its work when its compute node fails. Save checkpoints to /home/jl_fs often. When your program starts, make it load the newest checkpoint.

Multi-node jobs​

A failure of one compute node fails the whole job. Slurm can requeue the job, or not, depending on how the compute node failed.

After any failure, run squeue and sacct. Submit the job again if it is gone.

Two copies of a job can run​

A compute node can be alive when Slurm cannot reach it. The old copy of the job then continues while the new copy starts. If both copies write to one file, the file can lose data. A checkpoint can get damage in the same way.

To prevent this damage, do one of these:

  • Put $SLURM_JOB_ID and ${SLURM_RESTART_COUNT:-0} in your checkpoint file names, and do not append to one file.
  • Turn requeue off for the job.
#SBATCH --no-requeue

With --no-requeue, the job ends NODE_FAIL when its compute node fails.

Get a warning before a time limit​

Slurm can send your job a signal before the time limit. Your program catches USR1, saves and exits. This script runs on 1 GPU, in a container, with resume.

#!/bin/bash
#SBATCH --gres=gpu:1 -c 8 --mem=64G --time=4 -J requeue -o %x-%j.out
#SBATCH --requeue --open-mode=append --signal=B:USR1@90
on_usr1() {
  kill -USR1 $PID   # tell the program to save
  wait $PID
  scontrol requeue $SLURM_JOB_ID
  exit 0
}
trap on_usr1 USR1
srun --container-image=$HOME/ptib.sqsh --container-workdir=$PWD python train.py &
PID=$!
wait $PID

These points are important:

  • Keep & and wait. With the program in the foreground, the handler does not run, and the job ends TIMEOUT with nothing saved.
  • The signal can come early or late. Save often, and leave a wide margin.
  • --requeue repeats this until the work finishes. A program that never finishes continues until you scancel it.
  • SLURM_RESTART_COUNT tells you which run this is. It is not set on the first run, so use ${SLURM_RESTART_COUNT:-0}.

Held jobs​

A job that shows launch failed requeued held does not start by itself. The Jobs tab and squeue show this reason.

  1. Read the reason on the Compute nodes tab. The usual cause is a drained compute node.

  2. Correct the cause.

  3. Release the job.

    scontrol release <jobid>

The job then waits about 3 minutes (reason BeginTime) before it starts.

The scheduler VM is down​

These things happen when the scheduler VM is down:

  • Jobs that already run continue.
  • New jobs fail with Unable to contact slurm controller.
  • squeue and sinfo fail. ssh node-1 can fail too.
  • The console says Can't reach the cluster's scheduler right now.
  • The People tab refuses changes, as it reaches all VMs through the scheduler VM.

You cannot restart the scheduler VM. Contact support.

When it comes back, run squeue. A job that started at the time of the crash can be gone or held. Submit it again, or release it.

Do not power off a VM​

Warning:

Do not run poweroff on any VM. The VM stays off, it still shows Running, and you pay the full price for it.

If a compute node is off, replace it:

  1. Wait until it shows down* in sinfo. This takes about 7 minutes.

  2. Take it out of Slurm.

    scontrol delete nodename=node-2

  3. Delete it on the Compute nodes tab.

  4. Add a new compute node on the Compute nodes tab.

If the scheduler VM is off, contact support.

GPU heat​

The health check does not take a compute node out of service for a hot GPU. If a job runs slower than you expect, check GPU temperature in nvidia-smi.