Check jobs and compute nodes
The queue
squeue
squeue -u $USER
Code in squeue | Meaning |
|---|---|
R | The job runs. |
PD | The job waits. |
CG | The job has ended, and Slurm cleans up after it. |
To see why your jobs wait:
squeue -u $USER -o '%i %j %T %r'
To see when a job can start, as an estimate:
squeue -u $USER --start
To see all details of one job:
scontrol show job <jobid>
Compute nodes
sinfo
sinfo -N
Word in sinfo | Meaning |
|---|---|
idle | Free. |
mix | Partly in use. |
alloc | Full. |
comp | Slurm cleans up after a job. |
drng | Its jobs finish, and it takes no new ones. |
drain | Drained: it takes no new jobs, and it has none. |
down | Not usable. |
plnd | Slurm has planned a job from the queue onto it. |
Marks after a word have these meanings:
| Mark | Meaning |
|---|---|
* | The compute node does not answer, as in idle* or down*. |
- | Slurm holds the compute node for a job in the queue. |
@ | Slurm has a reboot planned. |
^ | The compute node reboots now. |
To see free GPUs on each compute node:
sinfo -O NodeHost,GresUsed:30,Gres
To see why compute nodes are out of service:
sinfo -R
sinfo -R cuts long reasons, for example to Kill task failed (Jo. The Compute nodes tab in the console shows the whole reason. So does scontrol show node <name>.
Job states
The Jobs tab and sacct use these words.
| State | Meaning |
|---|---|
PENDING | The job waits. The reason says why. |
RUNNING | The job runs. |
COMPLETED | The job ended with no error. |
CANCELLED | Someone cancelled the job. |
FAILED | The job ended with an error. |
TIMEOUT | The job went past its time limit. |
NODE_FAIL | A compute node failed under the job. |
OUT_OF_MEMORY | The job used more memory than it asked for. |
PREEMPTED | A more important job ended this job. |
DEADLINE | The job reached its deadline. |
The one flag that you often see is COMPLETING. The job has ended, and Slurm cleans up on its compute nodes. They stay busy until that ends.
Why a job waits
| Reason | Meaning | What to do |
|---|---|---|
Priority | Other jobs are ahead of you. | Wait. |
Resources | No compute node is free yet. | Wait. |
BeginTime | Slurm requeued the job, or its --begin time has not come. After a requeue, Slurm waits about 3 minutes before it starts the job again. | Wait. |
ReqNodeNotAvail, May be reserved for other job | A compute node that the job needs is down, drained or held for another job. | Often you only wait. If it lasts, run sinfo -R. |
Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions | The same as above. It also shows on healthy compute nodes when a partition with a higher tier exists. | Often you only wait. If it lasts, run sinfo -R. |
BadConstraints | The job needs a compute node that is gone. | Cancel it. |
JobHeldUser | Someone held the job. | Run scontrol release <jobid>. |
launch failed requeued held | Slurm held the job after a failed start. | Refer to Held jobs. |
PartitionConfig | The job does not fit the partition as it is now, for example when the partition has no compute nodes. | Check the partition. |
Dependency | The job waits for another job. | Wait. |
DependencyNeverSatisfied | The job that it waits for failed, so it never starts. | Cancel it. |
Prolog | A compute node prolog still runs. The job shows RUNNING while this lasts. | Wait. Refer to Slurm settings. |
Starts with Assoc or QOS | A limit on your account or QOS holds the job, such as AssocGrpGRES or QOSMaxGRESPerUser. | A job larger than its team's whole quota waits for ever. Cancel it and ask for less. Refer to Teams. |
A job that asks for more compute nodes than the cluster has also waits. srun shows Requested partition configuration not available now, then queued and waiting for resources. An admin can make Slurm refuse such jobs at submit with EnforcePartLimits=ALL. Refer to Slurm settings.
GPU use of a job
Run this command on the login VM. It shows your job's GPUs, numbered from 0.
srun --jobid=<id> --overlap -n1 nvidia-smi
To get a shell inside your job:
srun --jobid=<id> --overlap --pty bash
sstat shows CPU and memory use while the job runs. sacct shows GPU use in TRESUsageInMax, as gres/gpuutil and gres/gpumem. Slurm samples this every 30 seconds, so it reads 0 for a job with less than about 30 seconds of GPU work. seff is not available.
Finished jobs
Finished jobs leave squeue and the Jobs tab. sacct keeps them:
sacct -j <jobid> --format=jobid,state,exitcode,elapsed
sacct -X -n -o jobid,state
Refer to Job history for more.
Control your jobs
| To | Run |
|---|---|
| Cancel a job | scancel <jobid> |
| Hold a job in the queue | sbatch --hold at submit |
| Release a held job | scontrol release <jobid> |
| Lower your job's time limit | scontrol update jobid=<id> timelimit=5 |
The console has no button to cancel a job. Use scancel.
A user can cancel only their own jobs. scancel -u <someone else> shows nothing, exits with 0 and cancels nothing. An admin can cancel all jobs. Refer to People and roles.