Job options

Default values

A job with no options gets 2 CPUs, about 19 GB of memory, no GPU and no time limit. It goes to the gpu partition.

You ask for	You get
Nothing	2 CPUs, about 19 GB, no GPU.
-c 8	8 CPUs and about 77 GB. Memory follows CPUs, at about 9.6 GB per CPU.
--gres=gpu:1 with no -c	1 GPU, 28 CPUs and about 270 GB.
--gres=gpu:8 with no -c	8 GPUs and all 224 CPUs.
--gres=gpu:1 -c 8 --mem=64G	1 GPU, 8 CPUs and 64 GB.

GPUs

• --gres=gpu:N asks for GPUs per compute node. So -N2 --gres=gpu:8 is 16 GPUs.
• Use gpu:N. The form gpu:h200:N fails with Requested node configuration is not available.
• Each job sees only its own GPUs, numbered from 0.
• Forget --gres, and the job runs on no GPU, with no error.
• --gres=shard:1 is not available. A job takes whole GPUs.

Use --gres=gpu:N for DDP. The options --gpus-per-task and --ntasks-per-gpu crash DDP when two ranks share a compute node. The error is ncclUnhandledCudaError: Call to CUDA function failed.

CPUs and memory

Memory follows CPUs, so add --mem if you need more. --mem-per-gpu=48G also works.

With no -c, each GPU takes 28 CPUs and about 270 GB. So one small CPU job on a compute node keeps an 8-GPU job in the queue. With -c 8, the same job can start at once.

A job in the queue does not show this size. scontrol show job shows cpu=1 until the job starts.

These forms can surprise you:

Form	What happens
--mem=0	The job asks for all the memory of a compute node, so it waits for an empty one.
srun -c1 <command> with no -n	The command runs twice, because one core is 2 CPUs. Add -n1.
-c3	You get 4 CPUs.
srun --exclusive -n1	The job holds the whole compute node, but the command sees no GPU and 2 CPUs.
sbatch --exclusive	You get all 8 GPUs.
salloc -N1	You get 2 CPUs and no GPU. nvidia-smi says No devices were found.

A large CPU job can keep free GPUs out of use. For example, a 206-CPU job leaves too few CPUs for a --gres=gpu:1 job with no -c. That job waits with reason Resources. Give GPU jobs a small -c. When you size a large CPU job, leave some CPUs free.

Python's cpu_count() ignores your job's limit, so mp.Pool() can start 224 workers in a 4-CPU job. Set the number of workers yourself.

Time

• --time is in minutes. --time=30 is 30 minutes.
• With no limit, a job can run for ever.
• You can lower your own job's limit: scontrol update jobid=<id> timelimit=5.
• Only an admin can raise a limit. A user gets Access/permission denied.

A job with no --time gets the partition's default time. A job whose --time is over the partition's maximum waits with reason PartitionTimeLimit. Refer to Partitions.

Caution:

A job with no time limit holds its GPUs for ever if it hangs. Set --time on each job.

End a job when a process crashes

By default, one crashed process leaves the job RUNNING until its time limit. The other processes spin and the GPUs stay busy. Add --kill-on-bad-exit=1 to srun:

srun --kill-on-bad-exit=1 python train.py

An admin can set this for the whole cluster with KillOnBadExit=1. Refer to Slurm settings.

With this option, the exit code can change. A program that exits with 3 can show FAILED 143:0.

Keep srun as the last line

If another line follows srun in a script, sacct -X can show COMPLETED 0:0 when the training failed. The same happens if you pipe srun to another command. Keep srun as the last line.

Open files

Jobs have a soft limit of 1024 open files. A DataLoader with many workers can end with received 0 items of ancdata. Add this line to your job script:

ulimit -n 65536

Hold and release

To hold a job in the queue, and then release it:

sbatch --hold myjob.sbatch
scontrol release <jobid>

Arrays

An array runs one script many times. Each task has its own number in SLURM_ARRAY_TASK_ID.

sbatch --array=1-1000%50 myjob.sbatch

%50 runs at most 50 tasks at the same time. In file names, %A is the array's job ID and %a is the task number.

By default, an array can have task numbers up to 1000. A larger range fails with Invalid job array specification, and the message does not name the limit. By default, the cluster holds at most 10,000 jobs.

Caution:

Do not give a new --container-name to an array. Tasks that start together fail, and the name stays broken on that compute node. Refer to Containers.

Dependencies

A job can wait for another job:

sbatch --dependency=afterok:<jobid> next.sbatch

afterok, afterany, afternotok and singleton all work. If the first job fails, an afterok job waits for ever with reason DependencyNeverSatisfied. Cancel it.

Job email

Slurm accepts --mail-type, but it sends no email. It also shows no message about this.

Messages at submit

Slurm refuses these requests at submit. Each message appears after sbatch: error: or srun: error:.

Message	Cause
Requested node configuration is not available	Too many GPUs or too much memory for one compute node, or the form gpu:h200:N.
More processors requested than permitted	Too many CPUs for one compute node.
invalid partition specified: nosuch	The partition does not exist.
Invalid account or account/partition combination specified	You are not in that account. With EnforcePartLimits=ALL, also a partition that does not allow your account.
Invalid qos specification	Nobody gave you that QOS.
Requested time limit is invalid (missing or exceeds some limit)	--deadline is too close. With EnforcePartLimits=ALL, also a time over the partition's maximum.
Temporary disk specification can not be satisfied	--tmp is larger than the /tmp disk of a compute node.
Job violates accounting/QOS policy	Your job is over a limit on queued jobs.

Socket timed out

sbatch or squeue can show Socket timed out on send/recv operation when the scheduler VM is busy.

Caution:

sbatch with this message can still have made the job. Run squeue -u $USER before you try again, or you get 2 copies.

A timed-out squeue shows an error and no jobs. That does not mean that the queue is empty.