Concepts
This page tells you how a job goes through the cluster. The Glossary defines each term.
How a job runs
A user wants to train a model on 2 GPUs. They log in to the login VM with SSH. Their home folder is /home/jl_fs/user1, on the shared filesystem.
They write a job script and run sbatch train.sbatch. The scheduler VM puts the job in the queue. When a compute node has 2 free GPUs, and enough free CPUs and memory, Slurm starts the job there.
The job reads and writes files in their home folder. The same files are on the login VM, so they can read the results when the job ends. They do not need to copy them between VMs.
Three ways to start a job
| Command | What it does | Use it for |
|---|---|---|
sbatch | Puts a script in the queue. You can log out. | Training and all long work. |
srun | Runs one command and shows its output in your terminal. The job ends if your connection drops. | Short checks, and steps inside a job script. |
salloc | Gives you a shell on a compute node. | Interactive work and tests. |
Where work happens
Use the login VM to edit files, submit jobs and read results. Do not run real work on it. All people share it, and nothing limits one person's CPU or memory use.
Use compute nodes for all real work. This includes software installs, builds and training. The login VM does not have gcc, make, pip, mpicc or nvidia-smi.
How Slurm shares a compute node
Slurm gives CPUs, memory and GPUs to each job. Jobs from different people can share one compute node. Each job sees only its own GPUs and CPUs.
A job gets no GPU unless it asks for one. Refer to Job options for the default values.
Queue order
By default, all jobs have the same priority, so the first job in the queue goes first. Slurm can start a smaller job early if that job does not delay other jobs. You can turn on priorities and fair share on the Slurm settings tab. Refer to Teams, quotas and fair share.
Roles
Each person on the cluster has one role. The role controls what the person can do inside the cluster.
| Role | What it adds |
|---|---|
| User | Submit jobs, and control your own jobs. |
| Admin | Use sudo on the login VM and compute nodes. Change partitions, compute node states and limits. SSH to any compute node. |
The first person on a cluster is an admin. At least one admin must stay on the cluster. Refer to People and roles for the full list.
Roles apply inside the cluster. Anyone with access to the console account can use the buttons on the cluster's page, such as Restart.
What stays and what goes
| Item | When you delete a compute node | When you delete the cluster |
|---|---|---|
Files in /home/jl_fs | Stay | Stay |
Files in /tmp and containers on the compute node | Lost | Lost |
| Job history | Stays | Lost |
| Teams, quotas and limits | Stay | Lost |
The shared filesystem has its own life and its own bill. You can build a new cluster on the same filesystem and find your files there.
Failures
Slurm requeues a job when its compute node fails. The job starts again from the beginning, unless your program loads a checkpoint. Save checkpoints in your home folder. Refer to Node failures and checkpoints.