Slurm clusters
Slurm is software that queues programs and gives them GPUs. A Jarvislabs Slurm cluster gives your team real Slurm on H200 GPUs. You log in to one VM and submit jobs. Slurm starts each job on compute nodes when the GPUs are free.
Jarvislabs builds the cluster and runs the scheduler VM. You control the people, the jobs, the Slurm settings and the number of compute nodes.
What a cluster contains
| Part | What it is |
|---|---|
| Compute node | A VM with 8 H200 GPUs and 8 InfiniBand cards. Jobs run here. You create a cluster with 1 to 128 compute nodes (up to 1,024 GPUs), and you can add and delete them later. |
| Login VM | A small VM that all people share. You log in here and submit jobs from here. |
| Scheduler VM | The VM that runs Slurm's own services. Jarvislabs manages it. You cannot log in to it. |
| Shared filesystem | Your storage at /home/jl_fs. The login VM and all compute nodes see the same files. |
The Slurm version is 26.05.5. The cluster also includes containers through Pyxis and Enroot, two roles (admin and user) and a page in the console for each cluster. Slurm clusters are available in the India-02 region.
When to use a Slurm cluster
Use a Slurm cluster for these needs:
- Several people share a set of GPUs, and jobs must wait in a queue.
- One training job runs across several compute nodes over InfiniBand.
- Teams need GPU limits, time limits or a fair share of the cluster.
- You want standard Slurm commands, such as
sbatchandsrun.
Use a VM for these needs:
- One person needs fewer than 8 GPUs.
- You want to pause the VM when you do not use it.
You pay for each whole compute node, with or without jobs. Slurm does not stop or start compute nodes for you. Refer to Billing.
Pages to read first
Read these pages in this order to get from no account to your first results:
Then read Storage before you run real work on several compute nodes. It tells you the safe way to use the shared filesystem.
Common commands
Run these commands on the login VM. Refer to Your first job for examples.
| To | Run |
|---|---|
| See the compute nodes | sinfo |
| See the queue | squeue |
| See your own jobs | squeue -u $USER |
| Submit a job script | sbatch myjob.sbatch |
| Run one command and watch it | srun -N1 hostname |
| Get a shell on a compute node | salloc -N1 --time=30 |
| See GPU use of your job | srun --jobid=<id> --overlap -n1 nvidia-smi |
| See why a job waits | squeue -u $USER -o '%i %j %T %r' |
| See finished jobs | sacct -X -n -o jobid,state |
| Cancel a job | scancel <jobid> |