Your first job

This page takes you from your computer to results. The examples use the login name user1.

See the cluster

Log in to the login VM, then run these commands:

sinfo
squeue

sinfo shows the compute nodes and their state. squeue shows the queue. Refer to Check jobs and compute nodes for what the words mean.

Run one command with srun

srun runs one command and shows its output:

srun -N1 hostname

-N1 asks for one compute node. To see the GPUs of a compute node, ask for them:

srun -N1 --gres=gpu:8 --time=2 bash -c "nvidia-smi -L | wc -l"

The output is 8. --gres=gpu:8 asks for 8 GPUs, and --time=2 sets a limit of 2 minutes.

Submit a script with sbatch

sbatch puts a script in the queue and returns at once. You can log out while the job runs.

1. Save this text as myjob.sbatch, with your login name in place of user1.

   #!/bin/bash
   #SBATCH -N2 --ntasks-per-node=2 --cpus-per-task=4 --gres=gpu:2 --time=30 -J myjob -o /home/jl_fs/user1/myjob-%j.out
   srun hostname

2. Submit the script.

   sbatch myjob.sbatch

3. Note the job ID in the output: Submitted batch job 12.

4. Read the output file when the job ends.

   cat ~/myjob-12.out

This job asks for 2 compute nodes, 2 tasks on each, and 2 GPUs on each. %j in the file name becomes the job ID. Without -o, the output goes to slurm-<jobid>.out in the folder where you ran sbatch.

Get a shell on a compute node with salloc

salloc gives you a shell on a compute node. Type exit when you finish. This command gets 2 CPUs and no GPU:

salloc -N1 --time=30

To get GPUs, ask for them:

salloc -N1 --gres=gpu:2 -c 8 --time=30

Run a training job

Step 1: Copy your code

Run this command on your computer:

scp train.py user1@203.0.113.10:~/

Step 2: Make your Python environment

Do this once. It needs about 5 GB of the shared filesystem.

1. On the login VM, save this text as setup.sbatch.

   #!/bin/bash
   #SBATCH -N1 -n1 -c16 --mem=32G --time=30 -J setup
   curl -LsSf https://astral.sh/uv/install.sh | sh
   export PATH=$HOME/.local/bin:$PATH
   uv python install 3.10
   uv venv --managed-python --python 3.10 ~/venv
   source ~/venv/bin/activate
   uv pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu126
   uv pip install numpy
   python -m compileall -q -j 4 ~/venv/lib/python3.10/site-packages

2. Submit it.

   sbatch setup.sbatch

3. Wait until it leaves the queue.

   squeue -u $USER

4. Read its output file, slurm-<jobid>.out, for errors.

The last line of the script compiles the Python files now. Then two compute nodes do not write those files together on your first run. Refer to Storage.

Step 3: Submit the training job

1. Save this text as train.sbatch in the folder with train.py.

   #!/bin/bash
   #SBATCH --gres=gpu:1 -c 8 --mem=64G --time=30 -J train
   source ~/venv/bin/activate
   python train.py

2. Submit it from that folder.

   sbatch train.sbatch

For more than one GPU, refer to GPU jobs and multi-node training.

Step 4: Watch the job

To	Run
See the queue	squeue -u $USER
See the output	tail slurm-<jobid>.out
See GPU use. This shows your job's GPUs, numbered from 0.	srun --jobid=<id> --overlap -n1 nvidia-smi
Stop the job	scancel <jobid>

Step 5: Copy the results back

Run this command on your computer:

scp user1@203.0.113.10:~/results/model.pt .

Common mistakes

Mistake	What you see	Correct action
You forget --gres.	The job runs with no GPU and no error. torch.cuda.is_available() is False.	Add --gres=gpu:N.
You run python train.py on the login VM.	Command 'python' not found, or No module named 'torch'.	Run it in a job.
You use sbatch --wrap with source.	source: not found. --wrap runs /bin/sh.	Use a script that starts with #!/bin/bash.
You forget #! in the script.	sbatch shows a message that asks for it.	Add #!/bin/bash as the first line.
You give a folder that does not exist in -D, or an output file that you cannot write.	The job ends FAILED 0:53, with no output file.	Check the path. A folder that does not exist in -o is fine: Slurm makes it.
You mistype a job ID in scancel.	No message. The command exits with 0.	Check with squeue -u $USER.