SSH to compute nodes

The rule

A user can SSH to a compute node only with a job on that compute node. Otherwise, the user sees this message:

Access denied by pam_slurm_adopt: you have no active jobs on this node

An admin can always SSH to a compute node.

Connect from the login VM with the compute node's name:

ssh node-1

What a session gets

The session joins your job. So these rules apply:

• The session gets only the CPUs, memory and GPUs of that job. For example, in a --gres=gpu:2 job, nvidia-smi -L shows 2 GPUs.
• The session ends when the job ends, along with all that it started.
• scancel and the time limit stop work that you started over SSH.

A session can start real training, for example with torchrun. It runs at normal speed, and scancel stops it. For most work, srun in a job script is still simpler.

An admin with no job on the compute node gets a session outside all jobs. Slurm does not count that work, and it does not stop it. The next job can land on GPUs in use. An admin with a job there joins that job.

Two jobs on one compute node

With 2 of your jobs on one compute node, SSH from the login VM can join either one. It does not always pick the newest. In a job with no GPU, nvidia-smi shows No devices were found.

SSH from inside a job joins that job. To run a command in a job that you choose, use this command on the login VM:

srun --jobid=<id> --overlap nvidia-smi -L

What works in a session

Tool	Result
tmux, scp, sftp, rsync	Work.
sudo	Works for admins.
systemctl --user, loginctl	Do not work.

A stall on the first connection

A first SSH to a compute node soon after a change on the People tab can stall for some minutes. It clears by itself. Refer to Storage.

The broken .ssh warning

The People tab can show this warning:

user1’s ~/.ssh folder can’t be updated, so SSH between VMs may not work for them. Everything else was applied.

This happens when someone breaks their own ~/.ssh, for example when it is a link to another folder. The person must correct it: ~/.ssh must be a normal folder that the person owns.

If someone else owns the home folder

If /home/jl_fs/<login-name> already exists and belongs to someone else, the person can log in, but cannot write to their home folder. SSH between VMs also fails for them. Refer to People and roles.

Failed logins

Failed SSH logins lead to a ban on all VMs: 3 failures from one IP address in 10 minutes ban it for 1 hour. Repeat bans last longer, and a further rule can ban the IP address for 1 week. Refer to Connect and log in.

Do not use a compute node's IP address. ssh to it fails with Host key verification failed.