Skip to main content

Restart a compute node or the login VM

Use Restart for a VM that hangs or does not answer. You cannot restart the scheduler VM. For that VM, contact support.

Anyone with access to the console account can click Restart. Slurm's admin and user roles do not apply.

Restart a compute node​

The restart does not ask Slurm and it does not wait for jobs.

Caution:

A restart stops all jobs on the compute node. A job across several compute nodes stops entirely. To let jobs finish, drain the compute node first.

  1. To let jobs finish, drain the compute node on the login VM. Skip this step if you want to stop the jobs.

    scontrol update nodename=node-1 state=drain reason=restart

  2. If you drained it, wait until its jobs are gone. Check with squeue.

  3. On the Compute nodes tab, click the restart icon on the compute node's row.

  4. Read the dialog. It tells you how many jobs run on the compute node.

  5. Click Restart.

  6. Wait about 9 to 10 minutes.

  7. Check that the compute node is back. Refer to the next section.

  8. If sinfo -R lists the compute node, resume it.

    scontrol update nodename=node-1 state=resume

The compute node keeps its name, IP addresses and disk. If its server is out of service, it moves to a free server. If none is free, the restart fails and the compute node keeps running. Its people and /home/jl_fs also stay. /tmp is empty after the restart.

How to know that the compute node is back​

No single column proves that the compute node is back. Check all of them.

Time after the clickVM columnState column
About 1 minuteRestartingIDLE, which Slurm has not updated yet
About 3 minutesRunning, though the VM is not back yetIDLE, which Slurm has not updated yet
About 4 minutesRunningIDLE, with flag NOT_RESPONDING
About 8 minutesRunningDOWN
About 9 minutesRunningIDLE or MIXED, and the compute node is back
Caution:

IDLE does not prove that the VM is back. Slurm can show IDLE for minutes while the VM is down. Do not use the compute node until you see the VM column at Running and at least 9 minutes have passed.

To confirm, run a small job on the compute node:

srun -n1 -t1 -w node-1 hostname

A restart can take longer. Wait and try again if the command does not work at first.

A restart does not clear a drain​

A restart does not clear a drain, whoever or whatever set it. Check sinfo -R after each restart. If the compute node appears there, run the resume command in step 8.

Messages​

MessageMeaning
Restart failed in the VM column, and the tooltip The last restart failed. You can try again.Click the restart icon again.
node-1 can't be restarted: its last restart failed. Our team has been told and is looking at it.Contact support if it lasts.
node-1 is already restarting. Wait for it to finish.Wait.
The restart of node-1 failed, and its VM may be down. Our team has been told.Contact support if it lasts.
We couldn’t confirm this. Check the compute node’s status.The connection stopped. The restart can continue. Read the VM column.

Restart the login VM​

Caution:

A login VM restart ends everyone's SSH sessions, tmux sessions and srun commands on the login VM. Jobs started with srun from the login VM stop, even inside tmux. sbatch jobs on compute nodes continue.

  1. Tell the people on the cluster.
  2. Open the Overview tab.
  3. On the Login VM card, click Restart.
  4. Read the dialog.
  5. Click Restart.
  6. Wait about a minute.
  7. Connect again with the same SSH command.

The login VM keeps its IP addresses, disk and host key. The Running status can show a few seconds before SSH answers.

The console dialog says that jobs on the compute nodes are not affected. This is true for sbatch jobs, and not for jobs started with srun from the login VM.