Skip to main content

Add and delete compute nodes

Add compute nodes​

  1. Open the cluster's Compute nodes tab.
  2. Click Add compute nodes.
  3. Enter the number of compute nodes. The dialog shows how many more you can add, and the price per hour.
  4. Click Add compute nodes.

The new compute node joins Slurm by itself. This usually takes 10 to 15 minutes.

Warning:

Billing starts when the VM starts.

A new compute node gets the same disk size as the other compute nodes, and today's people and Slurm settings. It joins partitions that use nodes=ALL.

Compute node names​

A new compute node gets a new name. Jarvislabs never uses a name twice in a cluster, so after a replacement you can have node-1 and node-5. To see the names, run this command:

sinfo -N

Delete a compute node​

An admin takes the compute node out of Slurm first, on the login VM. This example deletes node-2.

  1. Drain the compute node, so that it takes no new jobs.

    scontrol update nodename=node-2 state=drain reason=retire

  2. Run squeue until no job shows on node-2.

  3. Take the compute node out of Slurm.

    scontrol delete nodename=node-2

  4. Right away, on the Compute nodes tab, click the bin icon on the compute node's row.

  5. Type the compute node's name, and click Delete.

The VM is deleted and its billing stops.

Caution:

Do step 4 right away. A compute node that you took out of Slurm joins Slurm again if it restarts.

Warning:

You lose the compute node's disk, with its files in /tmp and its saved containers. Files in /home/jl_fs stay.

If scontrol delete shows Requested nodes are busy, a job still runs on the compute node. Wait for it to end, or cancel it.

Replace a compute node​

Add the new compute node first, then delete the old one.

If you think that the hardware is bad, tell support before you delete the compute node. Otherwise, the same hardware can come back to you or go to another customer.