The cluster page
The Slurm clusters page lists your clusters. Each card shows the name, status, size, Slurm version, region and price. Click View cluster to open the cluster's page.
The console marks Slurm clusters as Beta. The page has six tabs. The Compute nodes, Jobs, Partitions and Slurm settings tabs read live data from the cluster. If the scheduler VM does not answer, these tabs show a notice that ends with Can't reach the cluster's scheduler right now., and this note: Running jobs keep running. New jobs can’t start until it’s back.
The Instances page also lists the cluster's VMs. They have no actions there. You manage them from the cluster's page.
Banners
| Banner | Meaning |
|---|---|
Building the cluster. This usually takes 10 to 15 minutes. | The create continues. |
We couldn't build the cluster. Delete it and create it again, or contact support with this cluster's ID. | The create failed. |
Deleting the cluster’s VMs. | The delete continues. |
Your balance has run out. Add balance to keep the cluster: Jarvislabs can stop or suspend a cluster with a negative balance. | Refer to Billing. |
Your extra Slurm settings weren’t applied: ... | Correct the lines on the Slurm settings tab, then click Apply. |
Overview
The tab shows the SSH command at the top, and three cards.
| Card | Rows |
|---|---|
| Cluster | Size, Disk per compute node, Price, Region, Slurm, /home/jl_fs, VPC, Created. IN2 in the Region row is the short name of India-02. |
| Login VM | Size, Public IP, Private IP, and the Restart button. |
| Scheduler VM | Size, and Access: Managed by Jarvislabs. |
The Delete cluster button is at the bottom. Refer to Delete a cluster.
Compute nodes
| Column | Meaning |
|---|---|
| Compute node | The name, and a label if there is one. |
| VM | The VM's status: Starting, Running, setting up Slurm, Running, Restarting, Restart failed, Removing or Failed. Running does not prove that the VM is on. |
| State | Slurm's state: IDLE is free, ALLOCATED is full, MIXED is partly in use, DOWN is not usable. |
| Flags | DRAIN takes no new jobs. NOT_RESPONDING means that Slurm cannot reach it. COMPLETING means cleanup after a job. |
| Partitions | The partitions that hold this compute node. |
| GPUs used | Used and total, such as 2/8. |
| CPUs used | Used and total. |
| Jobs | The job IDs on this compute node. |
| Reason | The cause of DRAIN or DOWN, and who set the reason. |
Each row has a restart icon and a bin icon.
| Label by the name | Meaning |
|---|---|
Starting | New, and not in Slurm yet. |
Not in Slurm | Someone took it out of Slurm, or its start failed. You can now delete it. |
Not a cluster VM | Slurm lists this name, but it is not one of this cluster's VMs. |
The tab has Refresh and Add compute nodes buttons. Refer to Add and delete compute nodes and Restart a compute node or the login VM.
The console has no button to drain or resume a compute node. Admins use scontrol.
Jobs
This tab shows the jobs in Slurm's queue: pending, running and suspended jobs, and jobs that Slurm still cleans up after (COMPLETING). A job leaves the tab when it ends. The tab reads again every 15 seconds. For jobs that ended, run sacct on the login VM. Refer to Job history.
You can filter by state (RUNNING, PENDING, SUSPENDED, COMPLETING) and by user. The tab shows the newest 1,000 jobs.
| Column | Meaning |
|---|---|
| ID, Name, User, Partition | The job's details. |
| State | Refer to Job states. |
| Flags | COMPLETING means that Slurm still cleans up after the job. |
| Reason | Why the job waits. Refer to Why a job waits. |
| Compute nodes, GPUs | Where the job runs and how many GPUs it has. |
| Elapsed, Time limit, Submitted | The job's times. |
The tab is read-only. It has no button to cancel a job. Use scancel.
A burst of thousands of short array tasks slows this tab and the Compute nodes and Partitions tabs. They can take more than a minute to load. They get fast again about a minute after the burst.
Partitions
This tab shows each partition: Name, Compute nodes, Max time, Default time, State, Allowed groups, Priority tier, Max compute nodes per job and Jobs. The tab is read-only. Refer to Partitions.
People
This tab shows each person: Login name, Role, UID, SSH keys and Added. Each row has Edit keys, Change role and Remove. Refer to People and roles.
Slurm settings
This tab has the Extra slurm.conf box and the four script boxes, with Refresh, Discard changes and Apply. Refer to Slurm settings.