Partitions
A partition is a named group of compute nodes with its own limits. A new cluster has one partition, gpu. It has all compute nodes and no time limit.
The Partitions tab in the console shows partitions and it is read-only. An admin changes them on the login VM with scontrol. You cannot set partitions in the Slurm settings tab.
Create and delete a partition
To create a partition:
scontrol create partitionname=debug nodes=ALL maxtime=1:00:00 prioritytier=10
To change a limit:
scontrol update partitionname=debug maxtime=30
To delete a partition:
scontrol delete partitionname=debug
A job picks a partition with -p debug. Partitions stay after a settings save and after a restart of the Slurm services.
What works
| What | Result |
|---|---|
| Limits on real jobs | MaxTime, MaxNodes, AllowAccounts, DenyAccounts, PriorityTier, OverSubscribe=FORCE:2 and a Nodes= subset all work. |
state=drain, state=inactive | New jobs get a refusal. Jobs that run continue. |
state=down | Slurm accepts new jobs and holds them. Jobs that run continue. |
| Delete an empty partition | Works. |
| Delete a partition with jobs | Partition is in use |
| Delete the default partition | Allowed. Jobs then get No partition specified or system default partition until you set another default. |
Jobs get these messages from a partition's limits:
| Limit | Message |
|---|---|
Over MaxTime | Requested time limit is invalid (missing or exceeds some limit) |
AllowAccounts | Invalid account or account/partition combination specified |
Use nodes=ALL
A partition that names a compute node loses it for good when you delete that compute node. The partition shows Nodes=(null), and the compute node does not return when it comes back. To correct this, name the compute node again:
scontrol update partitionname=debug nodes=node-5
A nodes=ALL partition gets its compute nodes back by itself. If you must name compute nodes, use the real names, as ranges can fail. Refer to Add a compute node.
Limit a partition to a team
Use AllowAccounts. Refer to Teams.
AllowGroups=<team group> does not work. Jobs get uid ... not in group permitted to use this partition. It works only with a person's own group.
The Partitions tab has an Allowed groups column but no column for accounts. So a partition for one team looks open there. Check it with scontrol show partition.
Default time
You cannot clear a partition's DefaultTime. defaulttime=NONE shows Invalid input DefaultTime=NONE. defaulttime=UNLIMITED exits with 0 and changes nothing while MaxTime is finite. It applies only once you set maxtime=UNLIMITED. So think before you set a default time.
Reasons that point to the wrong cause
If one partition has a higher PriorityTier, jobs in the others can show this reason on healthy compute nodes:
Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions
Weight of a compute node
A compute node's weight that you set with scontrol goes away when the Slurm services on the scheduler VM restart. Its features stay.
Reservations
The end of a reservation ends the jobs in it. Jobs in it end with state TIMEOUT, even when they have hours left.
A reservation on a compute node also blocks jobs with no --time on a partition with no limit. They show ReqNodeNotAvail, May be reserved for other job, though the compute node is fine.