Slurm settings

The Slurm settings tab changes Slurm's settings for the whole cluster. You can set the same boxes at create.

Box	What it does	Limit
Extra slurm.conf	Key=Value lines that Slurm adds to its settings.	8 KB
Compute node prolog	A script that runs as root on each compute node before a job.	16 KB
Compute node epilog	A script that runs as root on each compute node after a job.	16 KB
Scheduler prolog	A script that runs on the scheduler VM before a job starts, as the slurm user.	16 KB
Scheduler epilog	A script that runs on the scheduler VM after a job ends, as the slurm user.	16 KB

Change a setting

1. Open the cluster's Slurm settings tab.
2. Edit a box.
3. Click Apply.
4. Wait for the message Slurm settings applied. It usually takes about a second, and up to 2 minutes.

Jobs that run are not affected. A new script applies from the next job. If you leave the tab with unsaved edits, the console asks you first. Discard changes puts the saved text back.

A refused save changes nothing, and your old settings stay in force.

Caution:

Save settings while the cluster is quiet. Do not save many times in a row, or while thousands of short jobs finish. Each save reloads Slurm. Many saves close together can leave compute nodes in comp* for minutes.

If compute nodes show comp* after a save, an admin runs this command on them. The stop can take about 90 seconds. Jobs continue.

sudo systemctl restart slurmd

Rules for Extra slurm.conf

• Write one Key=Value on each line. Lines that start with # are comments.
• Each key can appear once.
• A value can have at most 512 characters, with no spaces, quotes or ;, #, \, ` or $.
• A key that takes a number must have a whole number, UNLIMITED or INFINITE.
• A key that takes YES or NO accepts yes, no, true, false, 1, 0, up or down.
• SlurmctldDebug, SlurmdDebug, SlurmctldSyslogDebug and SlurmdSyslogDebug can be at most verbose.

Note:

The check accepts some values that Slurm ignores, such as KillOnBadExit=2. A wrong value can look saved and do nothing. Check the value with scontrol show config.

Rules for the scripts

A script must start with a #! line, for example #!/bin/bash. An empty box means no script. A prolog cannot set a job's environment.

Settings you may want

Line	What it does
EnforcePartLimits=ALL	Refuses at submit a job that cannot run on this partition, so it does not wait. Without it, a job that asks for too many compute nodes waits for ever.
KillOnBadExit=1	Ends the srun step when one process crashes, on every job. A crash then does not hold the GPUs until the time limit.
PrivateData=jobs	Hides other people's jobs from squeue and scontrol show job. It does not hide them from sacct -a, sreport, sshare, sacctmgr show assoc or the console's Jobs tab.
PrologEpilogTimeout=1800	The number of seconds that a prolog or epilog can run. The default is 300.
PriorityWeightFairshare=100000	Turns fair share on. Refer to Teams.
DependencyParameters=kill_invalid_depend	Cancels a job whose dependency can never be met.

Settings that you cannot change

Jarvislabs sets these keys, because the cluster depends on them. If you enter one, the console refuses the save with a message like this:

Line 1 (ClusterName): ClusterName is set by Jarvislabs and can't be changed.

Connection between the VMs

These settings connect the VMs of the cluster. A different value breaks the cluster. There is no alternative. To add or delete compute nodes, use the Compute nodes tab.

ClusterName, ControlMachine, ControlAddr, BackupController, BackupAddr, SlurmctldHost, SlurmctldAddr, SlurmctldPort, SlurmdPort, SlurmctldParameters, SlurmctldPidFile, SlurmdPidFile, StateSaveLocation, SlurmdSpoolDir, SlurmUser, SlurmdUser, MaxNodeCount, SrunPortRange, TopologyPlugin, RoutePlugin, SwitchType, LaunchType, PluginDir, TreeWidth

Authentication

These settings control how the VMs trust each other. There is no alternative.

AuthType, AuthInfo, AuthAltTypes, AuthAltParameters, CredType, CryptoType, HashPlugin, TLSType, TLSParameters, CertmgrType, CertmgrParameters, CertgenType, CertgenParameters, SlurmctldHttpAuthParameters, SlurmdHttpAuthParameters, HttpParserType

Accounting

Job history and limits depend on these settings. To set limits, use sacctmgr. To read job history, use sacct.

AccountingStorageType, AccountingStorageHost, AccountingStorageBackupHost, AccountingStorageExternalHost, AccountingStoragePort, AccountingStoragePass, AccountingStorageEnforce, AccountingStorageTRES, AccountingStorageParameters, JobAcctGatherType

Job isolation

These settings keep each job on its own CPUs, memory and GPUs. To change what a job gets, use job options. To run steps around each job, use the prolog and epilog boxes.

ProctrackType, TaskPlugin, SelectType, GresTypes, PrologFlags, JobContainerType, NamespaceType, PlugStackConfig, DisableRootJobs, UsePAM

Log files

These settings fix where Slurm writes its logs. To read a compute node's log, use sudo to read /var/log/slurm/slurmd.log on that compute node. To change how much Slurm logs, use the Debug keys above, up to verbose.

SlurmctldLogFile, SlurmdLogFile, SlurmSchedLogFile

Programs on the scheduler VM

These settings run programs on the scheduler VM. You cannot log in to that VM. To run a script there, use the Scheduler prolog and Scheduler epilog boxes. Email is not available.

MailProg, RebootProgram, ResumeProgram, SuspendProgram, ResumeFailProgram, SlurmctldPrimaryOnProg, SlurmctldPrimaryOffProg, ResvProlog, ResvEpilog, JobCompType, JobCompLoc, JobCompPassScript, JobSubmitPlugins, BurstBufferType, NodeFeaturesPlugins, PrEpPlugins, PrioritySiteFactorPlugin, MCSPlugin, MetricsType, MetricsParameters, MetricsAuthUsers, PowerAction

Temporary disk

Each compute node tells Slurm the size of /tmp, so --tmp works. A different value takes compute nodes out of service. Use /tmp as it is. Refer to Storage.

TmpFS

Health check on compute nodes

Jarvislabs sets these keys for its own health check on compute nodes. To run your own checks, use the Compute node prolog box.

HealthCheckProgram, HealthCheckInterval, HealthCheckNodeState, HealthCheckTimeout

Lines that belong elsewhere

The console also refuses these lines, and it tells you where to go:

Line	Where to go
PartitionName	Admins change partitions on the login VM with scontrol. Refer to Partitions.
NodeName, NodeSet	Add and delete compute nodes on the Compute nodes tab.
DownNodes	Drain or resume compute nodes with scontrol on the login VM.
Include	You cannot include extra files.
Prolog	Use the Compute node prolog box.
Epilog	Use the Compute node epilog box.
PrologSlurmctld	Use the Scheduler prolog box.
EpilogSlurmctld	Use the Scheduler epilog box.
TaskProlog, TaskEpilog	Not available.

Messages

You see	Why
Line 1 (FooBarBaz): FooBarBaz is not a slurm.conf key in this Slurm version.	The setting is not real.
Line 1 (ClusterName): ClusterName is set by Jarvislabs and can't be changed.	The setting is in the lists above.
Line 1 (NodeName): not allowed here; add and remove compute nodes on the Compute nodes tab.	Use the tab that it names.
Line 1 (KillWait): KillWait takes a whole number.	The value has the wrong type.
Line 1 (x): use one Key=Value per line.	The line has no =, or no value.
Slurm refused this. Nothing was changed. ...	Slurm's own check said no.
The Slurm settings changed since you loaded them. Refresh and try again.	Someone else saved first.
Another change to this cluster's Slurm settings is running. Try again in a moment.	Two saves at the same time.
Slurm couldn't reload. The previous settings are back in place. ...	Slurm did not answer in time, often on a busy cluster. A valid setting can get this message. Try again when the cluster is quieter.
The change didn't finish in time. Load the settings again to see what is in place.	Click Refresh.
We couldn’t confirm this. Press Refresh to see what’s in place.	The connection stopped. Click Refresh.
Couldn't write the files on the scheduler VM. Its disk may be full. The previous settings are still in place.	Contact support.
Slurm couldn't reload, and the previous settings couldn't be put back. Contact support with this cluster's ID.	Contact support.

When a prolog or epilog goes wrong

Fault	What happens	What to do
A compute node prolog fails or hangs	It drains each compute node that a job lands on, with reason Prolog error. Slurm holds the job.	Follow the steps below.
A compute node epilog fails or hangs	It drains the compute node, with reason Epilog error. The job still shows COMPLETED.	Correct the script, then resume the compute node.
A slow compute node prolog	The job shows reason Prolog. srun shows Prolog hung on node node-1.	Wait, or correct the script.
A scheduler prolog fails	It shows no reason. Slurm requeues the job, which waits with reason BeginTime. srun says PrologSlurmctld failed, job killed. You cannot read the scheduler VM's log.	Correct the script.
A scheduler epilog fails	Slurm ignores it.	Correct the script.

A hung compute node prolog stops after 300 seconds by default. A job that you cancel while its prolog hangs can come back as launch failed requeued held at that time. Cancel it a second time.

The scheduler VM does not have /home/jl_fs. A compute node's script output is in /var/log/slurm/slurmd.log on that compute node. Read it with sudo.

To recover from a compute node prolog fault:

1. Correct the script on the Slurm settings tab.

2. Resume the compute node.

   scontrol update nodename=node-1 state=resume

3. Release the job.

   scontrol release <jobid>