Teams, quotas and fair share
An admin sets teams, quotas and fair share with sacctmgr on the login VM. The console does not show them.
A cluster delete loses teams, quotas and limits. Keep your sacctmgr commands in a script, so that you can run them on a new cluster.
Make a team
A Slurm account groups people into a team. This example makes the team vision and adds user1 to it.
sacctmgr -i add account vision
sacctmgr -i add user user1 account=vision
A person picks the team for a job with --account=vision. To make it their default:
sacctmgr -i modify user user1 set defaultaccount=vision
Set limits
| Limit | Command | A job over the limit waits with |
|---|---|---|
| GPUs for the whole team | sacctmgr -i modify account vision set GrpTRES=gres/gpu=4 | AssocGrpGRES |
| Running jobs per person | sacctmgr -i modify user user1 account=vision set MaxJobs=1 | AssocMaxJobsLimit |
| Time per job | sacctmgr -i modify user user1 account=vision set MaxWall=02:00:00 | AssocMaxWallDurationPerJobLimit |
A value of -1 clears a limit.
QOS
A QOS is a named set of limits that you give to people.
sacctmgr -i add qos high set MaxTRESPerUser=gres/gpu=2
sacctmgr -i modify user user1 set qos+=high
A person uses it with --qos=high. A person without it gets Invalid qos specification.
Turn on priorities
By default, the first job in the queue goes first. Fair share and QOS priority do nothing until you turn them on. Put these lines in the Extra slurm.conf box on the Slurm settings tab:
PriorityWeightFairshare=100000
PriorityWeightQOS=10000
PriorityWeightAge=100
To see fair share, run sshare -a. To see why a job has its priority, run sprio.