Skip to main content

Job history

The console has no job history page. The Jobs tab shows only jobs in the queue. A job leaves it when it ends. Use sacct on the login VM for all finished jobs.

Read job history​

To see one job:

sacct -j <jobid> --format=jobid,state,exitcode,elapsed

To see your jobs, one line per job:

sacct -X -n -o jobid,state

These points help you read the output:

  • sacct cuts long words, as in CANCELLED+. For full words, use sacct -o jobid,state%20,exitcode.
  • Add -S <date> for older jobs.
  • A submit that Slurm refused shows as FAILED in sacct, though it never ran.
  • A job that you cancelled in the held state, or before its --begin time, shows only with sacct -j <id>. It does not appear in a list by time.
  • With --kill-on-bad-exit=1, the exit code can change. A program that exits with 3 can show FAILED 143:0.
  • All people can read all jobs with sacct -a. PrivateData=jobs does not change this.

GPU use and cost​

sacct does not show money. billing= in its output is a CPU count. Your cluster bills each VM by the hour, whether or not jobs run.

sreport also gives CPU and GPU hours, with -t hours -T cpu,gres/gpu. It fills in only after its hourly roll-up, and it reports whole hours. Add end=now, or it reports from midnight to 00:59.

Export job history before you delete a cluster​

A cluster delete also deletes its job history. To keep it, run these lines on the login VM first. Any person can run them. The file lists all people's jobs, and it stays in your home folder on the shared filesystem.

f=~/slurm-job-history-$(date +%F).txt
sacct --allusers --allocations --starttime=1970-01-01 --endtime=now+3650days --parsable2 \
  --format=JobID,User,Account,Partition,NodeList,AllocTRES,Submit,Start,End,Elapsed,State,ExitCode,JobName \
  > "$f.part" && mv "$f.part" "$f" && echo "saved $f"

The output line is saved /home/jl_fs/user1/slurm-job-history-2026-11-05.txt. A | separates the fields. JobName is the last field.