Troubleshooting

I cannot log in

Symptom	Cause	What to do
Permission denied (publickey)	Your key is not on the People tab, or SSH uses another key.	Check the People tab. Add -i <key file>.
The connection times out, or the VM refuses it	The SSH rules do not allow your IP address, or your IP address has a ban after failed logins.	Check the SSH rules. A ban lasts 1 hour or longer. Try from another IP address.
REMOTE HOST IDENTIFICATION HAS CHANGED	You created the cluster again, and the login VM got the old IP address.	Refer to Connect and log in.
Could not chdir to home directory: Permission denied	Someone else owns /home/jl_fs/<login-name>.	Refer to People and roles.
The login VM does not answer	It hangs.	Restart it on the Overview tab. Refer to Restart.

I cannot SSH to a compute node

Symptom	Cause	What to do
Access denied by pam_slurm_adopt: you have no active jobs on this node	You are a user with no job on that compute node.	Start a job there.
Could not resolve hostname	The compute node is new, or the scheduler VM is down.	Wait up to a minute. Check with sinfo.
Host key verification failed	You used an IP address.	Use the name, such as node-1.
nvidia-smi says No devices were found	You have 2 jobs on the compute node, and SSH joined the one with no GPU.	Use srun --jobid=<id> --overlap --pty bash.
The People tab shows user1’s ~/.ssh folder can’t be updated	The person's ~/.ssh is broken.	Refer to SSH to compute nodes.

A command is not found

Symptom	Cause	What to do
nvidia-smi, pip, gcc, make or mpicc not found on the login VM, with Ask your administrator	These tools are not on the login VM.	Ignore the text about your administrator. Run the command in a job.
python3 -m venv fails	venv is not installed.	Use uv. Refer to Software environments.
Command 'python' not found	There is no python on the login VM.	Run it in a job, in a venv.
python: command not found after activate on the login VM	The venv uses the compute node's Python.	Make it with --managed-python.
Broken symlink at venv/bin/python	The same cause.	Run uv pip in a job.
sudo: a password is required	You are a user.	Ask an admin.
permission denied while trying to connect to the docker API	You cannot use docker on the VMs.	Use Containers.
seff: command not found	seff is not available.	Use sacct.
source: not found	sbatch --wrap runs /bin/sh.	Use a script that starts with #!/bin/bash.

My job does not start

Run this command, and look up the reason in Why a job waits.

squeue -u $USER -o '%i %j %T %r'

Symptom	Cause	What to do
The job waits for ever, and srun shows Requested partition configuration not available now	The job asks for more compute nodes than the cluster has, or for a limit that the partition does not allow.	Cancel it and ask for less.
Reason AssocGrpGRES for a long time	The job is larger than its team's quota, or the team's GPUs are in use.	If it is larger than the quota, cancel it.
Reason Resources with GPUs free	A big job took the CPUs, or the job has no -c.	Add -c 8. Refer to Job options.
An 8-GPU job waits behind one small job	With no -c, the job needs all 224 CPUs.	Add -c 8.
--mem=0 waits	It asks for all the memory of a compute node.	Give a number.
Reason launch failed requeued held	Slurm held the job after a failed start.	Refer to Held jobs.
Reason BeginTime after a failure	Slurm requeued the job.	Wait about 3 minutes.
Reason DependencyNeverSatisfied	The first job failed.	Cancel the job.
Reason BadConstraints	The job needs a compute node that is gone.	Cancel it.
srun sits at queued and waiting for resources	No compute node can take it.	Run sinfo -R. A drained compute node can be the cause.

My job fails or does not do what I expect

Symptom	Cause	What to do
The job runs but sees no GPU, with no error	You forgot --gres.	Add --gres=gpu:N.
environment variable RANK expected, but not set	The script needs the rank variables.	Use a recipe from GPU jobs.
4 separate trainings, with no error	srun python with a script that does not read the variables.	Use a recipe from GPU jobs.
The second compute node stays idle	torchrun without srun.	Use the recipe.
ncclUnhandledCudaError: Call to CUDA function failed	--gpus-per-task or --ntasks-per-gpu with 2 ranks on a compute node.	Use --gres=gpu:N.
A crashed process, and the job stays RUNNING until TIMEOUT	Slurm's default.	Add --kill-on-bad-exit=1.
sacct shows COMPLETED 0:0, but training failed	Another line follows srun.	Keep srun as the last line.
FAILED 0:53, with no output file	The -D folder does not exist, or you cannot write the -o file.	Check the paths.
OUT_OF_MEMORY	The job used more than it asked for. A container's /tmp also counts.	Add --mem. Write temporary files to a mounted folder.
received 0 items of ancdata	The limit of 1024 open files.	Add ulimit -n 65536.
Python.h: No such file or directory	The venv has no headers.	Make it with --managed-python.
Failed to initialize NumPy	numpy is not in the venv.	Run uv pip install numpy.
OMPI was not built with SLURM's PMI support	An MPI program under plain srun.	Add --mpi=pmix.
MPI between compute nodes is slow	MPI uses Ethernet by default.	Add export OMPI_MCA_pml=ucx. Refer to MPI.
Multi-node training is much slower in a container	The image has no InfiniBand library.	Refer to Containers.
The job stops when your connection drops	A foreground srun ends with the connection.	Use sbatch.
Socket timed out on send/recv operation	The scheduler VM is busy.	Wait. Run squeue -u $USER before you submit again.
Unable to contact slurm controller	The scheduler VM is down.	Refer to The scheduler VM is down.

Containers fail

Symptom	Cause	What to do
a container with name "pyxis_pt271" does not exist, and --container-image is not set	The name is not on this compute node.	Give --container-image too.
File exists, in an array	Tasks made the same name at the same time.	Run one job first. Refer to Arrays.
failed to execute: /bin/sh	The name is broken on this compute node.	Run enroot remove -f pyxis_<name>.
wrong type 3 != 4	Someone wrote into a name on one compute node.	Use a new name.
can't open file '/workspace/train.py'	The container starts in the image's folder.	Add --container-workdir=$PWD.
Your shared folder is not there	Only your own home folder is in the container.	Add --container-mounts.
MPI hangs across 2 compute nodes	The compute node's MPI settings do not apply in a container.	Refer to MPI in a container.

The shared filesystem has a problem

Symptom	Cause	What to do
A process does not respond, and you cannot kill it	A read stalled.	Wait. Most stalls clear. Refer to Storage.
Stale file handle	Another VM replaced the file.	Retry the read, or give each version a new name.
database disk image is malformed	SQLite was written from several compute nodes.	Use one compute node, or /tmp.
FileNotFoundError or DatasetGenerationError on a first run	The Hugging Face cache is empty on 2 compute nodes.	Fill it with a 1-node job.
ModuleNotFoundError during a run	Someone installed packages during the run.	Do not install during a run.
The filesystem is full, though you deleted a venv	uv's cache still holds the files.	Run rm -rf ~/.cache/uv.

A compute node is out of service

Run this command to see the reasons:

sinfo -R

To put a compute node back, an admin runs this command. A user must ask an admin. A restart does not clear a drain.

scontrol update nodename=node-1 state=resume

Reason	Cause	What to do
Kill task failed	Refer to the next section.	Resume it, or restart it.
SlurmdSpoolDir is full	The compute node's disk is full.	Refer to A compute node disk is full.
Prolog error	A compute node prolog failed or hung.	Refer to Slurm settings.
Epilog error	A compute node epilog failed or hung.	Correct the script, then resume.
Not responding	Slurm cannot reach the compute node.	Wait. If it lasts, restart it. Refer to Restart.
comp* in sinfo	slurmd does not answer on that compute node after many reloads.	An admin runs sudo systemctl restart slurmd there.

Kill task failed

sinfo -R cuts this text to Kill task failed (Jo. A process stalled on the shared filesystem, and Slurm cannot stop it. The compute node takes no new jobs, and you still pay for it. It stays drained until someone acts.

1. Wait for the stall to clear. Most stalls clear in minutes.

2. As an admin, resume the compute node.

   scontrol update nodename=node-1 state=resume

3. If the fault does not clear, restart the compute node. Refer to Restart.

4. Resume the compute node again, because a restart does not clear a drain.

The job that caused this can belong to another person. It can also take a 2-node job's second compute node away, so 2-node jobs wait with Nodes required for job are DOWN, DRAINED or reserved....

A compute node disk is full

Slurm drains the compute node the first time that a job touches it. The reason is SlurmdSpoolDir is full. You lose no job, but a job can end up held. Container images fill this disk fastest.

1. Free space on the compute node. Refer to Containers.

2. As an admin, resume the compute node.

   scontrol update nodename=node-1 state=resume

3. Release any held job.

   scontrol release <jobid>

Files do not copy to compute nodes

sbcast of a large file, such as 6 GB, can fail with Header lengths are longer than data received. It leaves a partial file on each compute node. Do not use the same path for source and destination. That can destroy the source, with Bus error (core dumped). To put an image on several compute nodes, keep one .sqsh file in your home folder.

The console has a problem

Symptom	Cause	What to do
Can't reach the cluster's scheduler right now.	The scheduler VM does not answer. Jobs continue.	Try later. If it lasts, contact support.
The tabs are slow during array jobs	Thousands of short tasks slow the console's reads.	Wait. They get fast about a minute after the burst.
Couldn't read the cluster's answer. Try again.	The console cannot read the scheduler VM's answer.	Try again.
This needs a running cluster. or The cluster is still being created. Try again once it's running.	The cluster is not Running.	Wait.
The cluster failed to build. Delete it and create it again.	The create failed.	Delete it, and create it again.
Slurm couldn't reload. The previous settings are back in place.	The cluster is busy.	Refer to Slurm settings.

How to contact support

Send an email to support@jarvislabs.ai. Give the cluster's ID, the compute node's name, the job ID and the exact message. The cluster's ID is in the page's address, and the console shows it for a failed cluster.