Skip to main content

FAQ

Can I run a job on part of a compute node?​

Yes. A job asks for GPUs, CPUs and memory. Other jobs can use the rest of the compute node. You pay for the whole compute node in all cases.

Do compute nodes bill when no job runs?​

Yes. Each compute node bills all the time. Refer to Billing.

Can I pause a cluster?​

No. The console has no pause action for a cluster. To stop the charges for a compute node, delete it. Your files in /home/jl_fs stay.

What happens when my balance runs out?​

If your balance goes negative, Jarvislabs can stop or suspend the cluster. Keep your balance above zero. Refer to Billing.

Can I log in to the scheduler VM?​

No. Jarvislabs manages it. Admins change Slurm from the login VM and from the Slurm settings tab.

How do I change a Slurm setting?​

Use the Slurm settings tab. Partitions change with scontrol on the login VM. Limits change with sacctmgr on the login VM.

Can I install software on the VMs?​

Install software in your home folder, in a job. An admin has sudo, but a change to one VM does not reach the others. A new compute node does not get it either.

Why does pip fail?​

The VMs do not have pip. Use uv pip install in a venv, or pip in a container. Refer to Software environments.

Does my training use InfiniBand?​

Yes, if it runs in a uv venv. Multi-node PyTorch training uses InfiniBand with no settings.

In a container, only if the image has the InfiniBand library. The stock PyTorch image does not have it, so training uses Ethernet, with no error. Build an image with the library. Refer to Containers.

To check, add export NCCL_DEBUG=INFO to the job. The output shows Using network IB for InfiniBand, or Using network Socket for Ethernet.

Does MPI use InfiniBand?​

Not by default. Add export OMPI_MCA_pml=ucx, below 64 ranks. Refer to MPI.

Can I use fast scratch space?​

Use /tmp on the compute node. Many small files run much faster there. Refer to Storage.

Why does my container run out of memory when I write to /tmp?​

A container's /tmp is RAM. Mount a folder with --container-mounts=/var/tmp:/scratch. Refer to Containers.

How do I see my job's GPU use?​

Run srun --jobid=<id> --overlap -n1 nvidia-smi on the login VM.

What happens to my job if a compute node fails?​

Slurm requeues it after about 7 minutes. It starts again from the beginning, unless your program loads a checkpoint. A multi-node job can fail and not requeue. Refer to Node failures and checkpoints.

Can I serve a model from a job?​

Yes. A job can run a server, and you reach it through an SSH tunnel, as in Notebooks and TensorBoard. vLLM needs a venv made with --managed-python. The server stops at the job's time limit, and only an admin can raise that limit.

How do I cancel a job from the console?​

You cannot. Use scancel <jobid> on the login VM.

Can other people see my jobs?​

Yes. All people on the cluster can see all jobs, job names and paths. PrivateData=jobs hides other people's jobs from squeue and scontrol show job, but not from sacct -a, sreport, sshare or the console's Jobs tab. Refer to Slurm settings.

Can other people read my files?​

Your home folder is private to you, and admins can read it with sudo. Admins of any cluster or VM on the same filesystem can read it too. Other people's jobs on a compute node can read your files in /tmp if the file permissions allow it.

Can I use the same filesystem for two clusters?​

Not at the same time. One Slurm cluster can use a filesystem at a time.

Can I add a compute node of another type?​

No. New compute nodes use the same type and disk size as the cluster.

How do I schedule a job for a later time?​

Use --begin with sbatch. All VMs use UTC. scrontab is not available.