Containers

Containers run through srun, with Pyxis and Enroot. docker is on the VMs, but it refuses you with permission denied while trying to connect to the docker API.

Run a container

srun --gres=gpu:2 --container-image=pytorch/pytorch:2.7.1-cuda12.6-cudnn9-runtime --container-name=pt271 --container-workdir=$PWD python -c "import torch; print(torch.cuda.device_count())"

Option	What it does
--container-image	The image to fetch. Docker Hub, ghcr.io and quay.io work with no login.
--container-workdir	The folder to start in. Without it, you start in the image's own folder and your files are not found.
--container-name	Keeps the unpacked image, so the next start is fast.
--container-mounts	Other folders to show in the container.

The first use of an image on a compute node takes from seconds to a few minutes, depending on its size. With --container-name, the next start takes only seconds.

A plain image such as ubuntu:22.04 has no nvidia-smi unless you set NVIDIA_VISIBLE_DEVICES=all:

srun --gres=gpu:2 --time=3 --container-image=ubuntu:22.04 --export=ALL,NVIDIA_VISIBLE_DEVICES=all nvidia-smi -L

Files in a container

Your home folder is in the container, at the same path. Nothing else in /home/jl_fs is there. To use a shared folder, add it:

--container-mounts=/home/jl_fs/shared:/home/jl_fs/shared

Caution:

A container's /tmp is RAM, and it counts against your job's memory. A job that writes more there than its --mem allows ends OUT_OF_MEMORY. For temporary files on disk, mount a folder of the compute node.

--container-mounts=/var/tmp:/scratch

Container names

A name belongs to you and to one compute node. It lives on that compute node's disk, and it stays after a reboot. A new compute node starts with nothing. On another compute node, a name alone fails:

pyxis: error: a container with name "pyxis_pt271" does not exist, and --container-image is not set

In a job script, give both --container-image and --container-name. Then the job works on a compute node that does not have the name yet.

Arrays

Tasks that start together cannot all make the same name. They fail with File exists, and the name stays broken on that compute node. A later job then fails with failed to execute: /bin/sh.

Use one of these two ways:

• Run one job first to fetch the image, then run the array.
• Give no --container-name. Each task then starts more slowly.

To correct a broken name, remove it:

enroot remove -f pyxis_pt271

Writable names

A write into a name changes it on one compute node only. A later 2-node job with that name dies at start with wrong type 3 != 4. To change an image, build a new one with a new name. Refer to the next section.

Check that your container uses InfiniBand

The stock PyTorch image does not have libibverbs, the library that InfiniBand needs. It uses Ethernet with no error. Multi-node training only runs much slower.

To check an image, look for the library:

srun -N1 --time=3 --container-name=pt271 bash -c 'ls /usr/lib/x86_64-linux-gnu | grep -E "ibverbs|mlx5"'

No output and exit code 1 mean that the library is not there.

You can also run your job across 2 compute nodes with NCCL's log on. On one compute node, a stock image says Socket and it does no harm.

srun -N2 --ntasks-per-node=2 --gres=gpu:2 --time=3 --container-image=$HOME/ptib.sqsh --container-workdir=$PWD --export=ALL,NCCL_DEBUG=INFO python train.py 2>&1 | grep -E "NET/IB|Using network"

Log line	Meaning
Using network IB	InfiniBand
Failed to open libibverbs.so, then Using network Socket	Ethernet

Build an image with InfiniBand

Run this command once, on one compute node. It needs about 6 GB free in ~. Use a new --container-name, not a name that you use elsewhere.

srun -N1 -c8 --mem=32G -t 20 --container-image=pytorch/pytorch:2.7.1-cuda12.6-cudnn9-runtime --container-name=ptib-build --container-remap-root --container-writable --container-save=$HOME/ptib.sqsh bash -c 'apt-get update -qq; apt-get install -y -qq libibverbs1 ibverbs-providers'

Then use --container-image=$HOME/ptib.sqsh. This image gives Using network IB across 2 compute nodes.

Disk use

Each saved image takes the disk of the compute node: about 6 GB, plus 3 GB of cache for the PyTorch image. When it fills, Slurm drains the compute node. Refer to Troubleshooting.

To remove a saved image:

srun -w node-2 -t 5 enroot remove -f pyxis_pt271

This does not clear the cache. To clear your cache on that compute node:

srun -w node-2 -t 3 bash -c 'rm -rf /var/lib/enroot/cache/user-$(id -u)'

It prints a harmless Permission denied for the empty folder.

Notebooks and TensorBoard

Run them in a job, and reach them through the login VM.

Warning:

These commands listen on all network cards of the compute node. TensorBoard has no password. Other people on the cluster can reach it. Stop the job when you finish.

1. Install JupyterLab and TensorBoard in your home folder. The image is from the section above.

   srun -N1 -c4 --time=10 --container-image=$HOME/ptib.sqsh --container-workdir=$PWD pip install --user jupyterlab tensorboard

2. Start JupyterLab in a job.

   srun -N1 -c4 --mem=16G -t 60 --container-image=$HOME/ptib.sqsh --container-workdir=$PWD bash -c 'export PATH=$HOME/.local/bin:$PATH; jupyter lab --no-browser --ip=0.0.0.0 --port=8898'

3. Leave this terminal open. The job ends when the terminal closes.

4. Find the compute node's name with squeue. In this example, it is node-1.

5. On your computer, start a tunnel in a second terminal.

   ssh -L 8898:node-1:8898 user1@203.0.113.10

6. Open the http://127.0.0.1:8898/lab?token=... line from the job's output in your browser.

TensorBoard works the same way. Use this command in the job: tensorboard --logdir=<folder> --host 0.0.0.0 --port 8897. To stop either one, run scancel.