Software environments

Install software in a job on a compute node, and keep it in your home folder. Then it is on all VMs. The login VM cannot build software.

Compute nodes have gcc, make, mpicc and mpirun, and no pip. python3 -m venv fails on the VMs. module is not available. Use uv.

Which method to use

Method	Use it for
uv venv	Most work. It is the simplest method on one compute node and across several compute nodes.
Container	A fixed image. You also get pip inside it. Refer to Containers.
conda	Packages that are only on conda.

Make a venv with uv

Save this text as setup.sbatch, then run sbatch setup.sbatch on the login VM.

#!/bin/bash
#SBATCH -N1 -n1 -c16 --mem=32G --time=30 -J setup
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH=$HOME/.local/bin:$PATH
uv python install 3.10
uv venv --managed-python --python 3.10 ~/venv
source ~/venv/bin/activate
uv pip install torch==2.7.1 --index-url https://download.pytorch.org/whl/cu126
uv pip install numpy
python -m compileall -q -j 4 ~/venv/lib/python3.10/site-packages

Each part has a reason:

Line	Reason
--managed-python	It brings Python's headers. vLLM and DeepSpeed need them. Without them, you see Python.h: No such file or directory. This venv also runs on the login VM.
uv pip install numpy	Without it, PyTorch shows Failed to initialize NumPy.
python -m compileall	It writes Python's compiled files now, so 2 compute nodes do not write them together on the first run.

A venv has no pip. Use uv pip install.

A venv made on a compute node

A venv that uses the compute node's Python does not run on the login VM. You see venv/bin/python: No such file or directory. After activate, you see python: command not found. The compute nodes and the login VM have different Python versions.

A venv made with --managed-python runs on both. Use it if you want to run small scripts on the login VM.

Activate the venv in the job

Put source ~/venv/bin/activate in your job script, as in Your first job.

Caution:

Do not install packages while a job on another compute node imports from the same venv. Its imports can fail with ModuleNotFoundError.

Delete a venv and its cache

A venv delete frees little until you also delete uv's cache. The two share files.

rm -rf ~/venv
rm -rf ~/.cache/uv

Deletes are slow on the shared filesystem. Check the result with df -h ~.

pip in a container

pip works in a container, and it installs to ~/.local. You build ptib.sqsh in Containers.

srun -N1 -c4 --time=10 --container-image=$HOME/ptib.sqsh --container-workdir=$PWD pip install --user jupyterlab tensorboard

conda

Install Miniforge in a job. It needs about 650 MB and many small files. Run these lines in a job script:

curl -sSL -o /tmp/mf.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash /tmp/mf.sh -b -p ~/mf
~/mf/bin/conda create -q -y -n t python=3.11 numpy

Ray

Ray needs --node-ip-address set to each compute node's private IP address. Without it, the second compute node does not join, and you see no error.

1. Find the private IP address of each compute node.

   getent hosts node-1

2. Start the head on the first compute node, from inside a job that holds both compute nodes.

   srun -N1 -n1 --overlap --nodelist=node-1 ray start --head --node-ip-address=<head IP> --port=6379 --block

3. Start the other compute node, from inside the same job.

   srun -N1 -n1 --overlap --nodelist=node-2 ray start --address=<head IP>:6379 --node-ip-address=<its IP> --block

Tools that are not installed

These tools are not on the VMs: aws, rclone, gsutil, azcopy, unzip, zip, sqlite3, dos2unix and pv. Java is not on them either. Install what you need in your home folder.