MPI
The compute nodes have Open MPI, mpicc and mpirun. The login VM does not. Build your program on a compute node:
srun -N1 -n1 --time=2 mpicc hello.c -o hello
Start an MPI program
MPI programs need --mpi=pmix. Save this text as myjob.sbatch:
#!/bin/bash
#SBATCH -N2 --ntasks-per-node=2 --cpus-per-task=4 --time=30 -J myjob -o /home/jl_fs/user1/myjob-%j.out
srun --mpi=pmix ./hello
| How you start it | MPI program | Other programs |
|---|---|---|
sbatch, with srun --mpi=pmix inside | Works | Works |
srun --mpi=pmix from the login VM | Works | Works |
salloc, then srun --mpi=pmix | Works | Works |
salloc <command>, with srun --mpi=pmix inside | Works | Works |
mpirun inside an sbatch job | Works | Not relevant |
Plain srun, no --mpi | Fails at MPI_Init | Works |
srun --mpi=pmi2 | Fails: this Open MPI has no PMI-2 | Works |
Plain srun fails with OMPI was not built with SLURM's PMI support. That message points to the wrong cause. You do not need to build anything again. Add --mpi=pmix.
Use InfiniBand
By default, MPI between compute nodes uses Ethernet, with no warning. It is slow. To use InfiniBand with srun --mpi=pmix, add this line to the job before srun:
export OMPI_MCA_pml=ucx
With mpirun, add --mca pml ucx to the command instead.
InfiniBand is much faster than Ethernet for large messages.
Use this setting below 64 ranks. At 64 ranks or more, all-to-all programs can crash with it. Use the default for those jobs.
The crashes show MPI_ERR_TRUNCATE, Remote access or a segmentation fault.
To turn the setting off, remove the line, or run unset OMPI_MCA_pml.
Check which network MPI uses
Run this command in your job, with your own program:
OMPI_MCA_pml_base_verbose=10 UCX_LOG_LEVEL=info srun --mpi=pmix ./hello 2>&1 | grep -E "selected .* best|ep_cfg"
| Output | Network |
|---|---|
selected ucx and rc_mlx5/ibp... | InfiniBand |
selected ob1 | Ethernet |
One-sided MPI
MPI_Put and MPI_Win_create work by default, across compute nodes with srun --mpi=pmix.
MPI_Comm_spawn
MPI_Comm_spawn works under mpirun. Under srun --mpi=pmix, it fails with MPI_ERR_SPAWN.
MPI from Python
uv pip install mpi4py works in a venv. srun --mpi=pmix and mpirun both start mpi4py programs across 2 compute nodes.
MPI in a container
The compute node's MPI settings do not apply in a container. With no settings, a program hangs across 2 compute nodes. A small exchange can still pass, so a small test can hide the fault.
The image needs Open MPI. Use this form:
srun --mpi=pmix --container-image=$HOME/mpi.sqsh --container-workdir=$PWD --export=ALL,OMPI_MCA_btl_tcp_if_include=enp3s0,OMPI_MCA_oob_tcp_if_include=enp3s0,OMPI_MCA_pml=ucx,OMPI_MCA_osc=ucx ./your_program
enp3s0 is the private network card on H200 compute nodes. This form uses InfiniBand. Remove OMPI_MCA_pml=ucx,OMPI_MCA_osc=ucx to stay on Ethernet. Ethernet is much slower, and one-sided MPI then fails.
Refer to Containers to build an image.