Storage

Where to put your files

What	Where	Why
Code, data, checkpoints, results	Your home folder, ~ (/home/jl_fs/<login-name>)	All VMs see it. It stays when you delete a compute node or the cluster.
Temporary files in a normal job	/tmp or /var/tmp	These are the places for your own files on the compute node's disk.
Temporary files in a container	A folder that you mount, such as --container-mounts=/var/tmp:/scratch	A container's own /tmp is RAM, and it counts against your job's memory.

What is kept and what is lost

Event	Files in /home/jl_fs	Files in /tmp and /var/tmp	Saved containers
Job ends	Kept	Kept. Slurm does not clean them.	Kept
Reboot or restart of the compute node	Kept	Do not rely on them.	Kept
Delete the compute node	Kept	Lost	Lost
Delete the cluster	Kept	Lost	Lost
Delete the shared filesystem	Lost	Not affected	Not affected

Warning:

The shared filesystem is the only place that stays when you delete a cluster. Put all data that you need in your home folder.

The shared filesystem

The filesystem is at /home/jl_fs, on the login VM and all compute nodes. Each person's home folder is private to that person. Admins can read it with sudo.

Check the free space with this command:

df -h ~

There are no limits per person, and only an admin can see who uses the space. A venv takes about 5 GB, and a saved image takes about 6 GB. To delete a venv and its cache, refer to Software environments.

You can grow the filesystem on the Shared Filesystems page while jobs run. The new size shows at once. The filesystem's ID changes when you grow it, so a script that holds the old ID fails. You cannot shrink a filesystem.

A folder for the whole team

Each person's home folder is private. To share files, an admin makes one folder that all people can write to. Only a file's owner can delete that file.

sudo mkdir /home/jl_fs/shared && sudo chmod 1777 /home/jl_fs/shared

Group folders do not work. A job does not get a team's Linux group. Containers see this folder only if you add --container-mounts=/home/jl_fs/shared:/home/jl_fs/shared.

The safe way to use the shared filesystem

A read of a file that another VM wrote or replaced a short time ago can stall. Most stalls last from a few seconds to about a minute, and they clear by themselves. A few last much longer. Reads of other files stay fast during a stall.

While it stalls, the process does not respond and you cannot kill it. If its job ends in that time, Slurm drains the compute node. The reason is Kill task failed. The compute node then takes no new jobs, and you still pay for it, until an admin acts. Refer to Troubleshooting.

These rules reduce the risk.

Replace and poll

• Do not replace a file by rename while another VM reads that file in a fast loop. Opens can fail with Stale file handle, and a read can stall.
• Do not append to a file on one VM while another VM reads that file in a loop.
• Read a finished file once.
• If your program reads a "latest checkpoint" file that another VM writes, retry when it gets Stale file handle.

Python packages

Compile Python packages on one compute node before the first run on several compute nodes. Two compute nodes that import a new package together can both write Python's compiled files, and that can stall.

python -m compileall -q -j 4 ~/venv/lib/python3.10/site-packages

Do not install packages while a job on another compute node imports from the same venv.

File locks

File locks do not work between VMs. Two compute nodes can both get a lock on the same file.

• Do not write one SQLite file in ~ from several compute nodes. The file can get damage or lose commits. Use one compute node, or put the file in a compute node's /tmp.
• Fill Hugging Face and similar caches from one compute node first, with a 1-node job. After that, 2-node jobs work.
• A file made with O_EXCL is safe across compute nodes.

Data pipelines

A job can stall when it reads many files that other compute nodes wrote seconds before. The stall can drain the compute node. If a pipeline job does not end, check the compute node's reason with scontrol show node <name>.

Speed

Small writes to /home/jl_fs are slow, so write in large pieces. Many small files run much faster in /tmp. For large files, /tmp is not reliably faster.

Once a compute node has read the data, /tmp and /home/jl_fs read at the same speed. So a copy to /tmp helps only the first epoch, or a job with many small files.

Deletes of many small files are slow on the shared filesystem.

The compute node's disk

You set this disk's size at create. The system and all people's container images and /tmp files share it. So a little less than the disk size is free in /tmp.

About /tmp:

• Slurm does not clean it when your job ends. Delete your files in your job script.
• Other people's jobs on that compute node can read your files if the file permissions allow it.
• If it fills, Slurm takes the compute node out of service. Refer to Troubleshooting.
• You lose the whole disk when you delete the compute node.

--tmp asks for a compute node whose /tmp disk has at least that total size. It does not look at free space, and it does not reserve any. A request larger than the disk gets a refusal: Temporary disk specification can not be satisfied.

/dev/shm holds RAM. Slurm does not clean it after a job, and all people on the compute node share it. Delete your files there in your job script.

The login VM's disk

You set this disk's size at create. Home folders are not on it, because they are on the shared filesystem. Do not rely on it for data.