Problems in Containers with 'No space left on device' due to limited TMP directories¶

Problem:

For example, you want to use a Apptainer/Singularity container in the shell mode (with the contain flag set) to compile (within the container) a program. However, your compilation fails with an error like

error: error writing to /tmp/cceNuA3r.s: No space left on device

but (outside) the container you have sufficient space in your $HOME directory etc.

(for problems with not enough space while building a container see Apptainer/Singularity Cache and TMP Directories ) Solution

Your compilation needs sufficient space in temporary directories like /tmp (in the container) or /var/tmp (in the container) for some intermediary files. However, Apptainer/Singularity limits by default the available space in /tmp or /var/tmp to avoid problems, where container runs clutter temporary files.

So, we will start the Apptainer/Singularity container and point its temporary directories to directories beneath, that have more space available (there are also other options available, but we go for the bind-mount for educational reasons).

create explicitly TMP directories under your DUST scratch path (or some other file system, where you have sufficient free space

 mkdir -p /data/dust/user/YOURUSERNAME/scratch/tmp
 mkdir -p /data/dust/user/YOURUSERNAME/scratch/var/tmp

run the Apptainer/Singularity container and bind-mount these directories onto /tmp and /var/tmp in the container with the flag --bind SOURCE:DESTINATION

singularity shell --bind /data/dust/user/YOURUSERNAME/scratch/tmp:/tmp --bind /data/dust/user/YOURUSERNAME/scratch/var/tmp:/var/tmp   ...YOUR_OTHER_SINGULARITY_OPTIONS..

Now all file operations under /tmp or /var/tmp (in the container) would end up (outside the container) on your DUST directories Warning

Do not share these directories between containers. If container X and container Y would use the same temporary directories, nasty things can be expected to happen! Alternatively

With the '--workdir /path/to/DUST/dir' flag, the directories (in the container) /tmp, /var/tmp and $HOME are put into this workdir directory (on the outside) by Apptainer/Singularity already.