Container run error

The symptomatic is that running a container in apptainer fails with the error:

FATAL: container creation failed: mount hook function failure: mount /afs/desy.de/user/x/xxx->/var/lib/apptainer/mnt/session/afs/desy.de/user/x/xxx error: while mounting /afs/desy.de/user/x/xxx: destination /var/lib/apptainer/mnt/session/afs/desy.de/user/x/xxx doesn't exist in container

We were able to debug the reason behind it to the fact that apptainer does sometimes, spontaneously not like the symlinks inside the path to your home directory in AFS.

The problem occured before and was fixed at the time (in EL7 context) and we are in contact with auristor, hopefully the outcome will be another fix at one point.

For now we have a bushfix for it, tell apptainer to ignore your $HOME und explicitly just mount '/afs' by using

"-B /afs --no-home"

in the apptainer call. The result will be a fully normal behaviour. Here is an example:

[chbeyer@naf-cms14]~% apptainer exec -B /afs --no-home /cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/el7:x86_64 sh -c /bin/bash Singularity> pwd /afs/desy.de/user/c/chbeyer