1616342 : Apptainer not working for SOME jobs

Created: 2026-07-10T14:36:38Z - current status: new

Summary

A user reports intermittent Apptainer container failures affecting >10% of submitted jobs (~30 out of ~300). The errors occur during container creation due to permission issues when trying to mount /afs/desy.de. The specific error is:

FATAL: container creation failed: mount hook function failure: hook function for tag layer returns error: failed to create /afs/desy.de directory: mkdir /afs/desy.de: permission denied

Possible Solution

Based on known issues documented for NAF’s Apptainer support (source), this may relate to symlink handling in AFS paths. A workaround is to explicitly bind /afs while disabling automatic $HOME mounting:

apptainer exec -B /afs --no-home [other_args] [image] [command]

Alternatively, ensure the container has proper permissions for /afs or check if --fakeroot resolves the issue (as suggested in the error message).

Next Steps

  • Test with -B /afs --no-home in the Apptainer command.
  • Verify if the issue persists across different batch nodes (errors occurred on batch1185, batch1254, batch1370, batch1460).

Sources: - NAF Apptainer Support Documentation - Container Run Error FAQ