Singularity/Apptainer support in BIRD¶
Apptainer/Singularity enables users to have full control of their environment. Apptainer/Singularity containers can be used to package entire scientific workflows, software and libraries. This means that you don’t have to ask your cluster admin to install anything for you - you can put it in a Apptainer/Singularity container and run.
Do not use HTCondor Container standard setup
Make sure to stick to the syntax below - do NOT set the universe to "Docker" or "Container" (stick to "Vanilla" instead), do NOT use the "container_image" key ! It will not work in the NAF.
To use Apptainer in BIRD please use (in your config file):
+MySingularityImage = "/path//container/sl6" # example path to your singularity image
Optional:
+MySingularityArgs = "--my-args"
Can be used to put additional arguments into the Apptainer call on the workernode, only use this options if you know what you are doing, check your additional options against the given options in the Apptainer call (see below in the test example, oll FS paths are added by us) and test your individual options with a single job before submitting bigger clusters.
The image can be located in any of the shared filesystem (CVMFS/DUST/AFS)
Testing a Apptainer image¶
to test your Apptainer image login on a remote-submit host and use the following:
apptainer exec --contain --bind /afs:/afs --bind /nfs:/nfs --bind /data:/data --bind /pnfs:/pnfs --bind /cvmfs:/cvmfs --bind /var/lib/condor:/var/lib/condor --pwd </working/directory/of/job><--your-singularity-args> </path/singularity:image> <your executable and args>
An alternative method for maybe further debugging would be an interactive job (condor_submit -i) which will log you into the Apptainer jobcontainer.
Submitting jobs from inside containers¶
currently job submission from inside the CMS EL7 container is tested and possible, a prerequisite for using the below concept in other containers is that htcondor binaries are available inside the container you are using. (thanks got to JP for sharing this soultion !)
Here is ane example start script using the CMS EL7 image:
#!/bin/bash
export APPTAINER_BINDPATH=/afs,/cvmfs,/data/dust/user:/data/dust/user,/pnfs,/cvmfs/grid.cern.ch/etc/grid-security:/etc/grid-security,/cvmfs/grid.cern.ch/etc/grid-security/vomses:/etc/vomses,/etc/pki/ca-trust,/run/user,/tmp,/var/run/user,/etc/sysconfig,/etc:/orig/etc,/etc/krb5.conf:/etc/krb5.conf,/etc/krb5.conf.d:/etc/krb5.conf.d,/var/lib/sss/pubconf:/var/lib/sss/pubconf,/etc/krb5.keytab:/etc/krb5.keytab,/var/kerberos:/var/kerberos,/etc/crypto-policies/back-ends:/etc/crypto-policies/back-ends,/etc/condor-ce:/etc/condor-ce,/etc/condor:/etc/condor,/var/lib/condor:/var/lib/condor,/usr/bin/condor_docker_enter:/usr/bin/condor_docker_enter,/lib64/libcondor_utils_23_0_17.so:/lib64/libcondor_utils_23_0_17.so
apptainer -s exec --env PS1="$PS1" --no-home --bind /data,/cvmfs,/pnfs /cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/cms-cat/cmssw-lxplus/cmssw-el7-lxplus:latest/ sh -c "export _condor_schedd_host=bird-htc-sched21.desy.de && export _condor_schedd_name=bird-htc-sched21.desy.de && export _condor_condor_host=bird-htc-master21.desy.de && export _condor_credd_host=bird-htc-sched21.desy.de && export _condor_default_domain_name=desy.de && export _condor_uid_domain=desy.de && export _condor_filesystem_domain=desy.de && export _CondorPlatform=\"$CondorPlatform: x86_64_RedHat9 $\" && export _condor_sec_credential_producer=/var/lib/condor/util/get_token_cmd && /bin/bash -l"
The libcondor_utlis version will change over time
in that case you will need to check the actual version using the 'ls' cmd and adjust it inside the 'export' part of the start_cc7 script !
Additional options in the submit file of the jobs needed (will not work without these):¶
Request_OpSysAndVer = "RedHat9"
+CondorPlatform = "$CondorPlatform: x86_64_RedHat9 $"
Optional, if you want your job to run inside the image:
+MySingularityImage = "/cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/cc7:amd64"
Apptainer images¶
Apptainer is a user space based container system that does not require root access.
Please make sure, that a container (and the software versions it contains) is valid for your use case - as you would do for a software release
Use 'unpacked' versions of container images whenever possible as these images consume noticeably less memory than equivalent 'full blown' images.
CMS EL8 image¶
/cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/cc8:amd64
For simplified EL8 container usage for CMS see: EL8 Support for CMS
Known issues¶
Sometimes the apptainer call fails with
FATAL: container creation failed: mount hook function failure: mount /afs/desy.de/user/x/xxx->/var/lib/apptainer/mnt/session/afs/desy.de/user/x/xxx error: while mounting /afs/desy.de/user/x/xxx: destination /var/lib/apptainer/mnt/session/afs/desy.de/user/x/xxx doesn't exist in container
that's because the symlink in AFS home is irritating from time to time apparently, see here for a temporary solution: