Building Containers (Docker, Apptainer/Singularity)¶
Singularity to Apptainer renaming¶
Singularity has been renamed to Apptainer in 2022 due to legal constraints. In general, just the name has changed and all options are the same. Replacing the command singularity with apptainer should work on updated systems.
Preliminary Remark¶
While a Apptainer container can be run by an 'ordinary' user on Linux, building a fresh container requires sudo, so administrative rights to to a bit of magic during the build. Obviously, sysadmins will not give users such rights on shared resources (wink)
To build and run a Docker container, a user has to be member of the local 'Docker' group. Unfortunately, the capabilities of this 'Docker' group are quite mighty comparable to sudo in some aspects - to quote the Docker documentation "Only trusted users should be allowed to control your Docker daemon" So, security minded sysadmins will be reluctant to grant such powers to users. Recipes
Both, Docker and Apptainer, containers are created from a template or recipe containing all the steps to setup the container (classically called Dockerfile for Docker and Apptainer recipe for Apptainer).
There is a wide variety on examples out there on how to write a Dockerfile or a Apptainer recipes, so we will have only an intrudcution here Dockerfile
A Dockerfile is somewhat "line oriented", Each line starting with a Docker command will result in it's own "layer", and all layers in the end combined and seen as a single container image (technically, you have all these layers separately, which makes moving "a single image" around a bit cumbersome - but has the advantage that if you change only one line in the Dockerfile, only this section needs to be build again and the rest gets recycled)
cat Dockerfile
## from where do we bootstrap?
FROM centos:6
## always add comments and meta data so you know in two weeks, why you did things
MAINTAINER Thomas Hartmann @ DESY
## use RUN to do things in the container, e.g., create directories
RUN mkdir -p /etc/yum.repos.d/
## copy some files from the host namespace into the container
COPY host/etc/yum.repos.d/ /etc/yum.repos.d/
## and run additional commands in the container as updating the repository information
RUN yum -y update
## and each step bekomms another layer
RUN yum install -y curl
## so if you only change the previous one and keep all the rest, during a new build only the last step will be rerun
To create a new container from the Dockerfile run
(sudo) docker build -f /path/your/Dockerfile
Apptainer Recipe
A recipe for a Apptainer container is not much different from a Dockerfile except that it is not line/command oriented cat Apptainer
## generic things again: where do we bootstrap from, here we build from scratch
Bootstrap: yum
OSVersion: 6
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum
## alternatively you can also start with a Docker container as starting point, e.g.,
# Bootstrap: docker
# From: centos:6
## meta data go in the %label section
%labels
Maintainer Thomas Hartmann @ DESY
Version 0.0.1
## copying files from the host namespace into the container happens in the %file section
%files
host/etc/yum.repos.d/* /etc/yum.repos.d/
## and all the things you want to run in the container go into the %post section
%post
mkdir -p /etc/yum.repos.d/
yum -y update
yum install -y curl
To create a container image "mycontainer.apptainer" run
sudo apptainer build mycontainer.apptainer Apptainer.file
This will walk through the whole recipe and create a single file "mycontainer.apptainer", which you can copy around and can run as a user, e.g.,
apptainer shell --options... mycontainer.apptainer
As Apptainer needs to mount some auxiliary file systems and needs to use some kernel features, building a fresh container needs root/sudo permissions on the build host. E.g.,
> apptainer build --sandbox MyContainerName.d Apptainer
It is good practise (and strongly recommended) to write a proper Apptainer recipe file stating all the steps that define the container.
If you build up your container by running it as writable sandbox container from some base image and install/config stuff manually, you will end up with an unmaintainable blob. Take the advice from a sysadmin and take your time to think and construct your container as a template - your future self will thank you! Fixing Permissions
When you build a sandbox container under root, files in the resulting container's directory tree will probably belong to root. So, we better change the ownership of all files in the container to our own user
> sudo chown -R YOURUSER:YOURGROUP MyContainerName.d
Additional we can make the files and directories readable for all, if we plan to share the container with others
> chmod -R a+r MyContainerName.d
> find MyContainerName.d -type d -exec chmod 0755 {} \;
Fat Containers vs. Sandbox Dir Containers
When building a container, Apptainer will normally create a single image file, i.e., containing all the containers files in a packed blob. This has the advantage to be smaller and instantly portable.
However, the large disadvantage is, that to run in/from the container the whole image blob has to be loaded. That is probably not an issue, if you run just locally somewhere, but will increase latency etc. pp. when you reside on a network filesystem or cache from CVMFS. Better use in such cases the '--sandbox' flag during builds, i.e., the container will be just an expanded directory tree with all files lying around on their own.
For example
> apptainer exec --cleanenv --contain /cvmfs/grid.desy.de/container/sl6.d/ cat /etc/redhat-release | wc
1 4 28
> ls -hall /cvmfs/grid.desy.de/container/sl6/bin/cat
-rwxr-xr-x 1 cvmfs cvmfs 45K Jun 19 17:15 /cvmfs/grid.desy.de/container/sl6.d/bin/cat
will just cache from CVMFS the 'cat' binary in the container (plus the helper files around) from CVMFS and run it in the container's environment, where the output is send to stdout and can be further processed in your native environment.
In comparison, for ATLAS' fat SL6 container
> ls -hall /cvmfs/atlas.cern.ch/repo/containers/images/apptainer/centos6-20180227012153-b83ffa8a16125230f106f29c07c681cddfcebf75151deb294f8a69cbfad35b4f.img
-rwxr-xr-x 1 cvmfs cvmfs 377M Feb 27 2018 /cvmfs/atlas.cern.ch/repo/containers/images/apptainer/centos6-20180227012153-b83ffa8a16125230f106f29c07c681cddfcebf75151deb294f8a69cbfad35b4f.img
the whole 377M have to cached first on the host by CVMFS, before it can be executed just to run the 45k 'cat' in it...
(I prefer to name my containers with a trailing ".d" if they are sandbox containers, e.g., "condormon.d" - but that's just a matter of taste) Shipping Sandbox Containers
If you plan to send a sandbox container somewhere else, pack its directories in one tarball to dodge it's advantage/disadvantage being not already a single file blob
> tar -cJvf MyContainerName.d.tar.xz MyContainerName.d
to unpack the tarball, run
> tar -xJvf MyContainerName.d.tar.xz MyContainerName.d
but also starting on a packed container tarball works (of course with some unpacking overhead)
> apptainer shell MyContainerName.d.tar.xz
Building Apptainer Containers in Gitlab¶
Since building a Apptainer container requires root capabilities or a uid/gid mapping, it might be easier to use Gitlab at https://gitlab.desy.de to build your containers there.
Please see for details https://gitlab.desy.de/thomas.hartmann/apptainerbuilder
Quick&Dirty build machine¶
If you need a machine to build a Apptainer container but you don't have a Linux machine at hand, where you can become root/do sudo, a way around can be a virtual machine. Install Apptainer
Apptainer is available in the epel repository.
Check if you have this repo in your /etc/yum.repos.d/ directory
grep -i "epel" /etc/yum.repos.de/*
Only if you don't find the EPEL repository, install it with
yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
With epel, you can install apptainer straight forward
> yum clean all; yum update
> yum install -y apptainer
Installing CVMFS¶
If you need files from CVMFS, you will have to do some more additional steps... your millage may vary and depend on your local environment. To use CVMFS you will need a squid proxy in your network to talk to when asking for files in CVMFS - i.e., check in a machine in the same network with working CVMFS, what it's config file '/etc/cvmfs/default.local' says.
> yum install -y cvmfs cvmfs-config-default
# create/copy a config file
# e.g., if you are at CERN, you might be successful, if you copy the config file from a lxplus node
> /etc/cvmfs/default.local
# add cvmfs to the automount config
> echo "/cvmfs program:/etc/auto.cvmfs" >> /etc/auto.master
# enable and start the automounter unit
> systemctl enable autofs
> systemctl start autofs
# (reboot the box)
# run a test, if your box can reach the configured CVMFS repositories
> cvmfs_config probe