Singularity installation requires root, so, our admins need to install it in our sys branch as described here.
Local changes done by root include:
- create /var/lib/singularity/mnt on each Linux machine that's expecting to run container images - should be in Spacewalk
- sudo access for "singularity" command for User Services
Local changes done by hpcapps:
- create singularity module file - just adding path to singularity bin directory
To create a container, one needs to first create the image:
sudo singularity create --size 4096 ubuntu_$imgname.img
and then bootstrap (install) the OS, and other needed programs:
sudo singularity bootstrap ubuntu_$imgname.img ubuntu_$imgname.def
I prefer to have a script, called build_container.sh that calls these two commands.
The container definition file describes the bootstrap process, described here.
To create a new container, the easiest is to get one of the definition files and modify accordingly. Singularity has some examples here, or start from the examples on this git page.
Effort should be made to make the container building non-interactive, so they can be automatically rebuilt. Singularity developers also encourage doing everything from the def file, rather than launching singularity exec to add stuff to the container.
The strategy that I found works reasonably well is to bootstrap the base OS image, singularity shell into the container and then manually execute commands to build the particular package, while writing them down in a shell script. Oftentimes the packages have some kinds of shell scripts that install dependencies, and the program itself. Though, it can take time to iterate over and fix issues, mostly related to missing dependencies. Once things work, paste commands from this shell script as a scriptlet to the %post section of the def file.
To launch a shell in the new image, sudo singularity shell -w -s /bin/bash myimage.img, -w makes the image writeable, -s makes shell bash (easier to use than default sh). In the container, use apt-get or yum to install the required dependencies (when something is missing, google what package contains it), and finally wget the install files for the program, or download them to a local directory, and add "-B `pwd`:/mnt" to the singularity shell command to mount the local directory under /mnt in the container.
Once the application in the container is installed, and the scriptlet in the def file to do this installation is written, build the container again. If there's an error, fix it and iterate over, until the container builds with no error.
If install files need to be brought in from the host OS, including files that need to be downloaded interactively (e.g. CUDA installer) use the %setup section, which runs on the host. To put files in the container, use ${SINGULARITY_ROOTFS}. E.g. to put files to container's /usr/local, put it to ${SINGULARITY_ROOTFS}/usr/local. Example of this is our tensorflow def file.
To test the installation, use the %test section to put there commands that run tests.
- make sure to create mount points for CHPC file servers:
mkdir /uufs /scratch
- additions to default environment (PATH, LD_LIBRARY_PATH) can be put to /environment file in the container, e.g.
echo "
PATH=my_new_path:\$PATH
export PATH
" >> /environment
- if a non-root installation is needed (e.g. LinuxBrew), then create the non-root user and use
su -c 'command' userto do the non-root stuff, e.g.
useradd -m brewuser
su -c 'brew install freetype --build-from-source' brewuser
See our bioBakery def file for full definition file that shows this.
- it can be confusing to see if one is in a container or not. We can modify our prompt to reflect we're in a container, e.g. put something line this in your ~/.bashrc:
if [ -n "$SINGULARITY_CONTAINER" ] ; then
# set prompt = "[$USER@%m:%b%c2]%b "
PS1="$(lsb_release -i | awk '{ print $3; }')[\u@\h:\W]\$ "
else
# set prompt = "[$USER@%m:%b%c2]%b "
PS1="[\u@\h:\W]\$ "
fi
I also sometime run this to see if Singularity environment variables are defined, which tells that one is in the container: env|grep SING.
%testsection does not seem to bring environment from /environment created in%postsection, so, make sure to define PATH and LD_LIBRARY_PATH in the%testsection before running tests.- the
%postsection starts at/directory, so, cd to some other directory (e.g./root) before building programs. - to support NVidia GPUs in the container, one needs to instal a few NVidia driver libraries of the same version as the host driver. To find the version, run
rpm -qa | grep nvidia. Then either follow our tensorflow def file or bring libcuda.so and libnvidia-fatbinaryloader.so from the host. - to support InfiniBand, need to install the IB driver stack in the container and make sure the driver sos are in the LD_LIBRARY_PATH (see the ubuntu_mpi container recipe for details).
- Singularity container inherits the environment from the host shell, including PATH. One needs to be aware of this when setting things up. E.g. starting the container from a fairly clean environment may be a good idea. The only thing that it does not inherit are -- LD_LIBRARY_PATH -- shell functions (e.g. LMod defines its commands via shell functions)
- supporting modules (LMod) requires separate LMod installation for Ubuntu based containers and a few other modifications, detailed below
Singularity container is an executable so it can be run as is (which launches whatever is in %runscript section), or with singularity exec followed by the command within container, e.g.:
singularity exec -B /scratch debian_SEQLinkage.img seqlink [parameters]
singularity run debian_SEQLinkage.img
If more than one command are needed to be executed in the container, one can run singularity shell, e.g.
singularity shell -s /bin/bash -B /scratch /ubuntu_biobakery.img
To specify the shell to use (default /bin/sh is not very friendly), use the -s flag or environment variable SINGULARITY_SHELL=/bin/bash.
- home directory gets imported (bound) to the image automatically as long as the
/uufsmount point is created. - all scratches get imported when using
-B /scratchoption - to bring in sub-directories of
/uufs, such as the sys branch, add-B /uufs/chpc.utah.edu.
That is, to bring in home dir, scratches and the sys branch, we'd launch the container as
singularity shell -B /scratch -B /uufs/chpc.utah.edu -s /bin/bash ubuntu_tensorflow_gpu.img
Alternatively, use environment variable SINGULARITY_BINDPATH="/scratch,/uufs/chpc.utah.edu".
Singularity supports MPI (including IB), as described at http://singularity.lbl.gov/docs-hpc. In this description they discuss OpenMPI (> 2.1 which we don't have installed yet) but we have tested this with the lastest MPICH variants (MPICH, IMPI), and it works as well
In essence, if one is on a single node, it does not matter if the MPI program is called in our out of the container, but, for multi-node calculations, especially if run through SLURM, the recommended way is to have the mpirun to execute the container, which in turns executes the MPI program.
For example how to roll in IB support in a container, see our Ubuntu MPI container. On can add LMod as described below instead of sourcing the Intel stack directly, we have verified Intel MPI to work using the IB.
As for utility of running MPI containers, I can't think of any now, unless the application is really difficult to build, stick to host based execution.
A concern in the Python based containers was how numPy and sciPy performance would be using Ubuntu stock libraries. We did some testing and found that stock Ubuntu OpenBLAS performance is on par with MKL, except for AVX2 enabled CPUs where MKL is about 30% faster for matrix multiply.
- for details, see our Ubuntu Python container.
Modules support pulling programs from CHPC sys branch may be useful for some containers. In particular, we can use the Intel compiler/MPI stack to build MPI programs in the container, or use Intel Python Distribution. Both are built in distro-agnostic fashion so if installed on CentOS, they work on e.g. Ubuntu.
This description is for Ubuntu based containers, for CentOS, sys branch LMod works with the CHPC CentOS7 LMod installation after re-initializing LMod (because the shell functions don't get passed to the container). For Ubuntu containers, we need to do the following:
- have LMod installed in the sys branch using Ubuntu - can be done from the container -- now /uufs/chpc.utah.edu/sys/installdir/lmod/7.4-u16/ -- need to have specific Ubuntu install since some commands (tr, lua) and Lua libraries have different location on Ubuntu than on CentOS
- when building the container, set environment variable SINGULARITY_MOD=1
- the user needs to have the following at the end of ~/.custom.sh:
export OSVER=`lsb_release -r | awk '{ print $2; }'`
export OSREL=`lsb_release -i | awk '{ print $3; }'`
if [ -n "$SINGULARITY_CONTAINER" ] && [ -n "$SINGULARITY_MOD" ]; then
if [ $OSREL == "CentOS" ]; then # assume only CentOS7
source /uufs/chpc.utah.edu/sys/installdir/lmod/7.1.6-c7/init/bash
elif [ $OSREL == "Ubuntu" ]; then # assume only Ubuntu 16
source /uufs/chpc.utah.edu/sys/modulefiles/scripts/clear_lmod.sh
source /uufs/chpc.utah.edu/sys/installdir/lmod/7.4-u16/init/profile
fi
fi
- the container needs to be started with binding the sys branch, i.e. with
-B /uufs/chpc.utah.edu
For example of container that has the LMod support built in, see Ubuntu Python container.
X works without any major hassle. See the Paraview container for example. The only catch, which may need to be looked at in the future, is the lack of hardware OpenGL acceleration.
- copy the definition file and other potential needed files to the srcdir
- copy the container image (img file) to installdir
- create module file that wraps the package call through the container, for example see SEQLinkage module file.
- create SLURM batch script example, for example see [SEQLinkage batch script]/(https://github.com/mcuma/chpc_singularity/blob/master/seqlinkage/run_seqlink.slr) -- NOTE that LMod does not expand alias correctly in bash non-interactive shell, so, use tcsh for the SLURM batch scripts until this is resolved