Running Underworld docker containers on HPC with Shifter

Romain Beucher

Jun 20, 2018

Use docker container on PAWSEY MAGNUS

One image to rule them all…

The Underworld code development team has been actively promoting usage of Docker containers Read More here.

Software and data containerization are also efficient ways to ensure the reproducibility of Underworld models and research workflows. Containers such as Docker containers are essentialy a folder which contains files and a way to run a process in that folder. This means that containers can be released in a version controlled system and used to track changes and reproduce experiments.

On a more technical note they have proven to be an efficient solution to replace the somewhat painfull compilation process across multiple platforms. Containers allow to quickly deploy an application regardless of the environment in which it actually runs. For the users, it means that there is no need to worry about dependencies and compilation. The application is immediately ready to run whether you are running on MacOSX, Linux or Windows…

The technical advantages of containerization explain their rapid adoption by large companies such as Google or Facebook, etc. Adoption from the High Performance Computing scientific community has been relatively slower mainly because usage of containerization often requires the user to have elevated access on the machine. (You can read more about that subject here.) A tension also appears in that HPC machines require bespoke MPI configurations, while containers seek to provide platform independent runtime environments. Thus the deployment of containers on HPC machines, although technically possible, presents various technical and security challenges.

Several solutions have been proposed to remove the need for elevated access. Some take advantage of user namespaces available since Linux Kernel 3.8, (e.g runC, Charliecloud). Others still needs to be installed by an administrator (e.g. Singularity, Shifter, BubbleWrap) and provide a portal to access and execute containers without requiring elevated access from the user.

Shifter has been recently made available at the Pawsey Supercomputing Centre in Perth, WA. We have released a docker container image on Docker hub that can be used on the MAGNUS system. The Underworld development team and users at the University of Melbourne Computational Geodynamics Group have been very excited by this opportunity which, we think, is an important milestone towards achieving a universal Underworld container.

In the following blog post I will detail how to use that image to run Underworld models on that system.

Prerequisites

The following assumes that you have access to MAGNUS. We suggest reading our previous post on Docker and containers.

Getting started with Shifter

You will first neet to log in to magnus:

$ ssh your-user-name@magnus.pawsey.org.au

It’s also worth having a browse through the Magnus Shifter documentation page

Once you have logged in, Shifter module needs to be loaded:

$ module load shifter

Underworld Docker Images available on Magnus

You can list all images available using the following command:

$ shifter images

You will see that there is already a few underworld images available…

$ shifter images
magnus     docker     READY    19db604b56   2018-05-12T11:07:44 dansand/underworld2-dev:latest
magnus     docker     READY    e0650fc8c2   2018-06-13T14:47:33 rbeucher/underworld2_geodynamics:magnus
magnus     docker     READY    090727c2b1   2018-06-20T14:53:47 underworldcode/underworld2:2.5.1b_magnus
magnus     docker     READY    6027a4e870   2018-03-22T08:41:45 underworldcode/underworld2_untested:dev_magnus_test
magnus     docker     READY    cd718ff470   2018-04-04T10:23:28 underworldcode/underworld2_untested:test_2.5_magnus

Pulling an image from DockerHub

If you want to use your own image we recommend using the underworldcode/underworld2:2.5.1b_magnus image available in our dockerhub.

As an example, here is how to pull the most recent image for rbeucher/underworld2_geodynamics

$ shifter pull rbeucher/underworld2_geodynamics:magnus

Running an Underworld script on MAGNUS using Shifter

Once the image is available on MAGNUS you are pretty much all set.

All you need is an Underworld python script (.py) and a correct Slurm job submission script. A minimal example for a Slurm script (slurm.sh) running an Underworld script named *Underworld_script.py” on 2 nodes using 12 threads would be:

#!/bin/bash

#SBATCH --nodes=2
#SBATCH --time=24:00:00
#SBATCH --account=q97

# This is were you define the docker image to be used to create the container
IMAGE=rbeucher/underworld2_geodynamics:magnus
SCRIPT=./Underworld_script.py

module load shifter
srun -n 12 shifter run --mpi $IMAGE python $SCRIPT

You can then run your model by using the following command.

$ sbatch slurm

And that’s it!!!

Conclusion

Shifter provides an easy to use solution for running Underworld containers on PAWSEY MAGNUS. It allows the user to run the images provided by the Underworld team without the need to request an install from the PAWSEY IT support.