containerit.Rmd

---
title: "Generating Dockerfiles for Reproducible Research with R"
author: "Matthias Hinz, Daniel Nüst"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
#output: # use for blog post
#  md_document:
#    toc: true
#    toc_depth: 2
#    variant: markdown_mmd
vignette: >
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteIndexEntry{Generating Dockerfiles for reproducible research with R}
  %\VignetteEncoding{UTF-8}
---


```{r "setup"}
require("knitr")
temp_workspace <- tempfile(pattern = "containerit_temp")
dir.create(temp_workspace)
opts_knit$set(root.dir = temp_workspace)
```

## 1. Introduction

Even though R is designed for open and reproducible research, users who want to share their work with others are facing challenges.
Sharing merely the R script or R Markdown document should warrant reproducibility, but many analyses rely on additional resources and specific third party software as well.
An R script may produce unexpected results or errors when executed under a different version of R or another platform. 
Reproducibility is only assured by providing complete setup instructions and resources.
Long-term reproducibility can be achieved by either regular maintenance of the code, i.e. keeping it always working with the latest package versions from CRAN.
It can be supported by packages such as [packrat](https://rstudio.github.io/packrat/) and platforms such as [MRAN](https://mran.microsoft.com/), which provide means to capture a specific combination of R packages.
An alternative to updating or managing packages explicitly is providing the full runtime environment in its original state, using [virtual machines](https://en.wikipedia.org/wiki/Virtual_machine) or  [software containers](https://en.wikipedia.org/wiki/Operating-system-level_virtualization).

The R extension package `containerit` aims to facilitate the latter approach by making reproducible and archivable research with containers easier.
The development is supported by the DFG-funded project Opening Reproducible Research (o2r, [http://o2r.info](https://o2r.info)).
`containerit` relies on [Docker](https://docker.com/) and automatically generates a container manifest, or "recipe", with setup instructions to recreate a runtime environment based on a given R session, R script, R Markdown file or workspace directory.
The resulting [`Dockerfile`](https://docs.docker.com/engine/reference/builder/) can not only be read and understood by humans, but also be interpreted by the Docker engine to create a software container containing all the R packages and their system dependencies.
This way all requirements of an R workflow are packaged in an executable format.

The created Dockerfiles are based on the [Rocker](https://github.com/rocker-org/rocker) project ([Rocker on Docker Hub](https://hub.docker.com/u/rocker/), [introduction](http://dirk.eddelbuettel.com/blog/2014/10/23/#introducing_rocker)).
Using the stack of version-stable Rocker images, it is possible to match the container's R version with the local R installation or any R version the user requires.
`containerit` executes the provided input workspace or file first locally on the host machine in order to detect all dependencies.
For determining external software dependencies of attached packages, `containerit` relies (a) on the [sysreqs database](https://sysreqs.r-hub.io/) and makes use of the corresponding web API and R package, and (b) on internally defined rule sets for challenging configurations.

The Dockerfile created by `containerit` can then be used to build a Docker image. 
Running the image will start an R session that closely resembles the creating systems runtime environment.
The image can be shared and archived and works anywhere with a compatible Docker version. 

To build images and run containers, the package integrates with the [stevedore](https://github.com/richfitz/stevedore) package and adds a few convenience functions for interacting with Docker images and containers.
For concrete details on reading, loading, or installing the _exact_ versions of R packages including their system dependencies/libraries, this project focuses on the geospatial domain.
`containerit` uses the package [`futile.logger`](https://CRAN.R-project.org/package=futile.logger) to provide information to the user at a configurable level of detail, see [futile.logger documentation](https://cran.r-project.org/web/packages/futile.logger/README.html).

The related package [`liftr`](https://liftr.me/) creates Dockerfiles for rendering R Markdown documents, but relies on manually created metadata in the document's YAML header.

In the remainder of this vignette, we first introduce the main usage scenarios for `containerit` and document current challenges as well as directions for future work.

## 2. Creating a Dockerfile

### 2.1 Basics

The easiest way to generate a Dockerfile is to run an analysis in an interactive R session and create a Dockerfile for this session by loading 
`containerit` and calling the `dockerfile()`- method with default parameters.
As shown in the example below, the result can be pretty-printed and written to a file.
If no `file` argument is supplied to `write()`, the Dockerfile is written to the current working directory as `./Dockerfile`, following the typical naming convention of Docker.

When packaging any resources, it is essential that the R working directory is the same as the build context, to which the Dockerfile refers.
All resources must be located below this directory so that they can be referred to by relative paths (e.g. for copy instructions).
This must also be considered when packaging R scripts that use relative paths, e.g. for reading a file or sourcing another R script.

### 2.2 Packaging an interactive session

```{r interactive_session, fig.show='hide'}
library("containerit")

# do stuff, based on demo("krige")
library("gstat")
library("sp")

data(meuse)
coordinates(meuse) = ~x+y
data(meuse.grid)
gridded(meuse.grid) = ~x+y
v <- variogram(log(zinc)~1, meuse)
m <- fit.variogram(v, vgm(1, "Sph", 300, 1))
plot(v, model = m)

# create Dockerfile representation
dockerfile_object <- dockerfile()
```

The representation of a Dockerfile in R is an instance of the S4 class `Dockerfile`.

```{r interactive_session_df}
dockerfile_object
```

The printout below shows the rendered Dockerfile.
Its instructions follow a pre-defined order:

1. define the base image
1. define the maintainer label
1. install system dependencies and external software
1. install the R packages themselves
1. set the working directory
1. copy instructions and metadata labels (see examples in later sections)
1. `CMD` instruction (final line) defines the default command when running the container

Note that the maintainer label as well as the R version of the base image are detected from the runtime environment, if not set to different values manually.

```{r print_df, results='markup', comment=NA}
print(dockerfile_object)
```

Instead of printing out to the console, you can also write to a file:

```{r write_df, eval=FALSE}
containerit::write(dockerfile_object, file = tempfile(fileext = "_dockerfile"))
```

There also is a convenience function to build the `Dockerfile` object with `stevedore`:

```{r build_df, eval=FALSE}
containerit::docker_build(dockerfile_object)
```

### 2.3 Packaging an external session

Packaging an interactive session has the disadvantage that unnecessary dependencies might be added to the Dockerfile and subsequently to the container.
For instance the package `futile.logger` is a dependency of `containerit`, and it _will_ be added to the container because it was loaded into the same session where the analysis was executed.
It cannot be removed by default, because other packages in the session _might_ use it as well (even unintentionally in case of generic methods).
Therefore, it is safer not to tamper with the current session, but to run the analysis in an isolated _vanilla_ session, which does not have `containerit` in it.
The latter will batch-execute the commands in a separate instance of R and retrieves an object of class `sessionInfo`.
The session info is then used as input to `dockerfile()`.
This is also how `dockerfile()` works internally when packaging either expressions, scripts or R markdown files.

The following code creates a Dockerfile for a list of expressions in a vanilla session.

```{r clean_session}
exp <- c(expression(library(sp)),
         expression(data(meuse)), 
         expression(mean(meuse[["zinc"]])))
dockerfile_object <- dockerfile(from = exp)
```

```{r print_clean_df, results='markup', comment=NA}
print(dockerfile_object)
```

### 2.4 Packaging an R script

R scripts are packaged by just supplying the file path or paths to the argument `from` of `dockerfile()`.
They are automatically copied into the container's working directory.
In order to run the R script on start-up, rather than an interactive R session, a CMD instruction can be added by providing the value of the helper function `CMD_Rscript()` as an argument to `cmd`.

```{r script}
# create simple script file
scriptFile <- tempfile(pattern = "containerit_", fileext = ".R")
writeLines(c('library(rgdal)',
             'nc <- rgdal::readOGR(system.file("shapes/", package="maptools"), "sids", verbose = FALSE)',
             'proj4string(nc) <- CRS("+proj=longlat +datum=NAD27")',
             'plot(nc)'), scriptFile)

# use a custom startup command
scriptCmd <- CMD_Rscript(basename(scriptFile))

# create Dockerfile for the script
dockerfile_object <- dockerfile(from = scriptFile, silent = TRUE, cmd = scriptCmd)
```

```{r print_script_df, results='markup', comment=NA}
print(dockerfile_object)
```

### 2.5 Packaging an R Markdown file

Similarly to scripts, R Markdown files can be passed to the `from` argument.
In the following example, a vignette from the Simple Features package `sf` is packaged in a container.
To render the document at startup, the Dockerfile's `CMD` instruction must be changed.
To do this, the `cmd` argument passed to `dockerfile()` is constructed using the function `CMD_Render`.

```{r sf_vignette, results='hide', eval=FALSE}
file.copy(from = system.file("doc/sf6.Rmd", package = "sf"),
          to = file.path(temp_workspace, "sf6.Rmd"),
          overwrite = TRUE)
dockerfile_object <- dockerfile(from = file.path(temp_workspace, "sf6.Rmd"),
                                silent = TRUE, 
                                cmd = CMD_Render("sf6.Rmd"))
```

```{r print_sf_vignette, results='markup', comment=NA}
print(dockerfile_object)
```

### 2.6 Packaging a workspace directory

A typical case expected to be interesting for `containerit` users is packaging a local directory with a collection of data and code files.
If providing a directory path to the `dockerfile()` function, the package searches for the first occurence of an R script, or otherwise the first occurence of an R markdown file.
It then proceeds to package this file along with all other resources in the directory, as shown in the next section.

### 2.7 Packaging from a stored sessionInfo

You can also save your `sessionInfo()` into an `.RData` file and create a `Dockerfile` form it.
The `sessionInfo` object must have the one of the names `sessionInfo`, `sessioninfo`, or `session_info`.
It must be the only object in the `.RData` file.
This is useful when you want to re-create sessions on different machines.

```{r sessionInfo_RData}
suppressPackageStartupMessages(library("sf"))
sessionInfo <- sessionInfo()
save(sessionInfo, file = file.path(temp_workspace, "sessionInfo.RData"))
unloadNamespace("sf")
the_dockerfile <- dockerfile(from = file.path(temp_workspace, "sessionInfo.RData"))
print(the_dockerfile)
```

Two alternative packages provide their own version of a session info, both of which are supported as well.
The following examples first load the packages `fortunes` and `remotes`, store session information objects, and then unload the packages again before creating `Dockerfile`s from the stored objects.

```{r session_info}
suppressPackageStartupMessages(library("devtools"))
suppressPackageStartupMessages(library("sessioninfo"))

suppressPackageStartupMessages(library("fortunes"))
suppressPackageStartupMessages(library("remotes"))

session_info <- devtools::session_info()
save(session_info, file = file.path(temp_workspace, "session_info_devtools.RData"))

sessioninfo = sessioninfo::session_info()
save(sessioninfo, file = file.path(temp_workspace, "session_info.RData"))

unloadNamespace("abc")
unloadNamespace("A3")

df_sessioninfo <- dockerfile(from = file.path(temp_workspace, "session_info.RData"))
df_devtools <- dockerfile(from = file.path(temp_workspace, "session_info_devtools.RData"))
```

**`sessioninfo::session_info()`**

```{r sessioninfo_print, results='markup', comment=NA}
print(df_sessioninfo)
```

**`devtools::session_info()`**

```{r session_info_devtools_print, results='markup', comment=NA}
print(df_devtools)
```

### 2.8 Packaging a `DESCRIPTION` file or object

The [`DESCRIPTION` file format](https://cran.r-project.org/doc/manuals/r-release/R-exts.html#The-DESCRIPTION-file) contains basic information about R extension packages and their dependencies.
`containerit` can also create `Dockerfile`s based on a `DESCRIPTION` file or ob `description` objects from the [`desc`](https://cran.r-project.org/package=desc) package.
Unless specified otherwise, it uses the minimal R version mentioned in the `Depends`-field for the base image.

```{r description_file}
df_description <- dockerfile(from = system.file("DESCRIPTION", package = "dplyr"))
```

```{r description_file_print, results='markup', comment=NA}
print(df_description)
```

Use this to quickly create a container that contains everything needed to run your package, including system dependencies and installing the actual package in a specific version:

```{r description_object}
df_description_sf <- dockerfile(from = desc::desc(package = "sf"),
                                 maintainer = "o2r",
                                 versioned_packages = TRUE)
```

```{r description_object_print, results='markup', comment=NA}
print(df_description_sf)
```

## 3. Including resources

Analyses in R often rely on external files and resources that are located located in the workspace. When scripts or R markdown files are packaged, they are copied by default into the same location relative to the working directory. The argument `copy` influences how `dockerfile()` behaves in this matter. It can either have the values `script` (default behaviour), `script_dir` (copies the complete directory in which the input file is located), or a custom list of files and directories inside the current working directory

```{r resources}
file.copy(from = system.file("simple_test_script_resources", 
                             package = "containerit"),
          to = temp_workspace, recursive = TRUE)
setwd(file.path(temp_workspace, "simple_test_script_resources"))
dockerfile_resources <- dockerfile(from = ".",
              copy = "script_dir",
              cmd = CMD_Rscript("simple_test.R"))
```

```{r print_resources_df, comment=NA}
print(dockerfile_resources)
```
                         
Including R objects works similar to resources, using the argument `save_image`.
The argument can be set to `TRUE` to save _all_ objects of the current workspace to an .RData file, which is then copied to the container's working directory and loaded on startup (based on `save.image()`).

```{r save_image, comment=NA}
the_dockerfile <- dockerfile(save_image = TRUE)
print(the_dockerfile)
```

Alternatively, a object names as well as other arguments can be passed as a list, which then are passed to the `save()` function.

```{r save_image_list}
require("fortunes")
rm(list = ls())
calculation <- 41 + 1
frtn <- fortunes::fortune()
original_sessionInfo <- sessionInfo()

the_dockerfile <- dockerfile(silent = TRUE,
                 save_image = list("original_sessionInfo", "frtn"))
```

```{r print_save_image_list_df, comment=NA}
print(the_dockerfile)
```

## 4. Advanced Dockerfile instructions

### 4.1 Image metadata

Metadata can be added to Docker images using [Label instructions](https://docs.docker.com/engine/reference/builder/#label).
Label instructions are key-value pairs of arbitrary content.
A duplicate key overwrites existing ones.
Although it is up to the user how many labels are created, it is recommended to bundle them into one Label instruction in the Dockerfile.
Each use of the `Label()` function creates a separate instruction in the Dockerfile.

As shown in section 2, the maintainer label is set by default to the top as the dockerfile and contains the username of the current host system.
The maintainer can be changed with the `maintainer` argument of `dockerfile()`:

```{r maintainer}
labeled_dockerfile <- dockerfile(from = clean_session(), maintainer = "Jon_Doe@example.com")
```

Labels can be applied to the existing Dockerfile object using the `addInstructions()` function, which adds any newly created instructions to the end of the Dockerfile but before the CMD statement.
The `Label()` constructor can be used for creating labels of arbitrary content and works similar to creating named lists in R.

```{r labels}
# A simple label that occupies one line:
label1 <- Label(key1 = "this", key2 = "that", otherKey = "content")
addInstruction(labeled_dockerfile) <- label1

#label with fixed namespace for all keys
label2 <- Label("name"="A name", "description" = "A description", label_ns = "my.label.ns.")

# A multiline label with one key/value pair per line
label3 <- Label("info.o2r.name" = "myProject_ImageName", "org.label-schema.name"="ImageName", 
                "yet.another_labelname"="true", multi_line = TRUE)
addInstruction(labeled_dockerfile) <- list(label2, label3)
```

Metadata according to the [Label Schema](http://label-schema.org/rc1/) conventions can be created with a function constructed by the helper factory `LabelSchemaFactory()`. 

```{r label_schema}
Label_LabelSchema <- LabelSchemaFactory()
label <- Label_LabelSchema(name = "ImageName", description = "Description of the image", build_date = Sys.time())
addInstruction(labeled_dockerfile) <- label
```

You can also put session information, using either base R or `devtools`, into a label as plain text or as json:

```{r label_json}
addInstruction(labeled_dockerfile) <- Label_SessionInfo(session = clean_session())
addInstruction(labeled_dockerfile) <- Label_SessionInfo(session = devtools::session_info(), as_json = TRUE)
```

The resulting Dockerfile with all the labels:

```{r print_label_df, comment=NA}
print(labeled_dockerfile)
```

### 4.2 Exposing ports

If a Dockerfile contains a web service, specific ports of a running container might need to be exposed.

> The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. [..] It functions as a type of documentation between the person who builds the image and the person who runs the container, [..] To actually publish the port when running the container, use the -p flag on docker run to publish and map one or more ports [..] ([Docker docs](https://docs.docker.com/engine/reference/builder/#expose))

```{r expose}
dockerfile_with_expose <- dockerfile()
my_port_1 <- Expose(8000)
my_port_2 <- Expose(port = "80/tcp", host = 8080)
addInstruction(dockerfile_with_expose) <- list(my_port_1, my_port_2)
print(dockerfile_with_expose)
```

### 4.3 Entrypoint and command

The `ENTRYPOINT` instruction allows using a container like a binary executable.
The actually executed command is constructed from both `ENTRYPOINT` and `CMD` instructions - please check the [Docker docs](https://docs.docker.com/engine/reference/builder/#entrypoint) for details.
The `entrypoint` parameter of the `dockerfile()` function supports both "forms" as shown below.

```{r entrypoint_exec}
ep_exec <- Entrypoint(program = "R",
                 params = list(
                   "-e",
                   "pr <- plumber::plumb(commandArgs()[4]); pr$run(host='0.0.0.0', port=8080)"),
                 form = "exec")
dockerfile_with_entrypoint <- dockerfile(from = NULL,
                                         image = "trestletech/plumber",
                                         entrypoint = ep_exec,
                                         cmd = Cmd("schedule.R"))
print(dockerfile_with_entrypoint)
```

```{r entrypoint_cmd_shell}
ep_shell <- Entrypoint(program = "R",
                 params = list(
                   "-e",
                   "pr <- plumber::plumb(commandArgs()[4]); pr$run(host='0.0.0.0', port=8080)"),
                 form = "shell")
dockerfile_with_entrypoint <- dockerfile(from = NULL,
                                         image = "trestletech/plumber",
                                         entrypoint = ep_shell,
                                         cmd = Cmd("schedule.R", form = "shell"))
print(dockerfile_with_entrypoint)
```

### 4.4 Comments

Comments can be added to a Dockerfile.
By default they are the last instruction.

```{r comment, comment=NA}
df_comment <- dockerfile(from = NULL, image = getImageForVersion("3.3.3"), silent = TRUE)
addInstruction(df_comment) <- Comment(text = "Don't forget about this!")
print(df_comment)
```

## 5. Custom base image

The `dockerfile()` function allows further customization regarding the R version or the used base image (cf. Rocker stack).
Note that while choosing an R version for the Dockerfile explicitly is possible, the session to generate the required information (i.e. which packages are attached etc.) is still running the R version of the generating machine.

The following examples show usage of these options and the respective `FROM` statements in the Dockerfile.

```{r base_image, comment=NA}
df_custom <- dockerfile(from = NULL, image = getImageForVersion("3.3.3"), silent = TRUE)
print(df_custom@image)
df_custom <- dockerfile(from = NULL, image = "rocker/geospatial", silent = TRUE)
print(df_custom@image)
df_custom <- dockerfile(from = NULL, image = "rocker/verse:3.0.0", silent = TRUE)@image
print(df_custom@image)
```

When you change the base image, you might not want to re-install packages that are already in the base image.
The helper function `get_installed_packages()` is used internally to filter the installed packages and can be activated with `filter_baseimage_pkgs`.

```{r base_image_filter, comment=NA}
expressions <- c(expression(library("sp")))
session <- clean_session(expressions, echo = TRUE)
df_filtered <- dockerfile(from = session, maintainer = "o2r",
                   image = "rocker/geospatial:3.4.4",
                   filter_baseimage_pkgs = TRUE)
```

```{r base_image_filter_dockerfile}
print(df_filtered)
```

## 6. CLI

A command line interface to the package functions is also available for Linux based on [docopt.R](https://github.com/docopt/docopt.R).
This allows integration into workflows and tools written in other programming languages than R.

You can make the command `containerit` available on your machine by linking the R script file delivered with the package as follows:

`ln -s $(Rscript -e "cat(system.file(\"cli/container_it.R\", package=\"containerit\"))") /usr/local/bin/containerit`

CLI Examples:

```{bash cli, eval=FALSE}
  containerit --help
  
  # runs the first R markdown or R script file locally 
  # prints Dockerfile without writing a file
  containerit dir -p --no-write  
  
  # Packages R-script 
  # saves a workspace image (-i parameter)
  # Writes Dockerfile (overwrite with -f)
  # execute the script on start-up
  containerit file -ifp --cmd-R-file path/example.R

  # Creates an empty R session with the given R commands
  # Set R version of the container to 3.3.0
  containerit session -p -e "library(sp)" -e "demo(meuse, ask=FALSE)"
```

## 7. Challenges

We encountered several challenges during `containerit`'s development.
First and foremost, a well known limitation is that R packages don't define system dependencies and do not provide explicit versions for R package dependencies.
The `sysreqs` package is a promising approach towards handling system requirements, but so far lists package names but does not provide version information.
The [shinyapps-package-dependencies](https://github.com/rstudio/shinyapps-package-dependencies) demonstrate a (currently system dependent) alternative.
The high value of R might well lie in the fact that "packages currently on CRAN" should work well with each other.

An unmet challenge so far is the installation of specific versions of external libraries (see [issue](https://github.com/o2r-project/containerit/issues/46)).
A package like `sf` relies on well-tested and powerful system libraries, see `sf::sf_extSoftVersion()`, which ideally should be matched in the created container.

And of course users may do things that `containerit` cannot capture from the session state "after the analysis is completed", such as detaching packages or removing relevant files, and unknown side-effects might occur.

All software is presumed to be installed and run on the host system.
Although it is possible to use deviating versions of R or even create Dockerfiles using sessionInfo-objects created on a different host, this may lead to unexpected errors because the setup cannot be tested locally.

## 8. Conclusions and future work

`containerit` allows to create and customize Dockerfiles with minimal effort, which are suitable for packaging R analyses in the persistent runtime environment of a software container.
So far, we were able to reproduce complete R sessions regarding attached _but not "loaded only" packages and mitigate some challenges towards reproducible computational research.

Although we are able to package different versions of R, we still do not fully support the installation of specific versions of R packages and external software libraries, which R itself does not support.
This should be tested in the future by evaluating version-stable package repositories like MRAN and GRAN or utility packages such as packrat -- see the [GitHub issues](https://github.com/o2r-project/containerit/issues/new) for the status of these plans or provide your own ideas there.

Related to installing specific versions is support for other package repositories, such as Bioconductor, git, BitBucket, or even local files.
For now, it is recommended that users have all software up-to-date when building a software container, as the latest version is installed from CRAN during the image build, to have matching package versions between the creation runtime environment and the container.
All Dockerfiles and instructions are adjusted to the Rocker image stack and assume a Debian/Linux operating system.
As we are not yet supporting the build of Docker images from scratch, we are restricted to this setup.

The package is a first prototype available via GitHub.
While a publication on CRAN is a goal, it should be preceded by feedback from the user community and ideally be accompanied by related packages, such as `stevedore`, being available on CRAN, too.
The prototype of `containerit` was developed and tested only on Ubuntu/Linux, which should be extended before releasing a stable version on CRAN.

As part of the o2r project, it is planned to integrate `containerit` in a [web service](http://o2r.info/architecture) for creating archivable research in form of [Executable Research Compendia (ERC)](https://doi.org/10.1045/january2017-nuest).
Making `containerit` itself easier to use for end-users is a secondary but worthwhile goal, for example by building a graphical user interface for metadata creation.
Country locales are also not supported yet.
We may want to support other container OS (e.g. windows container or other Linux distributions) or even containerization solutions such as [Singularity](http://singularity.lbl.gov/) or the [Open Container Initiative](https://www.opencontainers.org/)'s (OCI) [Image Format](https://github.com/opencontainers/image-spec).

Feedback and contributions are highly welcome [on GitHub](https://github.com/o2r-project/containerit/issues) or [o2r_project](https://twitter.com/o2r_project) on Twitter.

## Metadata

```{r sessionInfo}
sessionInfo()
```