Create a larger build image and pipeline with extra packages

[myoung34]

It would make sense to have a third image type such:

Base Image
├── Base Image With Extras
│   └── Runner Final (Large)
└── Runner Final (Small)

Where Both Finals are identical but have FROM Base:tag or From Base:tag-large

This would let us have:

Base Image
├── Base Image With Extras
│   ├── Language specific packages (python, node, ruby, etc)
│   ├── Browser testing suites, etc
│   └── Compilation specific packages (gcc, make, etc)
└── Runner Final (Small)
    └── No change from current

Would resolve:

• Browsers in Runner Image #237
• Add clang and libclang1 to base image #256

References:

• https://github.com/nektos/act#runners
• https://github.com/actions/runner-images

[nmalaguti]

Objective

I investigated this. It would not be hard to do this, the question is what should be included in each image.

User stories:

1. I'm a user who just wants to run a container for self-hosted runners. I want it to behave as much like github provided runners as possible
2. I'm a user who just wants to run a container for self-hosted runners. I don't use anything too exotic and want the expected utilities and frameworks to be available. I plan on using other actions to install most complex dependencies.
3. I'm a user who wants the bare minimum image for self-hosted runners. I will use it as a base in my own image that will include exactly the additional dependencies I need.

Current State

Today the project is somewhere between 2 and 3.

Diff

docker-github-actions-runner/Dockerfile.base

Lines 20 to 49 in ac2e2b9

	&& apt-get install -y --no-install-recommends \
	gnupg \
	lsb-release \
	curl \
	tar \
	unzip \
	zip \
	apt-transport-https \
	ca-certificates \
	sudo \
	gpg-agent \
	software-properties-common \
	build-essential \
	zlib1g-dev \
	zstd \
	gettext \
	libcurl4-openssl-dev \
	inetutils-ping \
	jq \
	wget \
	dirmngr \
	openssh-client \
	locales \
	python3-pip \
	python3-setuptools \
	python3 \
	dumb-init \
	nodejs \
	rsync \
	gosu \

and https://github.com/actions/runner-images/blob/c92563577c066cae33ea5b41eb882f9bf70291e6/images/linux/toolsets/toolset-2004.json#L135-L222

(red is from this repo, green is from the official runner image)

-apt-transport-https
+acl
+aria2
+autoconf
+automake
+binutils
+bison
+brotli
 build-essential
-ca-certificates
+bzip2
+coreutils
 curl
-dirmngr
-dumb-init
-gettext
-gnupg
-gosu
-gpg-agent
-inetutils-ping
+dbus
+dnsutils
+dpkg
+fakeroot
+file
+flex
+fonts-noto-color-emoji
+ftp
+gnupg2
+haveged
+imagemagick
+iproute2
+iputils-ping
 jq
-libcurl4-openssl-dev
+lib32z1
+libc++-dev
+libc++abi-dev
+libcurl4
+libgbm-dev
+libgconf-2-4
+libgsl-dev
+libgtk-3-0
+libmagic-dev
+libmagickcore-dev
+libmagickwand-dev
+libsecret-1-dev
+libsqlite3-dev
+libtool
+libunwind8
+libxkbfile-dev
+libxss1
+libyaml-dev
 locales
-lsb-release
-nodejs
+m4
+mediainfo
+mercurial
+net-tools
+netcat
 openssh-client
-python3
-python3-pip
-python3-setuptools
+p7zip-full
+p7zip-rar
+parallel
+pass
+patchelf
+pkg-config
+pollinate
+python-is-python3
+rpm
 rsync
-software-properties-common
+shellcheck
+sphinxsearch
+sqlite3
+ssh
+sshpass
+subversion
 sudo
+swig
 tar
+telnet
+texinfo
+time
+tk
+tzdata
 unzip
+upx
 wget
+xorriso
+xvfb
+xz-utils
 zip
-zlib1g-dev
-zstd
+zsync

This repo also includes:

• liblttng-ust0 or liblttng-ust1
• awscli
• git (from git-core)
• git-lfs
• docker
• docker-compose v1
• podman
• buildah
• skopeo
• gh (github-cli)

All of which are also included in the official github runner image. There aren't many extraneous packages that aren't also needed for us setting up and running the actions runner (e.g. gosu).

Prior Art

Where should we draw the lines between base, default, and extra?

Some prior art to look at are the images suggested by https://github.com/nektos/act/blob/master/IMAGES.md which recommend catthehacker/ubuntu:act-20.04 and catthehacker/ubuntu:full-20.04 as the medium and large images respectively.

catthehacker/ubuntu:full-20.04

According to https://github.com/catthehacker/docker_images#images-available the catthehacker/ubuntu:full-20.04 is a 20GB compressed image that extracts to ~60GB. It is also out of date with the upstream github runner images and doesn't seem to be worth updating automatically due to the amount of upstream changes. I'm really not sure what the health is of the full image.

If we really wanted to provide an image to satisfy user story (1) above, building on top of that image is probably the best idea and comes with the smallest amount of maintenance for this project, but also is likely to be out of date and generate "I tried this and it didn't work" or "this image is too big" issues.

catthehacker/ubuntu:act-20.04

The setup script for this image is https://github.com/catthehacker/docker_images/blob/master/linux/ubuntu/scripts/act.sh. It is very similar to the image in this repo, except that it includes all of the apt packages in the official github runners and does not include:

• liblttng-ust0 or liblttng-ust1
• awscli
• podman
• buildah
• skopeo
• gh (github-cli)

This might make the most sense as the base image for our default image. However, they also have requests like catthehacker/docker_images#61 to add things like python3-pip to the act image. Clearly someone still needs to make some editorial decisions about what a reasonable base image includes.

Recommendation

Ultimately I'm not convinced that user story (3) will actually provide much value compared to (2). Supporting (1) is likely to come with a high maintenance or support burden and is likely to be overkill for most users.

For my use cases, the image today works great. I have my own Dockerfile that I use to add some additional python setup based on github's own setup script.

FROM myoung34/github-runner:ubuntu-focal

ENV AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache

RUN apt-get update && \
    cd /tmp && \
    git clone https://github.com/actions/virtual-environments.git && \
    echo '#!/usr/bin/env bash' > invoke_tests && \
    chmod +x invoke_tests && \
    cd virtual-environments/images/linux/scripts && \
    PATH="/tmp:$PATH" HELPER_SCRIPTS=$PWD/helpers bash installers/python.sh && \
    cd / && \
    rm -rf /tmp/* && \
    rm -rf /var/lib/apt/lists/* \
    ;

ENV PIPX_BIN_DIR=/opt/pipx_bin
ENV PIPX_HOME=/opt/pipx

RUN chown -R runner $PIPX_BIN_DIR $PIPX_HOME

ENV PATH="$PIPX_BIN_DIR:$PATH"

The biggest issue I encounter today is the additional apt repos sometimes breaking and causing issues in my CI runs. My preference would be to keep the number of 3rd party apt repos to a minimum.

The best path forward is probably to either build on top of catthehacker/ubuntu:act-20.04 or adopt their method of installing all of the suggested apt packages. This will make our image larger, but will also make it easier for users to follow my example above and install toolchains using https://github.com/actions/virtual-environments. We can then install a curated set of packages like gh, awscli, etc.

Our images would then be:

• base built from Dockerfile.base that includes:
  • the apt packages needed for running our setup scripts
  • any minimal editorializing of packages (gh, awscli, etc.)
• main image that includes the actions runner and setup scripts
• extra image based on main that includes:
  • pwsh (based on https://github.com/catthehacker/docker_images/blob/master/linux/ubuntu/scripts/pwsh.sh) which is needed for some of the official tooling
  • the setup needed to easily run additional toolchains and packages based on https://github.com/actions/runner-images/tree/main/images/linux/scripts/installers

Users would be expected to use the extra image as their base and then make their own image (this is unlikely to work in all cases for arm64).

FROM myoung34/github-runner:ubuntu-focal-extra

RUN /bin/bash /imagegeneration/installers/basic.sh \
  && /bin/bash /imagegeneration/installers/python.sh \
  && /bin/bash /imagegeneration/installers/firefox.sh \
  && /bin/bash /imagegeneration/installers/selenium.sh \
  ;

RUN . /etc/environment \
  && chown -R runner $PIPX_BIN_DIR $PIPX_HOME \
  ;

I think the only modification to the existing image would be to source /etc/environment and re-export all of the environment variables in entrypoint.sh.

[Austin-s-h]

@myoung34 I think I've developed something similar to this, please check it out!
https://github.com/sansterbioanalytics/unified-actions-runner

[myoung34]

Hey all, after thinking on it a bunch: I dont think this is something I personally want to maintain

That said, I think I'm happy to provide guidance etc, but I think the best option is for someone to automate a downstream "purpose built" image or images

I'm happy to help give guidance on automation, but ideally:

• A new repo(s) that triggers when I release a tag (which is triggered when actions/runner releases a tag)
• Does a FROM: myoung34/github-runner
• builds all the purpose built stuff

I say this because this repo is already a bit tricky as-is in someways but also I personally have 0 use case for this, so the idea of maintaining purpose built images for things I don't use or wish to troubleshoot is not appealing

I'm going to close this as "not possible" because the aim of this repo is to be inline with the actions/runner upstream capabilities for self-hosting or a platform for others to use as suggested here