Improve monolithic build

[romainx]

Hello,

As discussed in two recents PR (#1196 and #1202) the current build seems to be too monolithic.

If we add additional stacks, the build will grow and will become error prone and it will become a pain to publish all the stacks. The build takes between 40 min and 1 hour and if a single thing fails (a test, a package installation) everything fails.

We should benefit from GitHub actions that have been introduced by @parente in PR #1132 and / or by proposal made by @bollwyvl in his comment. My first idea is to split the treatment of each image in different jobs that can be run in parallel (at least for some of them) with dependencies between them to achieve more flexibility. We could benefit from advanced features like artifact sharing to achieve this.

A job could be responsible of the whole stuff for an image:

• Lint
• Build
• Test
• Hook / Manifest -> see also another improvement on this point Refactor duplicative, opaque run_hook scripts #1140
• Tag -> support more flexible tagging mechanism like tagging images with main components version (Downgrade Python from 3.7 to 3.6 or tag 3.6 version #976)
• Push

All these cool new stuff with the ability to build things locally.
So this is not so simple, there is a lot of things to solve / decide but there is room for improvement 😄.

Best.

[trallard]

I recently offered to help with some tasks for #1407, after reading @mathbunnyru proposal and additional comments it is clear that this issue - i.e. improving the monolithic build should be tackled first to ensure we have a solid foundation.

@manics I would be happy to explore and work on a POC leveraging GH actions + doit (for x86):

1. I can start with a smaller scoped POC to gauge how doit + GH actions would work for our scenario
2. If this is indeed the best path moving forward, this can be then further expanded to allow for images to be built/tested/tagged/etc. in parallel

Some advantages I see from this approach are:

1. We can leverage the existing GH actions work introduced by @parente
2. Exploring the use of doit vs Make - this should help with multiple automation tasks, also might make it easier for other folks to contribute to since doit is Python-based, and in general, help make the workflows a bit more flexible
3. Having this sorted out should also allow us to identify the best path moving forward to leverage cache or the use of artefacts to start optimising the builds in subsequent runs

thoughts?

[romainx]

@trallard In a recent PR (#1604) I've explored a way to standardise things by using docker buildx bake.

Buildx also aims to provide support for high-level build concepts that go beyond invoking a single build command. [...]
BuildKit efficiently handles multiple concurrent build requests and de-duplicating work. source

I'm convinced that it could be a an option since it support multi-platform and can scale by using several build nodes through various drivers. It is the solution used by others with similar issues to solve like Official Jenkins Docker image -> jenkinsci/docker#1137. It's a way of standardising and improving the build of several images that are closely related like we do here. Out of the box it will permit to build in // images that depends on the same upstream images (minimal-notebook and scipy-notebook have several downstream images). Then we can decide to build a group of images on a node and an other elsewhere. The concept of groups of targets is here for this purpose.

However other maintainers were not convinced, so if you have time to have a look 👀 and come with a confirmation that it's not the solution or with new arguments in favor of this solution it could be interesting.
Best.

[bollwyvl]

As Docker Desktop has recently... altered its terms (pray they do not alter them further), I've been looking into other cross-platform solutions as well for this kind of thing. minikube seems to be the most cross-platform approach at the bottom of the stack, as well as the most stable and well-documented.

At the build level, skaffold keeps coming up, and might be worth a look for build, test, and push. It can use a "real" docker dameon, or any number of others, including minikube, and supports parallel builds.

skaffold doesn't have built-in lint or vulnerability scanning, but hadolint and grype are also cross-platform. So a one could imagine a workflow with an outer dodo.py which handled mundane tasks (like filling up caches) in addition to the overall orchestration of when to skaffold build vs skaffold test). I've been meaning to get skaffold, etc. up onto conda-forge so that the whole toolchain can be specified in a portable way...