Manual image caching

[Taepper]

faster builds by copying @corneliusroemer image caching for our dependency images, which rarely change

[Taepper]

an example of the working cache:
https://github.com/GenSpectrum/LAPIS-SILO/actions/runs/8463811779

[corneliusroemer]

Nice! Would be great to quantify the speed ups :D

How long did it take before and now when there are cache hits for everything?

I didn't see the arm part here, do you use separate docker files for that?

[corneliusroemer]

How come the linter takes so long? Is there so much work done?

Also, I noticed you often compile with 4 parallel threads. GitHub actions runners only have 2. It might improve performance to reduce to 2 in CI

[JonasKellerer]

LGTM

The ci did not run the unit test image. (from the link you posted) Was that aborted by you?

[corneliusroemer]

The ci did not run the unit test image. (from the link you posted) Was that aborted by you?

@JonasKellerer it did, or is that something else?

https://github.com/GenSpectrum/LAPIS-SILO/actions/runs/8463828054/job/23187878018

the unit test image is called builder and used in the test run AFAICT:

LAPIS-SILO/.github/workflows/ci.yml

Lines 201 to 217 in 7867bc7

	- name: Build unit test image
	uses: docker/build-push-action@v5
	with:
	context: .
	target: builder
	tags: builder
	load: true
	cache-from: type=gha,ref=${{ github.ref_name }}-image-cache
	cache-to: type=gha,mode=min,ref=${{ github.ref_name }}-image-cache
	build-args: |
	DEPENDENCY_IMAGE=ghcr.io/genspectrum/lapis-silo-dependencies:${{ env.DIR_HASH }}
	- name: Run tests
	uses: addnab/docker-run-action@v3
	with:
	image: builder
	run: ./silo_test

[JonasKellerer]

from Alex comment. There the step of the unit image did not run. But maybe thats an older run? But if it did run all is fine :)

[Taepper]

I cancelled it after I saw that it hit the cache in the necessary job. I then just squash and rebased the commit for the PR. Then it rebuilt both dependency images as expected, as duckdb->0.10.1 dependency change was already on the changed main

[Taepper]

@corneliusroemer the quantification is not as straight forward, as the cache was hit also before but only sometimes

Mostly the first dependency jobs, which take 20 mins each (although in parallel) should now be saved, if no dependency were changed. In the attached screenshot the upper pipeline would be shortened by 21m2s, and the lower one hit the cache by itself by chance