Restoring the cache takes longer than running docker pull

[pjonsson]

Edit: I know that the additional images were saved in the post action in the job that populated the cache, I do not know how they ended up in docker image list in that job.

The log from a cache hit on contiki-ng/contiki-ng#1946 below.

Cache Size: is printed after 25 seconds, tar takes 59 seconds, and docker load takes 2 min and 14 seconds. The typical docker pull command without any cache takes less than 1 minute and 30 seconds.

The CI has an explicit step to pull the right version of contiker/contiki-ng, and it's possible that brings along the Ubuntu 18.04 image which it is based on. I am not sure where the other images come from.

Run ScribeMD/docker-cache@0.1.4
Received 192937984 of 3148759409 (6.1%), 183.8 MBs/sec
Received 423624704 of 3148759409 (13.5%), 201.6 MBs/sec
Received 650117120 of 3148759409 (20.6%), 206.4 MBs/sec
Received 876609536 of 3148759409 (27.8%), 208.7 MBs/sec
Received 1111490560 of 3148759409 (35.3%), 211.7 MBs/sec
Received 1354760192 of 3148759409 (43.0%), 214.9 MBs/sec
Received 1581252608 of 3148759409 (50.2%), 215.0 MBs/sec
Received 1820327936 of 3148759409 (57.8%), 216.4 MBs/sec
Received 2071986176 of 3148759409 (65.8%), 219.0 MBs/sec
Received 2147483647 of 3148759409 (68.2%), 115.8 MBs/sec
Received 2306867199 of 3148759409 (73.3%), 117.8 MBs/sec
Received 2462056447 of 3148759409 (78.2%), 119.3 MBs/sec
Received 2634022911 of 3148759409 (83.7%), 121.4 MBs/sec
Received 2822766591 of 3148759409 (89.6%), 124.1 MBs/sec
Received 3003121663 of 3148759409 (95.4%), 126.2 MBs/sec
Received 3148759409 of 3148759409 (100.0%), 116.5 MBs/sec
Cache Size: ~3003 MB (3148759409 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/aa3ae13e-32e5-4393-878e-987f4e070a15/cache.tzst -P -C /home/runner/work/contiki-ng/contiki-ng
Cache restored successfully

docker load --input ~/.docker-images.tar
Loaded image: node:14
Loaded image: node:16
Loaded image: buildpack-deps:buster
Loaded image: node:14-alpine
Loaded image: alpine:3.13
Loaded image: debian:9
Loaded image: moby/buildkit:latest
Loaded image: ubuntu:20.04
Loaded image: ubuntu:18.04
Loaded image: alpine:3.12
Loaded image: contiker/contiki-ng:63f7e91e3
Loaded image: buildpack-deps:stretch
Loaded image: node:12
Loaded image: alpine:3.14
Loaded image: ubuntu:16.04
Loaded image: buildpack-deps:bullseye
Loaded image: debian:10
Loaded image: debian:11
Loaded image: node:16-alpine
Loaded image: node:12-alpine

[Kurt-von-Laven]

I'm glad you are measuring the effect of the action in your circumstance. I don't recommend caching 3 GB worth of images since even if it were faster in an individual instance, it would likely result in a great deal of cache eviction given that GitHub Actions has a cache limit of 10 GB. If you are able to, I would pull from the GitHub Container Registry instead of using this action as pulls from there are very fast (much the same as the raw download time from the cache without the subsequent docker load step). Since you are building a Docker image, I expect you will also achieve better results by using the official Docker build push action instead if you aren't already.

My guess would be that there is a lot of overlap between those images, so Docker layer caching allows docker pull to pull way less than 3 GB worth of data. On the other hand, saving each of those images separately, even with compression, may not result in the same level of gains (relative to an uncompressed tarball).

[Kurt-von-Laven]

Closing this as a duplicate of #44 since the discussion there will likely interest anyone who reads this thread.

[Kurt-von-Laven]

Thanks to the hard work of @mwarres, version 0.2.3 filters out Docker images that are present when it is run, only caching those that are added after that step. Most significantly, this means that pre-cached images will no longer be cached. Pre-cached images are only accessible to the rootful Docker daemon, so rootless-docker users were never caching pre-cached images in the first place. The documentation will be updated accordingly, but generally users will simply experience a dramatic performance improvement both in terms of increased speed and reduced cache size upon upgrading.

[pjonsson]

I just tried this and the numbers are not directly comparable with the initial numbers since the image has increased a few hundred megabytes in size, but the saved size is now ~1250 MB instead of ~3003 MB.

The reduction in size has drastic effects on performance, Cache Size: is printed after 10 seconds instead of 25, tar takes 19 seconds instead of 59, and docker load takes 59 seconds instead of 2 min and 14 seconds.

[Kurt-von-Laven]

@mwarres updated the documentation. Please let us know if you have any feedback.

[Benjamin-Dobell]

Unfortunately, even with the latest release I'm seeing cache restoration be about 25-30% slower than just downloading the 2GB worth of Docker images (10 images - Supabase's local environment). No issues with extra images being cached, it's just the 10 expected; the downloading + decompression is just significantly slower than just downloading the images.

[Kurt-von-Laven]

I actually view this as a good thing in the sense that it means you are achieving excellent pull performance, and the main remaining ways to improve on it would be to optimize the image sizes themselves, reduce dependencies, or self-host. Where are you pulling the images from?

[Benjamin-Dobell]

ECR, although I'm not specifically pulling them myself. I'm using the Supabase CLI and grabs the images from there automatically. I think you're right though, if I want to improve performance given these constraints, then I think I'll have have to self-host my runners (with the docker images pre-installed on the runner) 😞

[Kurt-von-Laven]

Yeah, sorry docker-cache couldn't help you here. It sounds like Amazon Elastic Container Registry (ECR) may be an even faster option than GitHub Container Registry (GHCR) then. That's a good thing to note, particularly for those who control where their Docker images are hosted. I know some projects run up against Docker Hub's rate limits, so that is another use case for docker-cache in case that helps future readers of this thread.