CI: Cache libimagequant on Linux builds #7741
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On the CI, the step to "Install Linux dependencies" takes between 6m10s and 6m30s.
The step runs
.ci/install.sh
and its slowest commands are:They take ~25s, ~4m30s and ~40s respectively.
Given the Linux jobs mostly take around 8m30s, caching these steps can save up to 5m30s, and would greatly improve the CI time.
Let's start with the slowest: libimagequant.
First, add to the workflow a cache step:
If there's a matching cache file, it unpacks the cache to
~/cache-libimagequant
and we have a cache hit.Then we pass in the cache hit status as an environment variable (
GHA_LIBIMAGEQUANT_CACHE_HIT
) toinstall.sh
.install.sh
callsdepends/install_imagequant.sh
, which checks the env var.If there's a cache hit, it sudo copies the cached files from
~/cache-libimagequant
to/usr/...
.If there's a cache miss, we need to build from source, as before. Except we also copy the built files into
~/cache-libimagequant
so they can be cached for next time. (Delete and recreate the dir, just to be sure no old cached files are retained.)This reduces the "Install Linux dependencies" step from 6.5-8 minutes to to 1.5-2 minutes, and saves about 45 minutes of CPU time in total.
Total time for some jobs:
After this, we can do a similar thing for
install_raqm.sh
andinstall_webp.sh
to shave off another minute or so per job; I left them out because I wasn't sure what the filenames are that need caching; what are they?Also:
Include
pyproject.toml
along with.ci/*.sh
in the pip cache key, because we define test/docs dependencies in there tooDisable the wget progress bar but not all output, to omit hundreds of lines in the logs like: