WIP: Attempt to speed up CI by caching cargo dependencies

[justinmoon]

I'm attempting to adapt an "official example". This same technique worked for my project which uses Druid (7 minute build -> 2 minute build), but it doesn't speed things up here.

I suspect the problem might be that Cargo.lock isn't checked in -- as discussed in this thread.

Perhaps druid being a "workspace" also interferes somehow?

Note about CI in cargo book.

[cmyr]

cool, work on this stuff is definitely appreciated!

[Blisto91]

This Pull request at the "official example" repository might have some additional info and help.

[Blisto91]

I am pretty new to rust and github. So i hope my writing makes sense and that i'm doing it in the correct format. :)

I looked into the examples of the pull request linked above and read a bit more into how it worked.
The Testing a library example seems very useful for Druids use case.

We can lower the size of the cache by only caching ~/.cargo/registry/index and ~/.cargo/registry/cache instead of the whole ~/.cargo/registry directory.
The dependency crates are stored twice in the folder as unpacked files in /registry/cache and raw source code in /registry/src.
We can omit /registry/src to avoid redundant data.

[cmyr]

@justinmoon where did this work end up?

[Blisto91]

If you are having trouble getting it to work then i could try and take a swing at it in a separate PR. If you wouldn't mind :).

Been doing a lot of reading and some testing with the CI and i think i have something that should work.

[justinmoon]

@cmyr: I haven't had any time to play with it, and probably won't :(. I mostly wanted share what little progress I made -- seems like low-hanging fruit.

@Blisto91: Go for it

[Blisto91]

So i pulled down my own PR again that only handled the downloaded dependency crates and crates.io index, because over time i wasn't too happy with the modest improvement. Sometimes it actually took a bit longer because restoring a Github cache seems to fluctuate between a few seconds to several minutes, even for the same small cache.

Tho i do think there is proper gains to be had from caching the build target.
The biggest problem here is we have to satisfy the 5GB Github cache storage limit. Especially the Ubuntu build gets very large just on it's own (around 5GB on my own linux laptop) and most of the size is in the examples folder.
With the impressive compression the cache does this is actually reduced to a bit below 2GB when packed and stored.
The MacOS and Windows files are in the 300 - 500MB range when compressed.
There probably have to be separate cache for stable and nightly and maybe also the wasm files, so we blow the budget quite quickly.

My first attempt was to keep incremental compilation off and then clean only the druid artifacts with Cargo clean -p and or Powershell.
But the CI kept failing on macOS after restoring the files saying it couldn't find the rental-impl dependency for fluent-bundle, even tho it was present. I tried to investigate the problem and found a Rust github issue that linked to this nightly flag and was able to get it working on nightly only with it.
It also seemed work by reversing the order the druid crates are tested in. But i think it seemed like a bad solution.

The next attempt was to enable incremental and then just optimize the build target for size.
These build settings got the size down substantially on my linux laptop.
opt-level = 'z'
lto = 'fat' or 'thin'
codegen-units = 1
But i again ran into build trouble on macOS where it on nightly failed with lto enabled. But lto stood for most of the size savings on it's own.
Seems there are issues with lto on atleast nightly currently and people are still complaining on a closed github issue that highlighted the problem. But can't seem to find it again currently.

Sorry for the long ramble! Just wanted to share some findings.

Edit:
Some good reads relating to the size/caching/dependency problems i was having and some possible future solutions in Rust.
Cache usage meta tracking issue.
Stabilize Cargo's new feature resolver.
Unify build directories. This one is a bit stale but is still interesting.

[Blisto91]

Just read through this pinned issue over on the GitHub cache repository.

I think we should maybe put this on hold a bit until at least the cache restore times are more consistent.

[CleanCut]

@Blisto91 that pinned issue has been updated - v2.1.0 of the cache action has been released, which should resolve the inconsistent cache restore times. (v2 has already been updated to point to v2.1.0)

[Blisto91]

@CleanCut ye I saw and I've seen much more consistant restore times in my tests since then.
I did have one new cache where it took 15-20 minutes to save it. I think the new improvement might only apply to restore times?

Now I just want windows to use Zstd and the new rust feature resolver to get stabilized and then I'm a satisfied man 😁
A Github cache storage limit increase to 10 or 15GB wouldn't be unwelcome either hehe.

Edit: Saw this on the new github roadmap.
With a proper increased cache size limit we could get away with caching the whole build target without cleaning or needing external cloud storage. That would probably show as a nice speed boost and avoid dependency failures with the current feature resolver.

[CleanCut]

Edit: Saw this on the new github roadmap.
With a proper increased cache size limit we could get away with caching the whole build target without cleaning or needing external cloud storage. That would probably show as a nice speed boost and avoid dependency failures with the current feature resolver.

That would be really cool. 👍

[cmyr]

@justinmoon What's the status here? Do you think it's worth merging?

[Blisto91]

@cmyr I think there is a good chance the CI will fail on subsequent runs after the cache have been saved.
Especially had problems on mac during my testing.
I think saving the whole target folder is risky until the new rust feature resolver.

I have toyed a bit with sccache and i think that might work fine.
If @justinmoon doesn't mind then i could try to take another stab at it with a new PR.

[cmyr]

@Blisto91 be my guest, it would certainly be nice to get this sped up. Another option of course is to just break up the work into more separate jobs, that ultimately leads to more maintenance though so it's a tricky tradeoff 😒