Add cache for bazel to prove why bazel rocks

[bogdandrutu]

Some initial numbers.

Before

After:

Signed-off-by: Bogdan Drutu bogdandrutu@gmail.com

[bogdandrutu]

Please do not press update branch because I want to do that as the second round to prove the benefit :)

[codecov]

Codecov Report

Merging #505 (af52891) into master (e8ef7ea) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #505      +/-   ##
==========================================
+ Coverage   94.36%   94.37%   +0.01%     
==========================================
  Files         189      189              
  Lines        8410     8410              
==========================================
+ Hits         7936     7937       +1     
+ Misses        474      473       -1     

Impacted Files	Coverage Δ
sdk/src/logs/batch_log_processor.cc	95.06% <0.00%> (+1.23%)	⬆️

[maxgolov]

@bogdandrutu - What is the maximum amount of cached artifacts for the cache action - 5 GB? Cache Limits. A repository can have up to 5GB of caches. Once the 5GB limit is reached, older caches will be evicted based .. . I think we need to assume that the other parts of the build could also be cached in that shared cache.

Do you know roughly how much space the Bazel build consumes? vcpkg build with all deps, unfortunately, consumes about 8GB on Windows (exceeds the maximum allowed cache size).

[maxgolov]

What I am trying to say is once we start caching for all build loops (not just for Bazel), the cumulative net-benefit of that will be diminished.

[jsuereth]

Hey, this gives us more time to run benchmarks!

[bogdandrutu]

@maxgolov right now it is ~1GB per bazel job, but I think with this change we can actually combine multiple jobs and I think the cache will be lower.

[maxgolov]

@bogdandrutu - 1GB is very good. I can apply similar logic for CMake + add ninja as default build for CMake. There are a few pieces which we can dramatically speed-up. gRPC being the heaviest hitter (slowest to build) right now. We may build it with vcpkg , and cache the entire (or most of) vcpkg package cache. Thus, bringing CMake build times to what you are quoting for Bazel.

[ThomsonTan]

@bogdandrutu - 1GB is very good. I can apply similar logic for CMake + add ninja as default build for CMake. There are a few pieces which we can dramatically speed-up. gRPC being the heaviest hitter (slowest to build) right now. We may build it with vcpkg , and cache the entire (or most of) vcpkg package cache. Thus, bringing CMake build times to what you are quoting for Bazel.

As you mentioned gRPC installation with vcpkg could take much more than 5GB, also vcpkg build debug/release flavors at the same time, perhaps we could separate them into different builds.

[maxgolov]

As you mentioned gRPC installation with vcpkg could take much more than 5GB, also vcpkg build debug/release flavors at the same time, perhaps we could separate them into different builds.

I'm almost thinking now we have to build and cache a docker image nightly? Or similar setup that'd preserve the prebuilt deps in GitHub Action cache. This should include a prebuilt of fresh CMake 3.1x (for older distros), Google Test, Google Framework, and gRPC. We should be able to build the image on Linux, cache the image ( e.g. using this approach https://github.com/marketplace/actions/docker-layer-caching ), then run docker-in-docker on Linux.

For Windows / vcpkg: need to see what tools we can fetch, vs. what to cache using GitHub cache action. Huge size of vcpkg package install dir is an issue indeed..

[lalitb]

@bogdandrutu - Sorry to ask, but would you be able to resolve conflict so that we can push it to master.

[lalitb]

I'm almost thinking now we have to build and cache a docker image nightly? Or similar setup that'd preserve the prebuilt deps in GitHub Action cache. This should include a prebuilt of fresh CMake 3.1x (for older distros), Google Test, Google Framework, and gRPC. We should be able to build the image on Linux, cache the image ( e.g. using this approach https://github.com/marketplace/actions/docker-layer-caching ), then run docker-in-docker on Linux.

Agree, would be good to test how docker layer caching perform here, and switch to it if we see benefit. @maxgolov has checked-in docker files so these can be reused.