Implement lease extension

[coeuvre]

Add flag --experimental_remote_cache_lease_extension, which when set, Bazel will create a background thread periodically sending FindMissingBlobs requests to CAS during the build.

1. All the outputs that were not downloaded are within the scope of lease extension. The outputs are acquired from skyframe by traversing the action graph.
2. Lease extension starts after any action was built and ends after execution phase ended. The frequency is related to --experimental_remote_cache_ttl.
3. Lease extensions are performed on action basis, not by collecting all outputs and issue one giant FindMissingBlobs.
   • Collecting all outputs might increase memory watermark and cause OOM.
   • Sending one FindMissingBlobs request per action may increase the overhead of network roundtrip, but the cost should be saturated given that the lease extension happens at background and is not wall time critical.
4. For an incremental build, the same applies: lease extension starts after any action was executed.
   • We don't want lease extension blocking action execution, nor affecting build performance.
   • Since we have TTL based cache discarding, any expired blobs will be discarded.
   • Leases of blobs that are not downloaded, still used by this build (because they are referenced by skyframe) will be extended as normal.

Part of #16660.

TODO: Sending out now for early feedbacks. Unit and integration tests will be added later.

[brentleyjones]

Just a question, but why was 9c96529 done before this was merged?

[coeuvre]

I don't think is this a blocker for 9c96529. but this is now under review and should be merged soon anyway.

[brentleyjones]

I don't think is this a blocker for 9c96529

I disagree 😅, but I'm glad it's being merged soon! 😄