Add server_startup_timeout_ms

[Jake-Shadle]

Ever since we updated to 3.0 we've been having very rare (~1%) cases where the sccache --start-server will timeout with error: Timed out waiting for server startup. While the best option would be to figure that out and fix it, in the meantime just having a simple retry works around the issue. The problem was that the timeout is an incredibly long 10 seconds, so this PR just adds a new server_startup_timeout_ms to the config file so that users can specify the timeout they want, falling back to the 10 second default if one is not set, as when it works, this operation should complete in a few hundred milliseconds max allowing us to have a tighter retry loop.

This also removes the openssl dependency when using gcs since that seems to have snuck back in at some point #367, but let me know if you want to split that out.

[sylvestre]

I guess you don't have the courage to try to bisect the regression?

it would be nice to document this in the README.

And the CI isn't happy ;)

@glandium did you see that in our CI ?

[Jake-Shadle]

I mean I could at some point I suppose after I setup a test case to get it to reliably repro.

[codecov-commenter]

Codecov Report

Base: 23.47% // Head: 23.57% // Increases project coverage by +0.10% 🎉

Coverage data is based on head (6397c8b) compared to base (3bb3283).
Patch coverage: 44.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1438      +/-   ##
==========================================
+ Coverage   23.47%   23.57%   +0.10%     
==========================================
  Files          48       48              
  Lines       22310    22316       +6     
  Branches    10213    10211       -2     
==========================================
+ Hits         5237     5261      +24     
+ Misses      11469    11460       -9     
+ Partials     5604     5595       -9     

Impacted Files	Coverage Δ
tests/oauth.rs	0.64% <0.00%> (-0.01%)	⬇️
src/commands.rs	18.69% <30.00%> (+0.22%)	⬆️
src/config.rs	30.86% <53.84%> (+0.22%)	⬆️
tests/harness/mod.rs	42.68% <100.00%> (+0.70%)	⬆️
src/azure/blobstore.rs	21.05% <0.00%> (-1.58%)	⬇️
src/compiler/c.rs	38.30% <0.00%> (-0.46%)	⬇️
src/compiler/msvc.rs	43.52% <0.00%> (-0.14%)	⬇️
src/compiler/gcc.rs	54.92% <0.00%> (-0.11%)	⬇️
src/client.rs	0.20% <0.00%> (+0.03%)	⬆️
... and 12 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[sylvestre]

I guess you saw:

---- test_dist_restartedserver stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `0`', tests/dist.rs:121:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:182:5
   4: dist::test_dist_restartedserver::{{closure}}
             at ./tests/dist.rs:121:9
   5: dist::harness::get_stats::{{closure}}
             at ./tests/harness/mod.rs:82:13
   6: <predicates::function::FnPredicate<F,T> as predicates_core::core::Predicate<T>>::eval
             at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/predicates-2.1.2/src/function.rs:78:9
   7: predicates::utils::default_find_case
             at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/predicates-2.1.2/src/utils.rs:58:18
   8: <predicates::function::FnPredicate<F,T> as predicates_core::core::Predicate<T>>::find_case
             at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/predicates-2.1.2/src/function.rs:82:9
   9: assert_cmd::assert::Assert::stdout_impl
             at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cmd-2.0.6/src/assert.rs:377:33
  10: assert_cmd::assert::Assert::try_stdout
             at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cmd-2.0.6/src/assert.rs:371:9
  11: assert_cmd::assert::Assert::stdout
             at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cmd-2.0.6/src/assert.rs:362:9
  12: dist::harness::get_stats
             at ./tests/harness/mod.rs:76:5
  13: dist::test_dist_restartedserver
             at ./tests/dist.rs:120:5
  14: dist::test_dist_restartedserver::{{closure}}
             at ./tests/dist.rs:96:1
  15: core::ops::function::FnOnce::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ops/function.rs:227:5
  16: core::ops::function::FnOnce::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[Jake-Shadle]

I guess I hit this when using rustls instead of openssl, we can just keep those changes on our fork since they were unrelated to this change anyhow.