Race between active healthchecks and secrets delivery

[dastbe]

WIP: getting some reference configuration and logs to better illustrate the problem

We've been testing Envoy initialization times and we noticed that under certain scenarios Envoy takes much longer (60+s) to initialize. We were able to root cause this down to what we believe is a race condition between when TLS is configured on a cluster (ex. validation context) and when active healthcheck begins.

Specifically, we observe that the SDS request for tls context secret is sent roughly in parallel for when the active healthcheck begins. Because of this, the initial healthcheck will fail and the healthcheck interval will fall back to the no_traffic_interval. While for STATIC_DNS and EDS this appears to not delay Envoy initialization, it appears that Envoy using LOGICAL_DNS will wait out the no_traffic_interval to healthcheck again before it considers itself fully initialized.

[mattklein123]

I think this is a dup of #12389?

[dastbe]

Ah, yeah! I have another issue related to this (STATIC_DNS healthchecking) but will cut a new issue for that. Closing.