Backport of [API Gateway] Fix invalid cluster causing gateway programming delay into release/1.15.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #16661 to be assessed for backporting due to the inclusion of the label backport/1.15.

WARNING automatic cherry-pick of commits failed. Commits will require human attention.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Description

We were seeing that HTTP listeners on API Gateways were taking ~15 seconds or more to get programmed onto the envoy instances (this also happens on our Kubernetes offering). I finally tracked down what was the root cause. Because we are currently masquerading as ingress gateways for the purposes of xDS generation, the main entrypoint for each HTTP listener is a "virtual" service router, basically one that doesn't correspond to an actual service, but that contains our routing logic based on the rules in an HTTPRoute. What was happening is that when we use such a router, the discovery chain compilation process adds a "fallback target" for when no rules match that targets a service resolver with the same name as the router.

Thus, if our router was named gateway-listener-12345 -- we were generating a route and resolver node for gateway-listener-12345. The problem is that our xDS code then generates cluster targets for those services, attempting to resolve gateway-listener-12345.default.default... or some such hostname. Because we specify that we'll ship the resolved endpoints via EDS over ADS, envoy then blocks on applying its configuration, keeping the gateway-listener-12345 cluster in a warming state. This, in turn, keeps the whole listener in a warming state until the time at which the EDS config source's initial_fetch_timeout value is hit. Since we weren't setting that, this defaults to 15 seconds. So, any use of a virtual router causes an ~15 second delay in applying envoy configuration.

The temporary fix with our native API gateway until we move to fully native xDS generation is to munge the compiled discovery chain that we synthesize for the API gateway and filter out both the router's default route and the resolver node for the router.

Testing & Reproduction steps

Running this script shows these timings.

Before the fix:

➜  ./timer.sh
Config entry written: proxy-defaults/global
Config entry written: api-gateway/gateway
Config entry written: http-route/router
Registered service: service
gateway not listening yet
...
gateway listening

real	0m20.379s
user	0m3.641s
sys	0m15.512s

And after:

➜  ./timer.sh
Config entry written: proxy-defaults/global
Config entry written: api-gateway/gateway
Config entry written: http-route/router
Registered service: service
gateway not listening yet
...
gateway listening

real	0m0.734s
user	0m0.137s
sys	0m0.561s

PR Checklist

• updated test coverage
• external facing docs updated
• not a security concern

---

Overview of commits

• 9ea73b3 - d3cffde - 0848aac - 90b5e39 - 32fdba9

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR