Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: tcp listener is rejected when no route attached #4681

Merged
merged 13 commits into from
Nov 20, 2024

Conversation

zhaohuabing
Copy link
Member

@zhaohuabing zhaohuabing commented Nov 8, 2024

Create a default route pointing to a static cluster without endpoints when there is no TCPRoutes attached to a TCP listener. This avoids Envoy rejecting a listener without a filter chain and reporting warning.

Fixes #4680

Release Notes: Yes

Alternative consideration:
An alterantive approach could be to not create the xDS listener, but this behavior doesn't align with the Gateway semantics(which does defines a listener) and the behavior of the empty HTTP listener.

Other type of listeners:

  • A HTTP listener without HTTPRoute will create the xDS HTTP listener and an empty RouteConfiguration.
  • An UDP listener without UDPRoute won't create the xDS UDP listerner at all.

@zhaohuabing zhaohuabing requested a review from a team as a code owner November 8, 2024 08:49
@zhaohuabing zhaohuabing force-pushed the fix-tcp-listener-without-route branch 2 times, most recently from cb94a37 to 17f3476 Compare November 8, 2024 09:00
Copy link

codecov bot commented Nov 8, 2024

Codecov Report

Attention: Patch coverage is 68.42105% with 6 lines in your changes missing coverage. Please review.

Project coverage is 65.62%. Comparing base (8a01dd6) to head (b037abe).
Report is 17 commits behind head on main.

Files with missing lines Patch % Lines
internal/xds/translator/translator.go 68.42% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4681      +/-   ##
==========================================
+ Coverage   65.56%   65.62%   +0.05%     
==========================================
  Files         211      211              
  Lines       31998    32017      +19     
==========================================
+ Hits        20979    21010      +31     
+ Misses       9775     9764      -11     
+ Partials     1244     1243       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
@zhaohuabing zhaohuabing force-pushed the fix-tcp-listener-without-route branch from 17f3476 to 9bcea0b Compare November 8, 2024 09:11
@arkodg
Copy link
Contributor

arkodg commented Nov 8, 2024

thanks for looking into this @zhaohuabing
can you share what the flow looks like from a client perspective (e.g. using curl) when we either dont create a tcp listener vs tcp listener with dummy filter chain

@zhaohuabing
Copy link
Member Author

zhaohuabing commented Nov 12, 2024

thanks for looking into this @zhaohuabing can you share what the flow looks like from a client perspective (e.g. using curl) when we either dont create a tcp listener vs tcp listener with dummy filter chain

can't connect vs connection reset by peer, the latter seems make more sense as there's a listening port on Envoy for the defined Gateway listener.

no tcp listener:

curl -v http://${GATEWAY_HOST}:8080
*   Trying 172.18.0.200:8080...
* connect to 172.18.0.200 port 8080 from 172.18.0.1 port 54918 failed: Connection refused
* Failed to connect to 172.18.0.200 port 8080 after 0 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to 172.18.0.200 port 8080 after 0 ms: Couldn't connect to server

dummy filter chain

curl -v http://${GATEWAY_HOST}:8080                    
*   Trying 172.18.0.200:8080...
* Connected to 172.18.0.200 (172.18.0.200) port 8080
> GET / HTTP/1.1
> Host: 172.18.0.200:8080
> User-Agent: curl/8.5.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

@arkodg
Copy link
Contributor

arkodg commented Nov 13, 2024

hey @zhaohuabing I prefer if we dont generate an xds listener for this case
wdyt @envoyproxy/gateway-maintainers

@zirain
Copy link
Member

zirain commented Nov 19, 2024

hey @zhaohuabing I prefer if we dont generate an xds listener for this case wdyt @envoyproxy/gateway-maintainers

it would be better if possiable.

@arkodg
Copy link
Contributor

arkodg commented Nov 19, 2024

from @haq204's comment #4680 (comment) , we may want to have a listener up for responding to TCP health checks

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
// If there are no routes, add a route without a destination to the listener to create a filter chain
// This is needed because Envoy requires a filter chain to be present in the listener, otherwise it will reject the listener and report a warning
if len(tcpListener.Routes) == 0 {
nullRouteCluster := &clusterv3.Cluster{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey curious, what happens if we set xdsListener.FilterChains = [] for this case, does xDS still complain ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Envoy will complain no filter chains specified. It's a validation inside the Envoy listener impl instead of a proto type built-in validation.

https://github.com/envoyproxy/envoy/blob/79958991ffe0cf8d59a4a351646c76d672dada83/source/common/listener_manager/listener_impl.cc#L763-L770

type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) gateway-conformance-infra/same-namespace/tls: error adding listener '0.0.0.0:10443': no filter chains specified

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
- endpoints:
- host: "1.2.3.4"
port: 50000
routes:
Copy link
Member Author

@zhaohuabing zhaohuabing Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bug in the listener-connection-limit test. The routes field in the TCP Listener ir is missing.
This is not directly related to the issue addressed by this PR, but an empty cluster will be added if no route defined for the test.

- endpoints:
- host: "1.2.3.4"
port: 50000
routes:
Copy link
Member Author

@zhaohuabing zhaohuabing Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
- endpoints:
- host: "1.2.3.4"
port: 50000
routes:
Copy link
Member Author

@zhaohuabing zhaohuabing Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

- endpoints:
- host: "1.2.3.4"
port: 50000
routes:
Copy link
Member Author

@zhaohuabing zhaohuabing Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
port: 50000
- host: "5.6.7.8"
port: 50001
routes:
Copy link
Member Author

@zhaohuabing zhaohuabing Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
port: 50000
- host: "5.6.7.8"
port: 50001
routes:
Copy link
Member Author

@zhaohuabing zhaohuabing Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
@@ -66,6 +75,13 @@ tcp:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
name: envoy-gateway/gateway-1/tcp1
port: 10080
routes:
- destination:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

@zhaohuabing zhaohuabing requested a review from arkodg November 19, 2024 07:23
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Copy link
Contributor

@arkodg arkodg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks !

@zirain
Copy link
Member

zirain commented Nov 20, 2024

/retest

@arkodg arkodg merged commit f99c36c into envoyproxy:main Nov 20, 2024
23 of 24 checks passed
@zhaohuabing zhaohuabing deleted the fix-tcp-listener-without-route branch November 20, 2024 01:40
zhaohuabing added a commit to zhaohuabing/gateway that referenced this pull request Nov 22, 2024
* fix: tcp listener is rejected when no route attached

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* change cluter name

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connection limit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connetcp keepalive  test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp endpoint stats test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp-route-enable-req-resp-sizes-stats

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix extensionpolicy-tcp-udp-http test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
(cherry picked from commit f99c36c)
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
zhaohuabing added a commit that referenced this pull request Nov 27, 2024
* fix: tcp listener is rejected when no route attached (#4681)

* fix: tcp listener is rejected when no route attached

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* change cluter name

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connection limit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connetcp keepalive  test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp endpoint stats test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp-route-enable-req-resp-sizes-stats

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix extensionpolicy-tcp-udp-http test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
(cherry picked from commit f99c36c)
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix: remove backendrefs validation (#4705)

* remove backendrefs validation

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* add tests

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* add tests

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 5068698)
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix: translator reports errors for existing clusters and secretes (#4707)

* fix: existing clusters and secretes

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix cluster index for SP

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* minor change

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* minor change

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* minor change

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* minor change

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* add comment

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* remove index

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* xds: always use `::` and `IPv4Compact` for dynamic listener (#4743)

* enable IPv4Compact

Signed-off-by: zirain <zirain2009@gmail.com>

* fix xds test

Signed-off-by: zirain <zirain2009@gmail.com>

* release-notes

Signed-off-by: zirain <zirain2009@gmail.com>

* nit

Signed-off-by: zirain <zirain2009@gmail.com>

* gen

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 78da42c)
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* Fix: frequent 503 errors when connecting to a Service experiencing high Pod churn (#4754)

* Revert "fix: some status updates are discarded by the status updater (#4337)"

This reverts commit 14830c7.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* store update events and process it later

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* rename method

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* add release note

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* xds: use V4_PREFERRED dnsLookupFamily by default (#4745)

* use Cluster_V4_PREFERRED

Signed-off-by: zirain <zirain2009@gmail.com>

* release notes

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: zirain <zirain2009@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
guydc pushed a commit to guydc/gateway that referenced this pull request Nov 27, 2024
* fix: tcp listener is rejected when no route attached

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* change cluter name

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connection limit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connetcp keepalive  test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp endpoint stats test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp-route-enable-req-resp-sizes-stats

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix extensionpolicy-tcp-udp-http test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
(cherry picked from commit f99c36c)
Signed-off-by: Guy Daich <guy.daich@sap.com>
guydc added a commit that referenced this pull request Nov 27, 2024
* fix: BackendTlsPolicy specify multiple targetRefs of the same service, only one will work (#4630)

* add tests

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix matching comparison

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* add release note

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
(cherry picked from commit 44c2f74)
Signed-off-by: Guy Daich <guy.daich@sap.com>

* fix: tcp listener is rejected when no route attached (#4681)

* fix: tcp listener is rejected when no route attached

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* change cluter name

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connection limit test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix listener connetcp keepalive  test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp endpoint stats test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix tcp-route-enable-req-resp-sizes-stats

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix extensionpolicy-tcp-udp-http test

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
(cherry picked from commit f99c36c)
Signed-off-by: Guy Daich <guy.daich@sap.com>

* Fix: frequent 503 errors when connecting to a Service experiencing high Pod churn (#4754)

* Revert "fix: some status updates are discarded by the status updater (#4337)"

This reverts commit 14830c7.

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* store update events and process it later

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* rename method

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* add release note

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
(cherry picked from commit 8ec3095)
Signed-off-by: Guy Daich <guy.daich@sap.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Guy Daich <guy.daich@sap.com>
Co-authored-by: Huabing Zhao <zhaohuabing@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TCP listener creation failed when there is no TCP route defined
3 participants