Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection pool shut down on "zuul.host.*" property change #3406

Closed
denysivano opened this issue Mar 2, 2019 · 4 comments · Fixed by #3407
Closed

Connection pool shut down on "zuul.host.*" property change #3406

denysivano opened this issue Mar 2, 2019 · 4 comments · Fixed by #3407

Comments

@denysivano
Copy link
Contributor

Problem:

When using Spring Boot + Spring Cloud Netflix Zuul and changing any zuul.host.* property dynamically, an exception occurred on any subsequent forwarding request:

2019-03-02 17:59:28.310  WARN 21726 --- [nio-8080-exec-5] o.s.c.n.z.filters.post.SendErrorFilter   : Error during filtering

com.netflix.zuul.exception.ZuulException: Forwarding error
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.handleException(SimpleHostRoutingFilter.java:246) ~[spring-cloud-netflix-zuul-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:226) ~[spring-cloud-netflix-zuul-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:117) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.3.1.jar:1.3.1]
	at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.3.1.jar:1.3.1]
	at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:165) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44) [spring-cloud-netflix-zuul-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) [spring-webmvc-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90) [spring-boot-actuator-2.1.3.RELEASE.jar:2.1.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:117) [spring-boot-actuator-2.1.3.RELEASE.jar:2.1.3.RELEASE]
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106) [spring-boot-actuator-2.1.3.RELEASE.jar:2.1.3.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.1.5.RELEASE.jar:5.1.5.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.16.jar:9.0.16]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.11.jar:4.4.11]
	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:191) ~[httpcore-4.4.11.jar:4.4.11]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:268) ~[httpclient-4.5.7.jar:4.5.7]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.7.jar:4.5.7]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.7.jar:4.5.7]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.7.jar:4.5.7]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.7.jar:4.5.7]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.7.jar:4.5.7]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forwardRequest(SimpleHostRoutingFilter.java:396) ~[spring-cloud-netflix-zuul-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forward(SimpleHostRoutingFilter.java:315) ~[spring-cloud-netflix-zuul-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:221) ~[spring-cloud-netflix-zuul-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	... 63 common frames omitted

Cause:

When changing zuul.host.* property, SimpleHostRoutingFilter re-creates Apache HttpClient, closing the previous one.

By default, when HttpClient is being closed, it also closes the underlying HttpClientConnectionManager, except explicitly specifying .setConnectionManagerShared(true) on HttpClientBuilder instance during client creation.

SimpleHostRoutingFilter uses the single instance of HttpClientConnectionManager for creating new HttpClients. On property change, it closes HttpClient and the underlying HttpClientConnectionManager, which leads to an exception described above.

Code example:

Here is the simple app for reproducing this problem: connection-pool-shut-down.zip

It starts on port 8080, forwards requests /api/actuator/** to http://localhost:8080/actuator/** and provides endpoint /change for changing zuul.host.maxConnections property.

Steps to reproduce:

  • mvn clean verify spring-boot:run
  • curl http://localhost:8080/api/actuator/health
  • curl http://localhost:8080/change
  • curl http://localhost:8080/api/actuator/health
@spencergibb
Copy link
Member

Closing in favor #3407

@samhalpa
Copy link

samhalpa commented Mar 4, 2019

We are getting this error frequently. Is there a workaround for this for the time being?

@denysivano
Copy link
Contributor Author

denysivano commented Mar 5, 2019

We are getting this error frequently. Is there a workaround for this for the time being?

You can provide your own client by specifying the bean of type CloseableHttpClient, which will be used for creating SimpleHostRoutingFilter (see ZuulProxyAutoConfiguration):

@Bean
public CloseableHttpClient httpClient() {
    return HttpClientBuilder.create()
            // configure max connections, timeouts etc.
            .build();
}

Check SimpleHostRoutingFilter class for HTTP client configuration details.

Updated. Another approach is to explicitly define bean of type SimpleHostRoutingFilter with wrapped ApacheHttpClientFactory:

@Bean
public SimpleHostRoutingFilter simpleHostRoutingFilter(ProxyRequestHelper helper,
        ZuulProperties zuulProperties,
        ApacheHttpClientConnectionManagerFactory connectionManagerFactory,
        ApacheHttpClientFactory httpClientFactory) {

    // Define the connection manager to be shared, preventing it from
    // being shut down by the client
    ApacheHttpClientFactory newHttpClientFactory = () -> httpClientFactory
            .createBuilder().setConnectionManagerShared(true);

    return new SimpleHostRoutingFilter(helper, zuulProperties,
            connectionManagerFactory, newHttpClientFactory);
}

@samhalpa
Copy link

samhalpa commented Mar 6, 2019

Thanks, @denysivano, I 'll have a go on the mentioned workarounds.

ryanjbaxter pushed a commit that referenced this issue Sep 3, 2019
* Re-create connection manager on "zuul.host.*" property change 

Fixes #3406

* Wrap connectionManager.shutdown() in try/catch block. Polishing
sabareeshkkanan added a commit to sabareeshkkanan/spring-cloud-netflix that referenced this issue Dec 20, 2019
* Added symbolic link of index.adoc

* Added symbolic link of index.adoc

* set replication client filters in RefreshablePeerEurekaNodes (spring-cloud#3610)

fixes spring-cloudgh-3554

* Update SNAPSHOT to 2.2.0.M2

* Going back to snapshots

* Adding spring cloud circuitbreaker hystrix implementation

* Separated circuitbreaker auto config to its own file

* Initialize remoteRegionAppWhitelist with default value (spring-cloud#3634)

* Moves non-netflix dependencies out of bom into root pom.

fixes spring-cloudgh-3639

* Removes dependency management for okhttp3

* Moves okhttp3 dep mgmt back to bom

* Re-create connection manager on "zuul.host.*" property change (spring-cloud#3407)

* Re-create connection manager on "zuul.host.*" property change 

Fixes spring-cloud#3406

* Wrap connectionManager.shutdown() in try/catch block. Polishing

* Removes okhttp3 again after added back in older branch.

* Updating readme with note about building spring-cloud-netflix-hystrix-contract.  Fixes spring-cloud#3497

* Fromatting

* Upgrades eureka to 1.9.13 and excludes compactmap.

fixes spring-cloudgh-3636

* Update SNAPSHOT to 2.1.3.RELEASE

* Going back to snapshots

* Bumping versions to 2.1.4.BUILD-SNAPSHOT after release

* Add Spring Cloud LoadBalancer starter to Eureka starters. Fixes spring-cloudgh-3646. (spring-cloud#3647)

* Update docs. (spring-cloud#3650)

* ConditionalOnMissingBean on formBodyWrapperFilter, debugFilter… (spring-cloud#3609)

* Bumping versions

* Remove spring.provides.

* removes useless comments

* Fix RestTemplateEurekaHttpClient status update endpoint.  Fixes spring-cloud#3571 (spring-cloud#3657)

* polish

* Updates health check handler to use new StatusAggregator

* Gh 3409 turbine stream test (spring-cloud#3665)

* Add stream-test-support.

* Fix condition.

* Remove outdated workaround.

* Bumping versions

* Optimize code of eureka (spring-cloud#3660)

* Optimize code of eureka

* Merge newest code

* fix checkstyle bug

* Gh 3464 upgrade scc new (spring-cloud#3667)

* Upgrade Spring Cloud Contract version to 2.1.3.RELEASE.

* Gh 3409 turbine stream test (spring-cloud#3665)

* Add stream-test-support.

* Fix condition.

* Remove outdated workaround.

(cherry picked from commit 24f5e0b)

* Updates to use "components" rather than "details"

See spring-projects/spring-boot#17929

* Applying spring-cloud#3407 to the 2.1.x branch

* Added support for reactive service discovery

* Update SNAPSHOT to 2.2.0.M3

* Going back to snapshots

* Update SNAPSHOT to 2.2.0.RC1

* Going back to snapshots

* removing resource class from circle config

* Add property to disable spring cloud circuit breaker for hystrix

* Fix command key configuration.  Use id as command key and class as group key.

* Adding configuration metadata for spring cloud circuitbreaker

* Bumping versions

* Fix some dependencies that show up in the wrong scope

Apparently you can build on the command line but Eclipse is fussy
now and wouldn't compile these projects without explicit
dependencies.

* Also add build helper config for contract tests

* Added maven flatten plugin

* Bumping versions

* Update SNAPSHOT to 2.2.0.RC2

* Going back to snapshots

* Update configuration to use proxyBeanMethods=false.  Fixes spring-cloud#3677

* Create security.md

* Update issue templates

* Bumping versions

* Bumping versions

* Update SNAPSHOT to 2.2.0.RELEASE

* Going back to snapshots

* Bumping versions to 2.2.1.BUILD-SNAPSHOT after release

* Bumping versions

* Fix typo: clas -> class (spring-cloud#3710)

* removes .flattened-pom.xml

* ignores .flattened-pom.xml

* Gh 3718 add zoned loadbalancer instrumentation (spring-cloud#3720)

* Add instrumentation for zoned LoadBalancer.

* Add documentation.

* Fix after review.

* Fix after review.

* Update SNAPSHOT to 2.2.1.RELEASE

* Going back to snapshots

* Bumping versions to 2.2.2.BUILD-SNAPSHOT after release

Co-authored-by: Marcin Grzejszczak <marcin@grzejszczak.pl>
Co-authored-by: Yuxin Bai <LittleBaiBai@users.noreply.github.com>
Co-authored-by: Spencer Gibb <spencer@gibb.tech>
Co-authored-by: Spring Buildmaster <buildmaster@springframework.org>
Co-authored-by: Ryan Baxter <rbaxter@pivotal.io>
Co-authored-by: emilnkrastev <emilnkrastev@gmail.com>
Co-authored-by: Denys Ivano <denys.ivano@gmail.com>
Co-authored-by: Olga Maciaszek-Sharma <olga.maciaszek@gmail.com>
Co-authored-by: Rafał Żukowski <rzukow@gmail.com>
Co-authored-by: OLPMO <OLPMO@users.noreply.github.com>
Co-authored-by: Tim Ysewyn <Tim.Ysewyn@me.com>
Co-authored-by: Dave Syer <david_syer@hotmail.com>
Co-authored-by: Deepika Mohan <deepikadevidm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants