Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document that SimpleClientHttpRequestFactory (and URLConnection which it uses) is not suitable for the WebMVC gateway server. #3451

Open
haeuserd opened this issue Jul 11, 2024 · 18 comments

Comments

@haeuserd
Copy link

Describe the bug

With the changes of #3405 (introduced in version 4.1.4) the error handling does not work properly any more when using DefaultRestClient.

When the client returns any error status code (e.g. 400), the DefaultRestClient raises an org.springframework.web.client.ResourceAccessException when trying to read the response body which happens RestClientProxyExchange.

However the ResourceAccessException does not provide any accessible information about the status code other than the message text. Therefore we cannot handle it properly and only respond with 500 Internal Server Error by default.

A workaround is to use a different http client. But actually I would expect Spring Cloud Gateway to work out of the box with Spring's default client. However, you may have other opinions on this.

Sample

I'm about half an hour before my three week holiday starts, so unfortunately I'm not able to provide a reproducable example any more. I can do that when I get back, if that helps. All I can do for now is to provide the error stack trace:

2024-07-11T16:00:53.232+02:00  WARN 249503 --- [api-outages-proxy] [omcat-handler-0] d.e.a.p.e.ErrorEntityExceptionHandler    : I/O error on POST request for "http://localhost:11112/api/outages": Server returned HTTP response code: 400 for URL: http://localhost:11112/api/outages

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:11112/api/outages": Server returned HTTP response code: 400 for URL: http://localhost:11112/api/outages
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.createResourceAccessException(DefaultRestClient.java:575) ~[spring-web-6.1.10.jar:6.1.10]
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:498) ~[spring-web-6.1.10.jar:6.1.10]
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchange(DefaultRestClient.java:465) ~[spring-web-6.1.10.jar:6.1.10]
	at org.springframework.cloud.gateway.server.mvc.handler.RestClientProxyExchange.exchange(RestClientProxyExchange.java:42) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
	at org.springframework.cloud.gateway.server.mvc.handler.ProxyExchangeHandlerFunction.handle(ProxyExchangeHandlerFunction.java:120) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
	at org.springframework.cloud.gateway.server.mvc.handler.HandlerFunctions$LookupProxyExchangeHandlerFunction.handle(HandlerFunctions.java:107) ~[spring-cloud-gateway-server-mvc-4.1.4.jar:4.1.4]
@jordanjennings
Copy link

We have hit this bug also - we have a backend we expect to be returning a 404, but after upgrading to 4.1.4 that 404 produces a 500 error to the client instead.

Following the flow of the code it looks like HttpURLConnection throws a FileNotFound exception on 404, which due to this line of code is now throwing in a different place than it previously would have, and the exception isn't handled from here correctly:

https://github.com/spring-cloud/spring-cloud-gateway/blame/656ab5aba4a25738e2e7af1a2bb7846667667e3b/spring-cloud-gateway-server-mvc/src/main/java/org/springframework/cloud/gateway/server/mvc/handler/RestClientProxyExchange.java#L51

This is a blocker for upgrading for us, and I hope it can be addressed soon. For now we'll stay on the older version.

@haeuserd
Copy link
Author

Here is a minimal reproducable example:

application.properties:

spring.cloud.gateway.mvc.http-client.type=autodetect

Gateway Application:

@SpringBootApplication
public class GatewayDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(GatewayDemoApplication.class, args);
    }

    @Bean
    public RouterFunction<ServerResponse> getRoute() {
        return route().GET("/status/*", http("https://httpbin.org/")).build();
    }
}

Example request resulting in http status code 500 instead of correct response code:

curl -i http://localhost:8080/status/400

Stacktrace:

2024-07-30T14:33:35.442+02:00 ERROR 201951 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://httpbin.org/status/400": Server returned HTTP response code: 400 for URL: https://httpbin.org/status/400] with root cause

java.io.IOException: Server returned HTTP response code: 400 for URL: https://httpbin.org/status/400
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1998) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
	at org.springframework.http.client.SimpleClientHttpRequest.executeInternal(SimpleClientHttpRequest.java:88) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:492) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchange(DefaultRestClient.java:465) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.cloud.gateway.server.mvc.handler.RestClientProxyExchange.exchange(RestClientProxyExchange.java:42) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
	at org.springframework.cloud.gateway.server.mvc.handler.ProxyExchangeHandlerFunction.handle(ProxyExchangeHandlerFunction.java:120) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
	at org.springframework.cloud.gateway.server.mvc.handler.HandlerFunctions$LookupProxyExchangeHandlerFunction.handle(HandlerFunctions.java:107) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
	at org.springframework.web.servlet.function.support.HandlerFunctionAdapter.handle(HandlerFunctionAdapter.java:108) ~[spring-webmvc-6.1.11.jar:6.1.11]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.11.jar:6.1.11]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.11.jar:6.1.11]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.11.jar:6.1.11]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.1.11.jar:6.1.11]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.1.11.jar:6.1.11]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.springframework.cloud.gateway.server.mvc.filter.WeightCalculatorFilter.doFilter(WeightCalculatorFilter.java:229) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.springframework.cloud.gateway.server.mvc.filter.FormFilter.doFilter(FormFilter.java:93) ~[spring-cloud-gateway-server-mvc-4.1.5.jar:4.1.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.11.jar:6.1.11]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.11.jar:6.1.11]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.26.jar:10.1.26]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

@mendiCap
Copy link

Any update on this? Am stuck here for some time, maybe downgrading to an older version?

@spencergibb
Copy link
Member

But actually I would expect Spring Cloud Gateway to work out of the box with Spring's default client.

The default client in Spring Cloud Gateway WebMVC is the jdk HttpClient, not HttpURLConnection

@spencergibb spencergibb added this to the 4.1.6 milestone Oct 28, 2024
@spencergibb spencergibb moved this to Todo in 2024.0.0-RC1 Oct 28, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in 2024.0.0-RC1 Oct 28, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in 2023.0.4 Oct 28, 2024
@mendiCap
Copy link

mendiCap commented Oct 29, 2024

@spencergibb I've forked the latest version of branch 4.1.x to test the changes so that I can continue testing my gateway.
However, seems like the issue isn't quite fixed yet. The ResourceAccessException gets thrown after an IO error.
Which occurs here:

	try {
			InputStream body = clientResponse.getBody(); // ---------HERE
			// put the body input stream in a request attribute so filters can read it.
			MvcUtils.putAttribute(request.getServerRequest(), MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR, body);
		}
		catch (FileNotFoundException e) {
			// if using SimpleClientHttpRequestFactory
			return ServerResponse.notFound().build();
		}

So the catch doesn't get called.
Which causes a ResourceAccessException later and returning the default TomCat error HTML page in the API response.

Maybe the following might help:
Before the program executes clientResponse.getBody() I do it manually in the debugger, which causes:
First click

When executing it again in the debugger console:
It does work
second click

I can even parse the error message:
Parsed JSON error message

Is there any alternative till this issue is resolved?

@spencergibb
Copy link
Member

@mendiCap I don't get a ResourceAccessException, I get a FileNotFoundException. Can you tell me how to recreate your specific situation?

@spencergibb
Copy link
Member

Ah, I'm testing 404 specifically.

@spencergibb spencergibb reopened this Oct 29, 2024
@mendiCap
Copy link

mendiCap commented Oct 29, 2024

Currently I'm testing a 409.

Do you know of a previous version this would work?

@mendiCap
Copy link

@spencergibb
I managed to fix it with the following added catch method:

	try {
			InputStream body = clientResponse.getBody();
			// put the body input stream in a request attribute so filters can read it.
			MvcUtils.putAttribute(request.getServerRequest(), MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR, body);
		}
		catch (FileNotFoundException e) {
			// if using SimpleClientHttpRequestFactory
			return ServerResponse.notFound().build();
		}
		catch (IOException e) {
			return ServerResponse.status(clientResponse.getStatusCode())
				.body(new String(clientResponse.getBody().readAllBytes(), StandardCharsets.UTF_8));
		}

This returns the correct status code and error body for the downstream backend.

However, I'm not sure it's a good fix because I couldn't use an after-filter, the headers become read-only.

P.s: I used spring-cloud-starter-gateway-mvc, the initial changes for this bug were on spring-cloud-server-gateway-mvc.
Is there a difference or should it be changed on both?

@spencergibb
Copy link
Member

I don't think we can arbitrarily catch IOException here. Other filters, such as circuitbreaker and retry rely on exceptions to function correctly. I'm worried that this is a rabbit hole I don't want to go down and URLConnection is not a very good http client for a proxy. Is there a reason you are not using the jdk http client?

@Mendistern
Copy link

This is the default config, I haven't explicity used that.
How can I use the jdk one?

@spencergibb
Copy link
Member

spencergibb commented Oct 29, 2024

@Mendistern it is not the default. The jdk http client is the default unless you set spring.cloud.gateway.mvc.http-client.type=autodetect

@spencergibb spencergibb moved this from Done to In Progress in 2023.0.4 Oct 29, 2024
@spencergibb spencergibb moved this from Done to In Progress in 2024.0.0-RC1 Oct 29, 2024
@mendiCap
Copy link

mendiCap commented Oct 30, 2024

@spencergibb
Some context:
I've an Angular frontend, Spring Backend, Oauth, and the gateway.
The gateway handles the Oauth flow and routes requests to the frontend or backend.

I've removed the autodetect line. But without this, it doesn't forward requests and the browser stays on loading state.
Even though the filter was found:

 Predicate "/**" matches against "HTTP GET /my-deposits/new-modification-deposit"
2024-10-30T10:18:22.999+01:00 TRACE 17312 --- [edepot-api-gateway] [nio-7081-exec-6] o.s.w.s.f.support.RouterFunctionMapping  : Mapped to org.springframework.web.servlet.function.HandlerFilterFunction$$Lambda/0x000002331da8ce48@7f60d7ac
2024-10-30T10:18:22.999+01:00 DEBUG 17312 --- [edepot-api-gateway] [nio-7081-exec-6] o.j.s.OpenEntityManagerInViewInterceptor : Opening JPA EntityManager in 

I tried setting http-client.type=jdk, but this also doesn't help.
I tried manually setting an HttpClient bean:

   @Bean
    public HttpClient httpClient() {
        return HttpClient.newBuilder()
                .version(HttpClient.Version.HTTP_2)
                .build();
    }

Also doesn't work.

Here's my application.properties:

spring.application.name=api-gateway
server.port=${SERVER_PORT}


#### FILTERS

#BFF
spring.cloud.gateway.mvc.routes[0].id=api
spring.cloud.gateway.mvc.routes[0].uri=${backend-uri}
spring.cloud.gateway.mvc.routes[0].predicates[0]=Path=/api/**
spring.cloud.gateway.mvc.routes[0].filters[0].name=DedupeResponseHeader
spring.cloud.gateway.mvc.routes[0].filters[0].args[name]=Access-Control-Allow-Credentials Access-Control-Allow-Origin
spring.cloud.gateway.mvc.routes[0].filters[1].name=TokenRelay
#spring.cloud.gateway.mvc.routes[0].filters[2]=AddResponseHeader=Content-Type, application/json



#Back-channel logout
spring.cloud.gateway.mvc.routes[1].id=auth-route
spring.cloud.gateway.mvc.routes[1].uri=${scheme}://${hostname}:${bff-port}
spring.cloud.gateway.mvc.routes[1].predicates[0]=Path=/bff/**
#spring.cloud.gateway.mvc.routes[1].filters[0]=StripPrefix=1


#Frontend routes
spring.cloud.gateway.mvc.routes[2].id=frontend
spring.cloud.gateway.mvc.routes[2].uri=${frontend-uri}
spring.cloud.gateway.mvc.routes[2].predicates=Path=/**
# Forwarding support
#spring.cloud.gateway.mvc.http-client.type=jdk
#spring.cloud.gateway.mvc.http-client.connect-timeout=60s
#spring.cloud.gateway.mvc.http-client.read-timeout=60s
#spring.cloud.mvc.discovery.enabled=true

# Spring JDBC Session
spring.session.store-type=jdbc
server.servlet.session.timeout=600
spring.session.jdbc.initialize-schema=never
spring.session.jdbc.table-name=AUTH_SESSION
spring.datasource.url=jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?currentSchema=${DB_SCHEMA}
spring.datasource.username=${DB_USER}
spring.datasource.password=${DB_PASSWORD}
# Flyway
spring.flyway.enabled=true
spring.flyway.locations=classpath:db/migration
spring.flyway.table=flyway_spring_session
spring.flyway.baselineOnMigrate=true
spring.flyway.baselineVersion=0

#spring.cloud.gateway.httpserver.wiretap=true
#spring.cloud.gateway.httpclient.wiretap=true


Other than that I haven't customized anything besides the Oauth

@Mendistern
Copy link

I did manage to make it work with the apache http 5 module. Is this compatible with the gateway?

@spencergibb
Copy link
Member

@Mendistern yes it is.

@mendiCap
Copy link

Alright! Thank you for your help

@spencergibb
Copy link
Member

spencergibb commented Oct 31, 2024

After discussion with the team, I've reverted the original change for the FileNotFoundException 7a41f6a and won't be adding any other workarounds for SimpleClientHttpRequestFactory. We can use this issue to document that SimpleClientHttpRequestFactory (and URLConnection which it uses) is not suitable for the WebMVC gateway server.

@spencergibb spencergibb removed this from 2023.0.4 Oct 31, 2024
@spencergibb spencergibb removed this from the 4.1.6 milestone Oct 31, 2024
@spencergibb spencergibb changed the title Broken error handling when using DefaultRestClient that respons with error status code Document that SimpleClientHttpRequestFactory (and URLConnection which it uses) is not suitable for the WebMVC gateway server. Oct 31, 2024
@haeuserd
Copy link
Author

haeuserd commented Nov 12, 2024

Okay, thank you very much for clarifying this matter.

I misunderstood the actual purpose and impact of spring.cloud.gateway.mvc.http-client.type=autodetect.
I think with #3571 this gets much more intuitive 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants