Regression in Jetty 9.4.39 with Wicket and Multipart Requests: Early EOF

[StefanBauerTT]

• Jetty 9.4.38 (no issues) / Jetty 9.4.39 (regression)
• AdoptOpenJDK (build 11.0.10+9)
• Windows 10 Professional

In our project, we are facing a regression since updating from Jetty 9.4.38 to 9.4.39. In our pre-production environment, we are experiencing irregular but frequent connection losses with the following stacktrace:

org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Early EOF
	at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:350) ~[commons-fileupload-1.4.jar:1.4]
	at org.apache.wicket.protocol.http.servlet.MultipartServletWebRequestImpl.parseFileParts(MultipartServletWebRequestImpl.java:196) ~[wicket-core-8.11.0.jar:8.11.0]
	at UploadResource.newResourceResponse
	at org.apache.wicket.request.resource.AbstractResource.respond(AbstractResource.java:630) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.request.handler.resource.ResourceRequestHandler.respond(ResourceRequestHandler.java:105) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.request.handler.resource.ResourceReferenceRequestHandler.respond(ResourceReferenceRequestHandler.java:100) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:903) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.request.RequestHandlerExecutor.execute(RequestHandlerExecutor.java:65) ~[wicket-request-8.11.0.jar:8.11.0]
	at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:292) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:253) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:276) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:207) ~[wicket-core-8.11.0.jar:8.11.0]
	at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:306) ~[wicket-core-8.11.0.jar:8.11.0]
    --- private classes ---
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) ~[guice-servlet-4.2.3.jar:?]
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) ~[guice-servlet-4.2.3.jar:?]
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) ~[guice-servlet-4.2.3.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201) ~
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) 
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) 
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279) 
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) 
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) 
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) 
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) 
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) 
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) 
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) 
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) 
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) 
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) 
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) 
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) 
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) 
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) 
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.eclipse.jetty.io.EofException: Early EOF
	at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1187) ~[jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1175) ~[jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:333) ~[jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
	at org.apache.commons.fileupload.util.LimitedInputStream.read(LimitedInputStream.java:132) ~[commons-fileupload-1.4.jar:1.4]
	at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:1027) ~[commons-fileupload-1.4.jar:1.4]
	at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:931) ~[commons-fileupload-1.4.jar:1.4]
	at java.io.InputStream.read(InputStream.java:205) ~[?:?]
	at org.apache.commons.fileupload.util.Streams.copy(Streams.java:98) ~[commons-fileupload-1.4.jar:1.4]
	at org.apache.commons.fileupload.util.Streams.copy(Streams.java:68) ~[commons-fileupload-1.4.jar:1.4]
	at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:346) ~[commons-fileupload-1.4.jar:1.4]
	... 60 more

From the client side, the error looks like the server closed the connection (Example using Python's requests library):

('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

Our endpoint is implemented as a Wicket dynamic Resource that accepts a HTTP POST request with multipart body. As you can see in the stack trace, it fails when parsing the multipart content using Wicket's MultipartServletWebRequest, which in turn uses Apache Commons Fileupload. I cannot disclose the full source code of the Wicket Resource, but tried to strip it to a minimal example for your reference (see attached file).

Our technology stack is as follows:

• Windows 10 Professional
• AdoptOpenJDK (build 11.0.10+9)
• Jetty 9.4.38 (no issues) / Jetty 9.4.39 (regression)
• HTTPS served by Jetty's SslContextFactory.Server
• Wicket 8.11.0
• Commons Fileupload 1.4

In the server application, Jetty is directly embedded into the WAR file (using the maven shade plugin) so that the application can be started standalone like a JAR.

The clients which are sending the requests are implemented both in Java and in Python and have not changed.

Unfortunately I was not able to reproduce the issue on a development system, so my testing capabilities are limited to our pre-production environment.

[joakime]

I see SSL in your stacktrace, this is a likely side effect of what was recently fixed in #6082

Check your logs fore before the Early EOF failure.
Look for "Encrypted buffer max length exceeded".

If you can replicate easily, can you try 9.4.40-SNAPSHOT from the https://oss.sonatype.org/content/groups/jetty-with-staging/ (SNAPSHOT repo)

[joakime]

Note: Jetty 9.4.40.v20210413 was just released to Maven Central (with the fix for #6082), which should help with your issue on Early EOF with multipart.

[StefanBauerTT]

Thank you for the quick reply. I haven't seen the Early EOF problem since we updated to jetty 9.4.40. Looks like this issue can be closed. Thanks and have a great weekend!