Connections maxing out for requests

[souravgupta6nov1992]

Jetty version(s)
11.0.17
Java version/vendor (use: java -version)
17.0.8
Spring boot version(s)
2.7.20.2
OS type/version
Linux

We are using jetty over http/2
We are facing an issue where our requests start failing even on a lower traffic run for a particular fqdn. The server we are trying to connect to is working fine. When we collected jetty logs for our requests, we are observing that all the requests are trying to make a new connection with total connections maxed out and when it tries to reserve an entry from the pool, it is not able to do that(according to our understanding of jetty so far).Can you please have a look at the attached logs for a request and guide us on what could be leading to this situation. Every other request is going through the same repeated logs.
workerafter504msgid.log

[joakime]

Jetty 11 is now at End of Community Support

• EoCS (End of Community Support) for Jetty 10 / Jetty 11 - January 2024 #10485

You should be using Jetty 12 at this point in time.
Also note that your version, 11.0.17 is subject to several security advisories right now - https://eclipse.dev/jetty/security_reports.php

[gregw]

If you can reproduce on jetty 12, then can obtain a dump of both the JVM stacks and the jetty client (I assume it is the client you are reporting this issue against). You will need to call over if the dump APIs on the client.

[souravgupta6nov1992]

@gregw yes this is regarding jetty client.

PFB the dump we were able to collect:

+= HttpDestination[Origin@c26fdaf2[http://app1.ns-2.domain.clustername:8080,tag=null,protocol=Protocol@301a9e[proto=[h2c],nego=false]]]@d0c1763,state=STARTED,queue=4,pool=MultiplexConnectionPool@7ee82d8d[s=STARTED,c=0/4/4,a=0,i=0,q=4,p=@6597ae30[inUse=0,size=4,max=4,closed=false]],stale=false,idle=99 - STARTED
|  += MultiplexConnectionPool@7ee82d8d[s=STARTED,c=0/4/4,a=0,i=0,q=4,p=@6597ae30[inUse=0,size=4,max=4,closed=false]] - STARTED
|  |  +- @6597ae30[inUse=0,size=4,max=4,closed=false]
|  |     +> entries size=4
|  |        +> MultiEntry@77ef8fab{CLOSED,usage=0,multiplex=93,pooled=HttpConnectionOverHTTP2@16c14292(closed=true)[HTTP2ClientSession@4ee6f06d{local:null,remote:null,sendWindow=0,recvWindow=16389496,state=[streams=-2475,CLOSED,goAwayRecv=null,goAwaySent=GoAwayFrame@33012ade{0/no_error/stop},failure=java.io.IOException: no_error/stop]}]}
|  |        +> MultiEntry@381869c2{CLOSED,usage=0,multiplex=3,pooled=HttpConnectionOverHTTP2@24517981(closed=true)[HTTP2ClientSession@5dcd80fe{local:null,remote:null,sendWindow=0,recvWindow=16775722,state=[streams=-2483,CLOSED,goAwayRecv=null,goAwaySent=GoAwayFrame@67a01643{0/no_error/stop},failure=java.io.IOException: no_error/stop]}]}
|  |        +> MultiEntry@1c8d250c{CLOSED,usage=0,multiplex=73,pooled=HttpConnectionOverHTTP2@33bfda6d(closed=true)[HTTP2ClientSession@2c46694{local:null,remote:null,sendWindow=0,recvWindow=16777216,state=[streams=-3494,CLOSED,goAwayRecv=null,goAwaySent=GoAwayFrame@65316a81{0/no_error/stop},failure=java.io.IOException: no_error/stop]}]}
|  |        +> MultiEntry@1061866e{CLOSED,usage=0,multiplex=51,pooled=HttpConnectionOverHTTP2@5dc04abd(closed=true)[HTTP2ClientSession@7c917757{local:null,remote:null,sendWindow=0,recvWindow=16777216,state=[streams=-1382,CLOSED,goAwayRecv=null,goAwaySent=GoAwayFrame@121a6f26{0/no_error/stop},failure=java.io.IOException: no_error/stop]}]}
|  +> exchanges size=4
|     +> HttpExchange@ed4c9fd{req=HttpRequest[POST /app1-app1path1/v1/app1path2 HTTP/1.1]@58a19aa2[PENDING/null] res=HttpResponse[null 0 null]@511eef38[PENDING/null]}
|     +> HttpExchange@26421183{req=HttpRequest[GET /app1-app1path1/v1/app1path3 HTTP/1.1]@4c9ae3e6[PENDING/null] res=HttpResponse[null 0 null]@67e03932[PENDING/null]}
|     +> HttpExchange@4f538f12{req=HttpRequest[POST /app1-app1path1/v1/app1path2 HTTP/1.1]@1d28d92b[PENDING/null] res=HttpResponse[null 0 null]@6912ceef[PENDING/null]}
|     +> HttpExchange@5f6a0447{req=HttpRequest[GET /app1-app1path1/v1/app1path3 HTTP/1.1]@49439008[PENDING/null] res=HttpResponse[null 0 null]@39259f31[PENDING/null]}
+> requestListeners size=1

[souravgupta6nov1992]

@gregw we are currently unable to update to version 12.0.0 since there is some conflict coming due to Micro meter .. is there anything that can be done on this ?

[sbordet]

@souravgupta6nov1992 your dump shows negative numbers for streams in each session (should be impossible), and shows that the client was stopped apparently midway.

Without further information, we cannot help you.
What are you doing?
Have you modified Jetty?

[olamy]

@souravgupta6nov1992 micrometer should support Jetty 12 now with 1.13.0-M2 (see micrometer-metrics/micrometer#4753)

[souravgupta6nov1992]

@sbordet We have not overriden or modified jetty. This is happening with vanilla jetty client

[sbordet]

Still too little information to help.
Do you have a reproducer that we can try?