wso2mi 4.2.0 : Continous Socket connection errors in logs when pod count >1

[Sukeerth148]

[I am encountering an issue within my WSO2 Micro Integrator (WSO2 MI) 4.2.0 pods deployed on OpenShift. Specifically, this problem arises when there are multiple pods present. The error occurs when I configure the route.openshift.io/termination property in the Ingress resource to reencrypt. However, I've observed that this issue does not manifest when the property is set to edge.]

Version

Client Version: 4.12.0-202304070941.p0.g31aa3e8.assembly.stream-31aa3e8
Kustomize Version: v4.5.7
Server Version: 4.14.0
Kubernetes Version: v1.27.6+f67aeb3

Steps To Reproduce

1. Deploy an application pod with ingress.
2. Keep the ingress termination as reencrypt.

Current Result

[2024-`03-07 09:23:19,362] ERROR {SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:840)

Expected Result

No error with in the logs.

Additional Information

Debug logs:

[2024-03-06 18:12:20,444] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-6
[2024-03-06 18:12:20,444] DEBUG {LoggingNHttpServerConnection} - http-incoming-6: Shutdown connection
[2024-03-06 18:12:20,444] DEBUG {SSLIOSession} - I/O session http-incoming-6-4 10.255.6.92:8253<->10.255.14.1:59748[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Shutdown
[2024-03-06 18:12:20,444] DEBUG {SourceHandler} - http-incoming-6: Keep-Alive connection was closed: Remote Address : /10.255.14.1:59748
[2024-03-06 18:12:20,444] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-6
[2024-03-06 18:12:20,444] DEBUG {LoggingNHttpServerConnection} - http-incoming-6: Shutdown connection
[2024-03-06 18:12:20,444] DEBUG {SSLIOSession} - I/O session http-incoming-6-4 10.255.6.92:8253<->10.255.14.1:59748[CLOSED][][CLOSED][r][NOT_HANDSHAKING][0][0][0]: Shutdown
[2024-03-06 18:12:25,442] DEBUG {SSLIOSession} - I/O session http-incoming-7-5 10.255.6.92:8253<->10.255.18.1:47144[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 180000
[2024-03-06 18:12:25,442] DEBUG {SSLIOSession} - I/O session http-incoming-7-5 10.255.6.92:8253<->10.255.18.1:47144[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute CONNECTION_INFORMATION
[2024-03-06 18:12:25,446] DEBUG {SSLIOSession} - I/O session http-incoming-8-6 10.255.6.92:8253<->10.255.14.1:59758[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 180000
[2024-03-06 18:12:25,446] DEBUG {SSLIOSession} - I/O session http-incoming-8-6 10.255.6.92:8253<->10.255.14.1:59758[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute CONNECTION_INFORMATION
[2024-03-06 18:12:25,448] ERROR {SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:840)

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.