[FEATURE] Improve error message when a node with an incorrectly configured certificate attempts to connect

[ArranDengate-Netapp]

Problem

When setting up a cluster, if a node tries to join with a certificate that is not configured correctly in plugins.security.nodes_dn, OpenSearch provides an error message - "Transport client authentication no longer supported".

[2024-06-18T13:29:06,860][ERROR][o.o.s.t.SecurityRequestHandler] [opensearch-cluster-master-2] OpenSearchException[Transport client authentication no longer supported.]
[2024-06-18T13:29:07,387][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-2] handshake failed for [connectToRemoteMasterNode[<ip-redacted>:9300<http://<ip-redacted>:9300>]]
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-1][<ip-redacted>:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:80) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:294) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:155) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:841) ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:271) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:139) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:119) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:108) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:100) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:784) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.forwardFragments(NativeInboundBytesHandler.java:157) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.doHandleBytes(NativeInboundBytesHandler.java:94) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:143) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:119) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) [?:?]

This is a misleading error - under this circumstance, the problem has nothing to do with the transport client.

Context of the error message

My understanding of this error message is as follows.

OpenSearch clusters have two different ports:

• The REST API port, usually port 9200
• The internode communication port (aka 'transport' port), usually port 9300

In the early days of Elasticsearch, there was a popular application client, the transport client, that connected on the internode port. This old method of connecting has been deprecated since ES 7.0 in 2019. It was removed in OpenSearch 2.0 (#1701). Modern application clients connect on the REST API port.

From the perspective of OpenSearch, an incoming connection on port 9300 could be coming from a node, or it could be coming from a transport client.

There is some discussion of this in issue #4401 and the linked forum posts https://forum.opensearch.org/t/transport-client-authentication-no-longer-supported-error-when-deploying-cluster-with-security-plugin-enabled/9814/5 / https://forum.opensearch.org/t/transport-client-authentication-no-longer-supported-error-while-implementing-third-party-ca-cert-for-transport-layer/19694 .

Proposed solution

I would like to make this error message clearer.

If we could distinguish between a node joining and a transport client connecting, and provide different error messages for each, that would be ideal.

If it is not easy to tell the difference from the server side, we could still add more information to the error message - something like:

"A connection was attempted on this cluster's internode communication port, but it was refused. If you are attempting to join a node to this cluster, please confirm the Distinguished Name of the new node's certificate is configured in this node's plugins.security.nodes_dn setting. If you are attempting to connect to this cluster with a transport client, please note that transport clients are no longer supported."

It would be nice to also mention the DN that was rejected - for example:

"The distinguished name of the node certificate provided was: 'CN=node.other.com,OU=SSL,O=Test,L=Test,C=DE'"

This would help the user work out what needs to be configured in plugins.security.nodes_dn .

[stephen-crawford]

[Triage] Hi @ArranDengate-Netapp, thanks for filing this issue. This sounds like a good idea. Since you have already put time into investigating this issue perhaps you would like to open a pull request with the proposed change? Thank you.

[ArranDengate-Netapp]

@stephen-crawford sorry, I don't have capacity at the moment! However, my colleague may create a PR for this later.

[akolarkunnu]

@stephen-crawford Can you please assign this bug to me. I will work on this