PKIX path building failed

[rvbabilonia]

Hi Hakan and the rest of the community,

I'm currently integrating our API to a third bank with mTLS enabled.

I have the following codes:

    /**
     * Returns the {@link X509ExtendedKeyManager}.
     *
     * @param clientSslPublicKeyPath  the path to the client SSL public key
     * @param clientSslPrivateKeyPath the path to the client SSL private key
     * @param clientSslPublicKey      the content of the client SSL public key
     * @param clientSslPrivateKey     the content of the client SSL private key
     * @return the {@link X509ExtendedKeyManager}
     */
    public static X509ExtendedKeyManager getX509ExtendedKeyManager(final String clientSslPublicKeyPath,
                                                                   final String clientSslPrivateKeyPath,
                                                                   final String clientSslPublicKey,
                                                                   final String clientSslPrivateKey) {
        if (StringUtils.isNotBlank(clientSslPublicKey) && StringUtils.isNotBlank(clientSslPrivateKey)) {
            return PemUtils.parseIdentityMaterial(clientSslPublicKey, clientSslPrivateKey, null);
        }

        return PemUtils.loadIdentityMaterial(Paths.get(clientSslPublicKeyPath), Paths.get(clientSslPrivateKeyPath));
    }

    /**
     * Returns the {@link X509ExtendedKeyManager}.
     *
     * @param certificateAuthorityBundlePath  the path to the certificate authority bundle
     * @param certificateAuthorityBundle      the content of the certificate authority bundle
     * @return the {@link X509ExtendedKeyManager}
     */
    public static X509ExtendedTrustManager getX509ExtendedTrustManager(final String certificateAuthorityBundlePath,
                                                                       final String certificateAuthorityBundle) {
        if (StringUtils.isNotBlank(certificateAuthorityBundle)) {
            return PemUtils.parseTrustMaterial(certificateAuthorityBundle);
        }

        return PemUtils.loadTrustMaterial(Paths.get(certificateAuthorityBundlePath));
    }

    /**
     * Returns a {@link ClientHttpConnector} utilising 2-way/mutual TLS.
     *
     * @param client                         the client
     * @param publicKeyPath                  the path to the public key file
     * @param publicKey                      the public key
     * @param privateKeyPath                 the path to the private key file
     * @param privateKey                     the private key
     * @param certificateAuthorityBundlePath the path to the certificate authority bundle file
     * @param certificateAuthorityBundle     the certificate authority bundle
     * @return the {@link ClientHttpConnector}
     * @throws SSLException thrown when an SSL context cannot be created
     */
    public static ClientHttpConnector getClientHttpConnector(HttpClient client, final String publicKeyPath,
                                                             final String publicKey, final String privateKeyPath,
                                                             final String privateKey,
                                                             final String certificateAuthorityBundlePath,
                                                             final String certificateAuthorityBundle)
            throws SSLException {
        SslContext sslContext = NettySslUtils
                .forClient(getSSLContext(publicKeyPath, publicKey, privateKeyPath, privateKey,
                        certificateAuthorityBundlePath, certificateAuthorityBundle))
                .build();

        client = client.secure(spec -> spec.sslContext(sslContext));

        return new ReactorClientHttpConnector(client);
    }

    private static SSLFactory getSSLContext(String publicKeyPath, String publicKey, String privateKeyPath,
                                            String privateKey, String certificateAuthorityBundlePath,
                                            String certificateAuthorityBundle) {
        X509ExtendedKeyManager keyManager = KeyManager.getX509ExtendedKeyManager(publicKeyPath, privateKeyPath,
                publicKey, privateKey);

        X509ExtendedTrustManager trustManager = KeyManager.getX509ExtendedTrustManager(certificateAuthorityBundlePath,
                certificateAuthorityBundle);

        return SSLFactory.builder()
                .withIdentityMaterial(keyManager)
                .withTrustMaterial(trustManager)
                .build();
    }

I am able to send the request to the server:

But I get this stack trace:

org.springframework.web.reactive.function.client.WebClientRequestException: General OpenSslEngine problem; nested exception is javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
...
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1907)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:834)
	at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:564)
	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1041)
	at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:927)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1409)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1247)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1287)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)
	at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe$1.run(AbstractKQueueChannel.java:376)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:293)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:234)
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:773)
	at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36)
	at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48)
	at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1496)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:94)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1471)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
	... 24 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 38 more

Looking at the server response through PKIXValidator, I receive the server's public certificate and Java trusts both the intermediate and root certificates.

But when it gets to SunCertPathBuilder, the server's public certificate is not there:

Please advise. BTW I'm using Corretto 17, Spring Boot 2.7.2 and SSLContext Kickstart (for PEM and Netty) 7.4.6. The first and second bank integrations worked flawlessly, thanks to your library. I am able to transact with this third bank using a Python script, though.

Regards,
Rey

[Hakky54]

Hi Rey!

I think your issue is caused by the client which is either not trusting the server because the trusted certificate is not present or the certificate has been expired. Looking at your own analysis it seems like the server certificate is not present within the SunCertPathBuilder. That part is out of the scope of this library as it does not interact with those objects. Maybe Netty Reactor is filtering out the server certificate, but I am not sure... If you have configured the sslfactory with all of the trusted certificates it should work or else it would throw a runtime exception during startup time to indicate it failed to load the certificate. So my assumption is that it is filtered out by internal netty ssl libraries. We can have a quick call if you are willing to, it is hard to tell what the exact cause is although you have provided detailed information

[Hakky54]

Well 403 means it reached the server. So I assume the ssl handshake passed. But we can validate if that is really the case. Can you restart/rerun your client with the following VM arguments: -Djavax.net.debug=SSL,keymanager,trustmanager,ssl:handshake

It will generate a-lot logs regarding the ssl handshake. Can you share it, but please validate if there is any sensitive data which needs to be cleared...

[rvbabilonia]

Lots of logs indeed. The pertinent ones are related to apiservices (client) and api-qa (server).
log.txt

[Hakky54]

There is indeed an issue. The first part of the logs are generated correctly when it grabs the certificates with the CertificateUtils, the second part of the logs is missing the client hello and server hello messages... So it seems like the server is blocking it instantly without allowing it to perform an initial ssl handshake, but I don't know the reason.

I can try to help you through a video call where we debug it together, what do you think?

[rvbabilonia]

That would be great! Our JWT signing public certificate has expired over the weekend and I've given the bank a valid one. Just waiting for them to update their side. BTW I'm in NZ (GMT+12).

[Hakky54]

Sure, ping me whenever everything is ready on your side. I am mostly available through Discord. My Discord id is Hakky54#8471 we can do a videa call from there or a different tool. I am on CET+2 So I will be available from 9 AM onwords from my side and that will be probably around 7 PM on your side.