Automatically reload certificate and key PEM files

[tsaarni]

This change adds support to hot-reload TLS certificate and key without requiring server restart. It makes it possible to automatically reload PEM files, PKCS#12 and JKS keystores, when they are configured as file paths.

Closes #3780

Why support for hot-reloading should be implemented in Vert.x core:

Currently the core provides API for thr user to configure KeyStore and certificate & key file paths. The files are loaded into memory and path information is discarded. The SSL context is associated with this in-memory KeyStore. Even if the underlying files would change, the static in-memory KeyStore, built by Vert.x core, remains static. It seems unavoidable that the code paths dealing with the files, construction of KeyStores, associating KeyStores with KeyManagers and finally with SSLContexts is impacted.

Alternative to the above could be not to use the the API that Vert.x provides for certificate or keystore path configuration, but instead user can provide their own custom KeyManager. This implies that #3780 would not be implemented in Vert.x but by each user should implement it by themselves.

The approach taken by this PR:

The change implements new key stores by extending JSSE KeyStoreSpi:

• PemFileKeyStoreSpi loads and reloads PEM files, contructs single in-memory JSSE keystore from them and delegates operations to it.
• KeyStoreFileSpi loads and reloads keystore files (PKCS#12 & JKS) and delegates operations to JSSE keystore.

The files are reloaded when they are modified. Modification is observed by checking the file modification timestamp, but only when certificates and keys are used (when KeyStoreSPI methods are called) and the interval is capped by cacheTtl, currently fixed to 1 check per second at most.

Previously, handling of certificates was centered around crafting KeyStore(s) from values. The benefit was that handling of credentials configured "by-value" and "by-path" were all normalized into single "by-value" use case. To be able to reload, the path information must be kept around. The "reloading keystores" are constructed "by-path" directly in PemKeyCertOptions.java and KeyStoreOptionsBase.java.

Previously selection of server certificate according to TLS SNI servername was implemented by custom-built logic by splitting each certificate in individual KeyStore(s) and KeyManager(s) and picking one of those according to the SNI information from Netty, when handling new TLS connection. The NewSunX509 version of KeyManager (from JDK 8) has support for selecting from multiple certificates in keystore according to SNI and other criterias, such as key type. This change proposes to change to JSSE's built-in SNI support. The side effect of this is that SNI support cannot be disabled anymore setSni(false) server option will not be effective: correct server certificate (according to SNI) will be returned from keystore when it previously could return incorrect one.

For detailed information of the proposed implementation see this project and associated design document.

Signed-off-by: Tero Saarni tero.saarni@est.tech

Note: Change stacks on top of #4388 PR branch in order to pass the tests. JSSE SNI server cert selection support requires valid certificate dates.

TODOs

• Add new test cases for hot-reloading
• Remove unused code after moving from custom-built SNI support to JSSE SNI support

[pmlopes]

Why not use the JDK file watcher instead?

FileSystems.getDefault().newWatchService()

[tsaarni]

Thanks for reviewing!

I had another try before this PR with a WatchService and theads to run them master...Nordix:certificate-rotation.

My problem there is that I cannot find a reliable way to delete the watch filehandles and threads. The X509KeyManager / X509ExtendedKeyManager interface have no close method and therefore they are not closed explicitly. I ran out of inotify filehandles when running test suite, since the key managers and the watches they hold were not garbage collected. I'd appreciate any ideas here!

Now currently, I've been experimenting with another approach that avoids periodic polling:
When key or certificate is requested from key manager, return cached versions unless time-to-live period (of e.g. 1 second or something else) has passed. If cache TTL has passed, check file modify timestamp. If the files have been modified then reload.

[tsaarni]

Maybe another (secondary) reason to not use WatchService is that they are sensitive on how the files are modified. In that previous link with my experiments, I added an example in comments how Kubernetes does the atomic swap of the files https://github.com/Nordix/vert.x/blob/db145f3ba4c6735a80fa005df1f30bdffd611b1a/src/main/java/io/vertx/core/net/impl/FileWatcher.java#L48-L85. One size does not fit all, so some configurability might need to be raised up all the way to all applications (through the frameworks that wrap Vert.x).

[tsaarni]

I've updated the PR content and the description. I would greatly appreciate feedback, and comments wether you agree on the approach and would be interested in incorporating hot-reload functionality into Vert.x.

Note: Change stacks on top of #4388 PR branch in order to pass the tests. JSSE SNI server cert selection support requires valid certificate dates.

[vietj]

I will have a look

[tsaarni]

I will have a look

Friendly reminder, asking if you would have time for the review :)

On slightly related note, in case it would be useful for the review: I've created a separate project and a design doc showing the ideas from this PR as a standalone project.

[vietj]

ok, I would like to see if we can have it as something more decoupled from vertx core

[tsaarni]

ok, I would like to see if we can have it as something more decoupled from vertx core

Thanks! Curious to hear if you'd have ideas about this!

Since Vert.x is quite intensively manipulating KeyStores, it is already entangled with the code paths that would need to support hot-reload. Therefore it is bit difficult for me to figure out how decoupling could be achieved, That is, if #3780 is to be implemented in the first place. There is the "escape hatch" where application would not use Vert.X certificate loading support at all, but instead they would create and pass their own KeyManager, initialized with hot-reloading KeyStore.
I did not explore this path so far, as I thought it would be rather "heavy" approach to require applications to make the change - especially since some use Vert.x through additional layers like Quarkus.

[tsaarni]

I've updated the PR description according to the discussion above.

[vietj]

See #4568