Allow skipping hot reload dn validation

[parislarkins]

Description

This change adds two new boolean settings for the Security plugin:

• plugins.security.ssl.http.enforce_cert_reload_dn_verification
• plugins.security.ssl.transport.enforce_cert_reload_dn_verification

Both options will default to True, which will maintain the current behaviour where performing a hot reload requires the Distinguished Names (IssuerDN, SubjectDN, and SAN) in the new certificate to match the current certificate.

When set to False, the Distinguished Names validation will be skipped when hot reloading the respective certificate (HTTP or Transport).

The current behaviour is preventing us from hot-reloading new Let's Encrypt certificates that were signed by a different intermediate CA than the original certificate, meaning we need to perform a rolling restart of each of our clusters in order to rotate the certificates. According to Let's Encrypt, one of their root CAs is expected to expire as soon as 2030 (https://letsencrypt.org/certificates/), so an approach that allowed different intermediate CAs but still rejected changed root CAs will still require us to perform a rolling restart for our clusters when the root CAs are inevitably rotated.

We do not believe adding the ability to disable this validation for hot-reloads creates any added security risk, as long as it is properly considered considering organisational context. There is no similar validation performed when changing certificates by restarting OpenSearch, and an actor that could trigger a hot-reload (requiring them to modify the certificate files on the OpenSearch node) would also likely be able to bypass the validation by restarting OpenSearch.

For example, in our operational context only the super admin user can trigger a hot-reload. The certificate for the super admin user is stored only on the OpenSearch nodes, so any malicious actor who had the super admin certificate and ability to modify the OpenSearch certificate files would also have the ability to restart OpenSearch. This makes the validation performed when triggering a hot-reload irrelevant, as any malicious actor could simply restart OpenSearch instead.

• Category: New feature

Issues Resolved

• Resolves [BUG] Exception when hot-reloading Let's Encrypt certs issued by alternate intermediate CAs #4509, although this approach covers a slightly larger scope by allowing skipping the validation entirely, and may not be appropriate for all deployments.

Testing

Unit tests have been added that cover reloading certificates to a certificate signed by a different Certificate Authority (different root and signing CA), and validating that this is rejected when the new settings are set to true, and accepted when the settings are set to false.

Check List

• New functionality includes testing
• New functionality has been documented - draft: Update hot reload section to show how DN validation can be disabled parislarkins/documentation-website#1
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.97%. Comparing base (4f2e689) to head (d74905f).
Report is 55 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4752      +/-   ##
==========================================
+ Coverage   65.22%   70.97%   +5.75%     
==========================================
  Files         318      310       -8     
  Lines       22327    20946    -1381     
  Branches     3591     3326     -265     
==========================================
+ Hits        14562    14867     +305     
+ Misses       5968     4329    -1639     
+ Partials     1797     1750      -47     

Files with missing lines	Coverage Δ
...ensearch/security/ssl/DefaultSecurityKeyStore.java	68.57% <100.00%> (+1.90%)	⬆️
...arch/security/ssl/OpenSearchSecuritySSLPlugin.java	86.53% <100.00%> (+0.22%)	⬆️
...ensearch/security/ssl/util/SSLConfigConstants.java	77.77% <ø> (ø)

... and 112 files with indirect coverage changes

[shikharj05]

Thanks for the changes

[cwperks]

Thank you @parislarkins , this change looks reasonable to me.

What happens if you disable dn validation and change the dn of a node, but the dn is not in the nodes_dn list?

[parislarkins]

Thank you @parislarkins , this change looks reasonable to me.

What happens if you disable dn validation and change the dn of a node, but the dn is not in the nodes_dn list?

@cwperks I have not tested this specifically, but I expect the node would no longer be able to connect to the remote cluster. As I understand it, this is the same behaviour that would happen if the cert was changed by restarting OpenSearch as well.

[derek-ho]

Thanks