Support Jetty's live cert reload on HTTPS frontend

[mlsorensen]

Description

This PR implements Jetty's hot certificate reload per jetty/jetty.project#5042

When the keystore changes, the API server begins using the new certificate. Note that this functionality doesn't support live change of keystore password, only certificate.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested locally with self-signed certs. Changed the cert in the keystore while management server was running, reloaded the website, and inspected the https certificate.

Tested against cmk, which continued to function after the cert change. Also tested against UI - website needed to be refreshed due to browser security. Perhaps a future enhancement would catch these errors and auto-refresh?

Here you can see the local cloudstack management server using a new cert after it is loaded into a keystore:

[root@kvmlab1 management]# echo | openssl s_client -showcerts -connect localhost:8443 2>/dev/null | openssl x509 -inform pem -noout -text | grep "Not Before"
            Not Before: Mar 21 21:09:50 2023 GMT

change cert

[root@kvmlab1 management]# openssl x509 -req -in cloud.csr -signkey cloud.key > cloud.crt
Signature ok
subject=C = US, ST = UT, L = Alpine, O = Marcus, OU = Marcus, CN = kvmlab1, emailAddress = example@example.com
Getting Private key

[root@kvmlab1 management]# openssl pkcs12 -export -in cloud.crt -inkey cloud.key -name cloud -passout pass:****** > cloud-localhost.pk12

[root@kvmlab1 management]# keytool -importkeystore -srckeystore cloud-localhost.pk12 -srcstoretype PKCS12 -destkeystore cloud.jks  -deststoretype JKS -srcstorepass ****** -deststorepass ******  -destkeypass ****** -noprompt
Importing keystore cloud-localhost.pk12 to cloud.jks...
Warning: Overwriting existing alias cloud in destination keystore
Entry for alias cloud successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

check cert

[root@kvmlab1 management]# echo | openssl s_client -showcerts -connect localhost:8443 2>/dev/null | openssl x509 -inform pem -noout -text | grep "Not Before"
            Not Before: Mar 21 21:24:47 2023 GMT

[codecov]

Codecov Report

Merging #7355 (cc00c6a) into main (2aa3f98) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #7355   +/-   ##
=========================================
  Coverage     12.68%   12.68%           
  Complexity     8656     8656           
=========================================
  Files          2718     2718           
  Lines        256169   256169           
  Branches      39932    39932           
=========================================
  Hits          32504    32504           
  Misses       219531   219531           
  Partials       4134     4134           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[weizhouapache]

code lgtm

[rohityadavcloud]

Code LGTM - but haven't tested it.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[mlsorensen]

Merging based on 2x LGTM and manual testing of feature.

[rohityadavcloud]

Sorry @mlsorensen per community guidelines, there were no @blueorangutan smoketests were performed and generally, it's not preferred that PR author serves manual tests.

However, it's possible there's no regression as smoketests in Github Actions against simulator have passed. I'll kick a round of BO smoketests on #7344