[FEATURE] Update password settings to utilize secure variants

[chriswhite199]

Related to changes being proposed in security-plugin PR:

• Added secure settings for ssl related passwords: security#2296

The src/main/java/org/opensearch/commons/rest/SecureRestClientBuilder.java file assumes the HTTP Keystore password will be pulled from an insecure setting, as well as some Integration tests that set the insecure property.

git grep -i _PASSWORD
src/main/java/org/opensearch/commons/ConfigConstants.java:    public static final String OPENSEARCH_SECURITY_SSL_HTTP_KEYSTORE_PASSWORD = "plugins.security.ssl.http.keystore_password";
src/main/java/org/opensearch/commons/rest/SecureRestClientBuilder.java:        String passwd = settings.get(ConfigConstants.OPENSEARCH_SECURITY_SSL_HTTP_KEYSTORE_PASSWORD, null);
src/test/java/org/opensearch/commons/rest/IntegrationTests.java:import static org.opensearch.commons.ConfigConstants.OPENSEARCH_SECURITY_SSL_HTTP_KEYSTORE_PASSWORD;
src/test/java/org/opensearch/commons/rest/IntegrationTests.java:            .put(OPENSEARCH_SECURITY_SSL_HTTP_KEYSTORE_PASSWORD, "changeit")
src/test/java/org/opensearch/commons/rest/IntegrationTests.java:            .put(OPENSEARCH_SECURITY_SSL_HTTP_KEYSTORE_PASSWORD, "changeit")

This ticket is intended for discussion on how to best proceed in common-utils, esp updating SecureRestClientBuilder.java if the previously mentioned PR were to be merged.

Given a plugin that utilizes SecureRestClientBuilder, running on a cluster that uses the 'secure' variant of the plugins.security.ssl.http.keystore_password_secure setting, the password would not be correctly extracted from the settings

[lezzago]

@chriswhite199, it seems like for the SecureRestClientBuilder, we will need to update it such that the passwords are retrieved correctly.

I would suggest making a PR against common-utils main branch for the change to work with the new PR from security. Also is that security PR going to be backported to 2.x? If so, we will need to do the same for common-utils.

[chriswhite199]

@lezzago I can do that but there is a greater conversation around how we handle existing clusters and the use of insecure settings.

Originally the PR over in security marked all the _password settings as insecure, which would mean any existing configurations upgrading to this new plugin version would fail to start if the allow_insecure_settings wasn't set to true (it defaults to false).

It was discussed that maybe we should log.WARN insecure usages rather than throw an exception, but there hasn't been much further discussion about that approach.

The security PR has it's own InsecureStringSetting implementation to do the above, so that code would need to be copied here if that was the approach decided on.

• https://github.com/opensearch-project/security/pull/2296/files#diff-767206627d492f9a032bb35b0385855bc4638d9b40667a0cecd46de4505ac896R106

I'm happy to make whatever changes are suggested, but need some guidance on how to approach the issue of handling existing cluster setups that will be using 'insecure' passwords.

One idea i'd like to float is a change to the opensearch base project, to update InsecureStringSetting to allow a flag to be passed at construction to log warn usage rather than throwing the exception. That way legacy settings can be log warned as deprecated for a release cycle and then eventually reverted back to throwing the exception when used in a later release cycle.

Thoughts?