Create new secure setting type that is stored (encrypted) in cluster state

[talevy]

Goal:
There are some features (one example) that could make use of a secure setting that is consistent across nodes.

Problem with existing secure settings:
Existing secure settings stored in the keystore have the problem that they are not enforced to be consistent across nodes. The key-store settings are primarily used for node start-up secure settings.

Features that would enable such a setting:
In order to have an arbitrary set of settings stored securely out in the open in the cluster-state, there must be a way to encrypt and decrypt these values. One solution would be to introduce a sort-of system key that will be stored in the secure keystore, and all these new cluster-state-secure-settings would use this key to encrypt and decrypt their values when needed. These settings probably won't change, and it may make sense to enforce that by disallowing updates to these settings.

rotated system key:
Although these secure settings may not need key-rotation, the master system key used to cypher these settings should have a way to rotate itself. This means having a hand-off mechanism where old key decrypts and new key encrypts and re-inserts into the cluster-state

I've left further implementation details out of this description, since those may be hashed out upon investigation

cc @elastic/es-distributed @elastic/es-security

[DaveCTurner]

These settings probably won't change, and it may make sense to enforce that by disallowing updates to these settings.

This kind of setting would seem like a useful place to store credentials to third-party services that aren't needed for node start-up (e.g. snapshot repositories, Watcher actions) which are themselves subject to rotation. I think it'd be useful if they were updatable.

[albertzaharovits]

The problem with secure settings not being replicated on nodes is a limitation of the secret store "backend" we currently use.

Storing hidden and encrypted data in the cluster state achieves the consistency (single source of truth) and fail-over (each node has a copy of the secret). However, the encryption key has to be conveyed in a different manner. Having data encrypted with the key next to it adds no value to only hiding the plain text value. We should keep the way password is currently broadcasted away from the cluster state. But then we need a rotate key API? And a new update settings API, because you can only update secret values if you know the password?

I also think we could use HashiCorp Vault as an alternative secret store "backend". This will provide a single consistent source of secret values. In this case, nodes will use access tokens, instead of decryption passwords, to reach the "vault" service. This external service handles key rotation, time limited credentials, secret generation (for example generating S3 access tokens that can be revoked when the snapshot/restore is completed). The access token can be propagated with the existing node broadcast API.

In essence having an external service makes reasoning easier. The external service can also be used to store settings required before cluster formation. Another, more conceptual advantage of an external service, is that we avoid storing encrypted state: besides fail-over there is no point in keeping encrypted state because it cannot be used. No one reads this state, until the password ingress. But then, as a node, I can make a request for the encrypted content, I don't have to keep it around and update it constantly in case I need it at some time. But I need to keep it in case of fail-over. It has a different lifecycle altogether compared to the unencrypted state.

I think each alternative improves the existing one. But the devil is in the details and they come with their own complications. Are they worth it? A simple ansible playbook can rotate and update settings reliably and it is easy to understand for the administrator.

[jasontedor]

Having data encrypted with the key next to it adds no value to only hiding the plain text value. We should keep the way password is currently broadcasted away from the cluster state.

I think there is a misunderstanding of the proposal here. The key would be stored in the keystore, not in the cluster state.

I also think we could use HashiCorp Vault as an alternative secret store "backend".

We should avoid this as this will complicate our stack for the vast majority of our users. We should seek a solution that does not rely on third-party infrastructure.

[danielkasen]

So I want to +1 this and explain our current issue. We use the Pagerduty integrations for over 200 services. This means that each node has to include 200+ values in the keystore and when your cluster is 400+ nodes this takes a long time to execute. And basically it doesn't scale well. Now if we could store these in an-encrypted fashion in the cluster settings or a subset of .security index this would drastically increase the speed of our rollout.

[gwbrown]

This issue has been addressed by the introduction of consistent secure settings in #40416. While that PR did not address all concerns raised in this discussion (such as the high number of keystore entries causing slow rollouts), the original goal stated in the issue has now been unblocked. As such, I'm going to close this issue.

If you have a problem which is not addressed by consistent secure settings, please open a new issue on that topic with details of the problem.