Ephemeral encrypted saved-object encryption key

[kobelb]

Problem

When there's an ephemeral encryption key for the encrypted saved-object attributes plugin, as soon as the Kibana process restarts the encryption key is lost and anything that was encrypted with it can't be decrypted. Alerting is currently the primary consumer of encrypted saved-object attributes, as such I've focused on the impact this has on Alerting for a large portion of the problem statement. We have other ephemeral encryption keys in Kibana, but their impact is a lot less severe given their usage patterns.

If a user was to create an Alert with an ephemeral encryption key, and then the Kibana process restarts, the Alert would just stop working. To mitigate a user creating a bunch of Alerts which won't work as soon as the Kibana process restarts, Alerting blocks the user from creating Alerts until an encryption key is set in the kibana.yml.

Regardless of the encryption key being ephemeral or set in the kibana.yml, it should be the same value for all Kibana instances. Otherwise, in a high-availability deployment of Kibana with multiple instances, if an Alert is created by an instance of Kibana that has a different encryption key than the instance that tries to run it, the alert will fail:

Potential Solutions

Please feel free to recommend others besides the following, they're almost certainly not exhaustive.

When there is an ephemeral encryption key, warn the user and let them store the secrets unencrypted in plain-text

This works "out of the box" for high availability deployments of Kibana. The secrets are stored in plain-text in the .kibana index, and as such users would be able to read them. When the user later sets a proper encryption key, we'd have to encrypt all of the existing plain-text secrets.

Use a default encryption key and warn the user when they're using the default

This also works "out of the box" for high availability deployments of Kibana. The secrets aren't stored in plain-text in the .kibana index, but since they're using a well-known encryption key it's trivial for an attacker to decrypt them. This is honestly more obfuscation than real security. When the user later sets a proper encryption key, we'd have to decrypt all of the existing encrypted secrets using the default encryption key and use the new encryption key.

Generate a random encryption key on the first start-up and persist it to the kibana.yml or the keystore

This doesn't work "out of the box" with high availability deployments. It also is potentially problematic for situations like Docker where there isn't a persistent filesystem. It's secure because no users with access to the .kibana index are able to decrypt the values. At the moment, our linux packages (.deb/.rpm) allow the Kibana process to write to either the kibana.yml or the keystore, but we haven't utilized this approach elsewhere and ideally we could restrict write access to both of these to prevent an attacker from overwriting configuration values.

Create a CLI tool to walk users through configuration Kibana's various encryption keys

Users at the moment have to manually know the 3 settings which must be set to properly configure encryption for security, reporting and encrypted saved-object attributes. If we offered the user a CLI tool similar to Elasticsearch's cert-util we could interactively walk them through setting up Kibana, including the configuration of the encryption keys.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[kobelb]

/cc @alexfrancoeur @arisonl

[arisonl]

So the first two options potentially expose secrets like credentials and API keys, in which case I think they are unsuitable. The second one potentially increases the attack surface and may create problems with Docker. The fourth one is not ideal from a UX perspective but it might be acceptable? What do you think @kobelb @alexfrancoeur @mikecote? With this approach, can we generate a random key, as an option, so that user essentially only needs to do a copy-paste to kibana.yml? Can we streamline the interaction with the keystore at all?

[kobelb]

The fourth one is not ideal from a UX perspective but it might be acceptable?

Unfortunately, it's the best option I've come up with.

With this approach, can we generate a random key, as an option, so that user essentially only needs to do a copy-paste to kibana.yml?

Yup!

Can we streamline the interaction with the keystore at all?

We'll have to confirm whether or not there's some constraint preventing us from writing directly to the kibana.yml or the keystore. If either is possible, both should be. The elasticsearch-certutil and elasticsearch-certgen tools output all of the files to a .zip and don't directly augment the elasticsearch.yml/keystore so there might be some limitation or flaw to this approach that I'm just not aware of. Conceptually, it seems possible.

[mikecote]

The option of a cli tool sounds interesting when considering all the steps a user may have to do to get up and running with alerting / Kibana.

• Setting a value to xpack.encryptedSavedObjects.encryptionKey
• Setting up TLS between Kibana and Elasticsearch (when security enabled)
• Set a value to xpack.actions.whitelistedHosts (defaults to * for now to avoid another setup step)
• Possibly setting an email recipient (and server?) for out of the box alerts
• Possibly setting up a "network.publish_host" for URLs back to Kibana (Add "network.publish_host" style configuration in Kibana #45815)

Most of these can be handled automatically in Cloud but can definitely be a pain without such tool to setup on prem.

[kobelb]

I'm somewhat scared of creating a CLI tool that does everything... The Elasticsearch team has rather purpose built tools for the individual components, and otherwise just defers people to edit the elasticsearch.yml. The kibana.yml that we ship with the default distribution doesn't include commented out settings for xpack plugins, we can also augment that to include these common settings.

[legrego]

Use a default encryption key and warn the user when they're using the default

Elasticsearch used to ship with default passwords for the system accounts, but this was removed in 6.0 in favor of the password setup utility. This is similar in practice the concept of default passwords, so I'm hesitant to take this approach.

The CLI option is at least consistent with that ES provides today, but I don't love the UX.

Regardless of the encryption key being ephemeral or set in the kibana.yml, it should be the same value for all Kibana instances. Otherwise, in a high-availability deployment of Kibana with multiple instances, if an Alert is created by an instance of Kibana that has a different encryption key than the instance that tries to run it, the alert will fail

We have a number of settings now where we expect consistency between Kibana instances:

• If xpack.reporting.encryptionKey is not consistent (or ephemeral), then reporting will not work correctly in HA setups, or after restarts
• If xpack.security.encryptionKey is not consistent (or ephemeral), then security will no work correctly in HA setups, or after restarts
• If xpack.encryptedSavedObjects.encryptionKey is not consistent (or ephemeral), then alerting and other consumers of this features will not work correctly in HA setups, or after restarts

You could argue that a misconfigured xpack.encryptedSavedObjects.encryptionKey is more problematic than misconfiguring the other keys, but I feel like we should be trying to solve this problem in a way that satisfies all of our "synced" configuration settings.

This is a larger lift (and not fully thought out), but we could (re)explore making Kibana instances aware of each other (not "clustering" per se), so that they could coordinate these settings, or assert that they're consistent. Single node setups might be able to leverage the keystore to generate a random but consistent key on setup (similar to what ES does for the elastic user password before the system account passwords are setup by the administrator)

[kobelb]

This is a larger lift (and not fully thought out), but we could (re)explore making Kibana instances aware of each other (not "clustering" per se), so that they could coordinate these settings, or assert that they're consistent.

Currently, the only way that Kibana instances have to communicate with each other is via Elasticsearch. The xpack.reporting.encryptionKey and xpack.encryptedSavedObjects.encryptionKey are used to encrypt secrets which are stored in Elasticsearch, so if an attacker compromises one or many Elasticsearch indices they aren't able to decrypt the values in the indices. Therefore, we can't store the encryption keys themselves within any Elasticsearch index because that defeats their purpose.

We could store a hash of the encryption keys within Elasticsearch to allow nodes to detect whether or not there are other nodes with mismatched encryption keys. There are some complexities here to make sure that old instances of Kibana, which we may never see again, didn't register an invalid encryption key which always triggers the warning. While this would be a nice to have feature, I see it improving the user experience as an addition to providing a CLI to automatically generate a random encryption key. Perhaps I'm missing something and the addition of this functionality makes one of the other options viable.

Single node setups might be able to leverage the keystore to generate a random but consistent key on setup (similar to what ES does for the elastic user password before the system account passwords are setup by the administrator)

This could work for single node setups which have persistent storage. However, I'm not seeing how this would work for situations like Docker where there is no persistent storage.

[legrego]

Currently, the only way that Kibana instances have to communicate with each other is via Elasticsearch

This was the part I was proposing we change; I was originally thinking something along the lines of #36907, but I foolishly forgot that this required setting up TLS certs for each node, which is very likely a non-starter for what we're trying to do here. This would make it even harder to get started, so it's not something I'd want to pursue in its current form.

This could work for single node setups which have persistent storage. However, I'm not seeing how this would work for situations like Docker where there is no persistent storage.

Yeah this wouldn't work if your runtime doesn't have persistent storage, or if you aren't providing an equivalent value via environment variable.

-sorry for the noise on this idea-

---

Taking a completely different approach -- if/when Kibana has a true System Indices managed by an ES Kibana plugin, could we have that plugin configure these secrets automatically? This would reduce the burden for initial setup, and eliminate the coordination of these settings. This plugin could expose the ability to manually set these secrets if the randomly generated keys aren't suitable, or were somehow compromised. There are a lot of details to work out here, but if this sounds viable on the surface, then it's something we can explore further.

One complication that immediately comes to mind is where/how to store these secrets in ES. We might just be moving the problem, instead of solving it.

[kobelb]

Taking a completely different approach -- if/when Kibana has a true System Indices managed by an ES Kibana plugin, could we have that plugin configure these secrets automatically?

I'm not seeing how this could work without weakening security. If the ES Kibana plugin generates the encryption keys, it'd have to allow Kibana to retrieve the encryption keys. This means that an attacker would only have to compromise Elasticsearch to be able to decrypt all of the secrets. With the way this is currently implemented, an attacker has to compromise both Elasticsearch and Kibana to be able to decrypt all of the secrets.

[legrego]

At this point, it sounds like a CLI tool would be the most tolerable option. If this is the approach we take, then what should the initial scope of this tool be? Should we constrain it to only setting encryption keys?

I see overlap between this tool and other initiatives we have yet-to-document, such as a "setup mode" for Kibana which would give administrators a way to complete their setup visually. There are a lot of open questions here, so I don't want to couple these two projects. I only bring it up to raise awareness, in that we don't have to immediately solve all setup problems with the proposed CLI

[kobelb]

@legrego those are all good questions, which I have as well. I'm leaning toward making the CLI tool only create encryption keys, and seeing how the other initiatives play out. Thoughts?

[legrego]

I'm leaning toward making the CLI tool only create encryption keys, and seeing how the other initiatives play out. Thoughts?

Yep that makes sense to me too

[tylersmalley]

Spoke with @azasypkin regarding the RFC for key rotation #72828

For key rotation, the primary key (xpack.encryptedSavedObjects.encryptionKey) will be moved to the list of decryption-only keys (xpack.encryptedSavedObjects.keyRotation.decryptionOnlyKeys). One problematic aspect of this is there are multiple ways the encryption key could be defined (CLI option, kibana.yml, and Keystore). Docker uses environment variables, but those are converted to CLI options.

There appears to still be an open question as to where or if we persist these keys. Some options include the Keystore, kibana.yml, STDOUT, or a separate file (elasticsearch-certutil outputs in a zip).

@elastic/kibana-security do you have a preference here? Given the different ways Kibana could be configured, I wonder if it would be best to generate the keys and output information on how they could be persisted to either kibana.yml or the Keystore.

[legrego]

Given the different ways Kibana could be configured, I wonder if it would be best to generate the keys and output information on how they could be persisted to either kibana.yml or the Keystore.

I don't have a particularly strong opinion here, but my gut reaction is to persist to kibana.yml after prompting:

> Update kibana.yml with generated encryption keys? [Y/n]:

If they opt not to persist to kibana.yml, then we can just dump the keys to STDOUT with instructions on the different ways to configure Kibana (or a link to the instructions).

I say this because I view this CLI as a way to improve the "getting started" experience, and kibana.yml is one of the simpler ways to get Kibana up and running.

The keystore would be a more secure option though, so if we can make that straightforward enough for new users, then maybe that's a better option.

[tylersmalley]

• Write by default to YML. There are a few issues that need to be worked out with the keystore, mainly how would you read the decryption key after the fact when wanting to rotate.
• Raise issues regarding read-only config

A reminder that the goal of this is the improve the getting started experience.

[jbudz]

Pending a standalone config service we'll need to find a way to to merge the keystore and kibana.yml files. Currently the keystore takes precedence via overwrite, so we'll need to way to match that behavior.

[jbudz]

We just met briefly to discuss next steps. The current plan for v1 is to

• output a yml configuration to stdout
• cmd line option to prompt writes for first installs
• no decryption key rotation
• runtime check warnings point to the CLI