[Encrypted Saved Objects] Adds support for migrations in ESO

[gmmorris]

Summary

This PR provides the ground work needed to address #68994 and #67157.

The two main changes here are:

1. The addition of a createMigration api on the EncryptedSavedObjectsPluginSetup.
2. A change in SavedObjects migration to ensure they don't block the event loop.

Important Note:
This change introduces a potential breaking step in the migration step.
As we now try to migrated these Encrypted Saved Objects, a failure to decrypt these objects will result in Kibana failing to start up. In fact, this is the expected behaviour when Kibana is using an ephemeral encryption key which is automatically regenerated at startup.

This has been discussed with @legrego and @rudolf and we came to the conclusion that this is reasonable as:

1. This will only happen with ESO that define migrations and can be addressed by demanding a non-ephemeral key when these plugins are used (as has been done with Alerting, for example).
2. When this happens the users can roll back the migration and address this by deleting the encrypted objects (which are lost due to the ephemeral key anyway)
3. Security are working on a solution for rotating encryption keys which will reduce the danger of such a state.

Dev Docs

We now have an createMigration api on the EncryptedSavedObjectsPluginSetup which facilitates defining a migration for an EncryptedSavedObject type.

Defining migrations

EncryptedSavedObjects rely on standard SavedObject migrations, but due to the additional complexity introduced by the need to decrypt and reencrypt the migrated document, there are some caveats to how we support this.
The good news is, most of this complexity is abstracted away by the plugin and all you need to do is leverage our api.

The EncryptedSavedObjects Plugin SetupContract exposes an createMigration api which facilitates defining a migration for your EncryptedSavedObject type.

The createMigration function takes four arguments:

Argument	Description	Type
isMigrationNeededPredicate	A predicate which is called for each document, prior to being decrypted, which confirms whether a document requires migration or not. This predicate is important as the decryption step is costly and we would rather not decrypt and re-encrypt a document if we can avoid it.	function
migration	A migration function which will migrate each decrypted document from the old shape to the new one.	function
inputType	Optional. An EncryptedSavedObjectTypeRegistration which describes the ESOType of the input (the document prior to migration). If this type isn't provided, we'll assume the input doc follows the registered type.	object
migratedType	Optional. An EncryptedSavedObjectTypeRegistration which describes the ESOType of the output (the document after migration). If this type isn't provided, we'll assume the migrated doc follows the registered type.	object

Example: Migrating a Value

encryptedSavedObjects.registerType({
  type: 'alert',
  attributesToEncrypt: new Set(['apiKey']),
  attributesToExcludeFromAAD: new Set(['mutedInstanceIds', 'updatedBy']),
});

const migration790 = encryptedSavedObjects.createMigration<RawAlert, RawAlert>(
  function shouldBeMigrated(doc): doc is SavedObjectUnsanitizedDoc<RawAlert> {
    return doc.consumer === 'alerting' || doc.consumer === undefined;
  },
  (doc: SavedObjectUnsanitizedDoc<RawAlert>): SavedObjectUnsanitizedDoc<RawAlert> => {
    const {
      attributes: { consumer },
    } = doc;
    return {
      ...doc,
      attributes: {
        ...doc.attributes,
        consumer: consumer === 'alerting' || !consumer ? 'alerts' : consumer,
      },
    };
  }
);

In the above example you can see thwe following:

1. In shouldBeMigrated we limit the migrated alerts to those whose consumer field equals alerting or is undefined.
2. In the migration function we then migrate the value of consumer to the value we want (alerts or unknown, depending on the current value). In this function we can assume that only documents with a consumer of alerting or undefined will be passed in, but it's still safest not to, and so we use the current consumer as the default when needed.
3. Note that we haven't passed in any type definitions. This is because we can rely on the registered type, as the migration is changing a value and not the shape of the object.

As we said above, an EncryptedSavedObject migration is a normal SavedObjects migration, and so we can plug it into the underlying SavedObject just like any other kind of migration:

savedObjects.registerType({
    name: 'alert',
    hidden: true,
    namespaceType: 'single',
    migrations: {
        // apply this migration in 7.9.0
       '7.9.0': migration790,
    },
    mappings: { 
        //...
    },
});

Example: Migating a Type

If your migration needs to change the type by, for example, removing an encrypted field, you will have to specify the legacy type for the input.

encryptedSavedObjects.registerType({
  type: 'alert',
  attributesToEncrypt: new Set(['apiKey']),
  attributesToExcludeFromAAD: new Set(['mutedInstanceIds', 'updatedBy']),
});

const migration790 = encryptedSavedObjects.createMigration<RawAlert, RawAlert>(
  function shouldBeMigrated(doc): doc is SavedObjectUnsanitizedDoc<RawAlert> {
    return doc.consumer === 'alerting' || doc.consumer === undefined;
  },
  (doc: SavedObjectUnsanitizedDoc<RawAlert>): SavedObjectUnsanitizedDoc<RawAlert> => {
    const {
      attributes: { legacyEncryptedField, ...attributes },
    } = doc;
    return {
      ...doc,
      attributes: {
        ...attributes
      },
    };
  },
  {
    type: 'alert',
    attributesToEncrypt: new Set(['apiKey', 'legacyEncryptedField']),
    attributesToExcludeFromAAD: new Set(['mutedInstanceIds', 'updatedBy']),
  }
);

As you can see in this example how we provide a legacy type which describes the input which needs to be decrypted.
The migration function will default to using the registered type to encrypt the migrated document after the migration is applied.

If you need to migrate between two legacy types, you can specify both types at once:

encryptedSavedObjects.registerType({
  type: 'alert',
  attributesToEncrypt: new Set(['apiKey']),
  attributesToExcludeFromAAD: new Set(['mutedInstanceIds', 'updatedBy']),
});

const migration780 = encryptedSavedObjects.createMigration<RawAlert, RawAlert>(
  function shouldBeMigrated(doc): doc is SavedObjectUnsanitizedDoc<RawAlert> {
    // ...
  },
  (doc: SavedObjectUnsanitizedDoc<RawAlert>): SavedObjectUnsanitizedDoc<RawAlert> => {
    // ...
  },
  // legacy input type
  {
    type: 'alert',
    attributesToEncrypt: new Set(['apiKey', 'legacyEncryptedField']),
    attributesToExcludeFromAAD: new Set(['mutedInstanceIds', 'updatedBy']),
  },
  // legacy migration type
  {
    type: 'alert',
    attributesToEncrypt: new Set(['apiKey', 'legacyEncryptedField']),
    attributesToExcludeFromAAD: new Set(['mutedInstanceIds', 'updatedBy', 'legacyEncryptedField']),
  }
);

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
• This was checked for cross-browser compatibility, including a check against IE11

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

@elasticmachine merge upstream

[rudolf]

Left a few nits but otherwise Platform changes look good to me

[pmuellr]

reviewing for alerting - no alerting changes here (perhaps in a previous commit)

LGTM - shape looks like it will work for migrating our alerting ESOs

[azasypkin]

ACK: Will start reviewing tomorrow morning.

[azasypkin]

Looks good! I have a bunch of minor nits and one concern. I proposed one option, but happy to discuss any other options as well. Let me know what you think.

[azasypkin]

question: do we need all these objects here? Can we just have config, space and saved-object-with-migration or even only saved-object-with-migration?

[gmmorris]

I don't know 😬
I just used what was dumped by esarchiver, I'll take a look 🤷

[azasypkin]

Did you manage to figure that out? Not super important, mostly curiosity and desire to keep this file as free of unnecessary stuff as possible. I actually have a feeling that we just need one ESO object and the rest will be created automatically if needed.

[gmmorris]

Oh, oops, I misread your correction 🤦

[azasypkin]

Looks great, thanks for making these adjustments! I finished code review, only skipped tests for create_migration for now since, as we discussed over Slack, we want to first figure out migration behavior in case of changed encryption key (currently it seems we block/crash Kibana).

[azasypkin]

Did you manage to figure that out? Not super important, mostly curiosity and desire to keep this file as free of unnecessary stuff as possible. I actually have a feeling that we just need one ESO object and the rest will be created automatically if needed.

[azasypkin]

Soo, same suggestion here, type is redundantly duplicated ...

Suggested change

	doc: SavedObjectUnsanitizedDoc<InputAttributes> | SavedObjectUnsanitizedDoc<InputAttributes>,
	doc: SavedObjectUnsanitizedDoc<InputAttributes>,

[azasypkin]

LGTM, thanks!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 51eea24

Build metrics

✅ unchanged

History

• 💚 Build #56267 succeeded 6413d35
• 💚 Build #56250 succeeded e60e108
• 💚 Build #56023 succeeded 16e81b9
• 💔 Build #55892 failed ca13aee
• 💚 Build #55497 succeeded 998d0d4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream