Sharing saved-objects in multiple spaces: phase 2

[kobelb]

Phase 2 from #27004.

This is currently blocked on #59960 and #63358

Note to self: add UI docs for Spaces (sharing).

1. Overview

The “Sharing saved objects” feature (#27004) changes how saved objects are stored in Elasticsearch. Particularly, objects that exist in different spaces will no longer be able to have the same ID; each object of a given type must have a globally unique ID.

In Phase 2, we will add support for converting existing "single-namespace" saved object types into new "multi-namespace" types. This entails two core pieces of work:

• Migrations: Existing Kibana migration logic only applies consumer-defined "migration transforms" to each object when Kibana is upgraded. This will be augmented to apply platform-defined "conversion transforms" and "reference transforms" to change existing object fields (id, references), add new object fields (namespaces, originId), and create new Alias objects.
• Aliases: When we change object IDs, we don’t want to break URLs and integrations that may rely on thes. For this reason, we intend to create an Alias for each object that is converted, so it can still be accessed in a particular space using its old ID.

Note: the originId field is used during Copy and Transform operations, but it is not a substitute for Aliases.

2. Migrations

2.1. Usage

When registering an existing object type, consumers will need to change the namespaceType field and add a new convertToMultiNamespaceTypeVersion field. This new field will inform the Kibana platform that objects of this type need to be converted when Kibana is upgraded to the specified version.

(Example)

---

Before

In Kibana 7.10, the "doodad" object is a single-namespace type:

core.savedObjects.registerType({
  name: 'doodad',
  hidden: false,
  namespaceType: 'single',
  mappings: { ... },
  migrations: { ... },
});

After

In Kibana 8.0 and later, the "doodad" object is a multi-namespace type:

core.savedObjects.registerType({
  name: 'doodad',
  hidden: false,
  namespaceType: 'multiple',
  mappings: { ... },
  migrations: { ... },
  convertToMultiNamespaceTypeVersion: '8.0.0',
});

---

2.2. Technical Details

The DocumentMigrator currently applies consumer-defined "migration transforms" to each object with an outdated migrationVersion field at two times: 1. When Kibana is upgraded, and 2. When creating a new object. During upgrade only, we will apply two new types of transforms:

"Conversion transforms"

During the upgrade process, for each object that has an outdated migrationVersion field, if its type has a convertToMultiNamespaceTypeVersion defined, we apply a "conversion transform" to it.

The conversion process follows this algorithm:

1. Remove the namespace field and add the corresponding namespaces field.
2. If the object does not exist in the default namespace:
   1. Deterministically regenerate the id field (using uuidv5)
   2. Add the originId field, equivalent to the old ID
   3. Create a Legacy URL Alias for this object using the old ID and the new ID

"Reference transforms"

The SavedObject type will be updated to define a new coreMigrationVersion field. Whenever an object is created or Kibana is upgraded, this value is set to match the current Kibana version.

During the upgrade process, for each object that has an outdated coreMigrationVersion field, its references are enumerated, and we apply one or more "reference transforms" to it. These transforms are defined by other object types.

The reference transform is only applied to objects that do not exist in the default namespace. It deterministically regenerates the reference ID for the affected types.

Examples

Kibana 8.0 defines three object types:

{
  name: 'foo',
  namespaceType: 'multiple',
  convertToMultiNamespaceTypeVersion: '8.0.0',
},
{
  name: 'bar',
  namespaceType: 'multiple',
  migrations: {
    '8.0.0': (doc) => { ... },
  },
  convertToMultiNamespaceTypeVersion: '8.0.0',
},
{
  name: 'baz',
  namespaceType: 'single',
  migrations: {
    '7.9.0': (doc) => { ... },
    '7.10.0': (doc) => { ... },
    '8.0.0': (doc) => { ... },
  },
}

Scenario 1: "bar" object

---

Existing object:

{
  type: 'bar',
  id: '111',
  migrationVersion: {},
  references: [{ type: 'foo', id: '222' }],
  namespace: 'somespace',
}

Transforms: conversion-8.0.0-bar --> reference-8.0.0-foo --> migration-8.0.0-bar

Output object:

{
  type: 'bar',
  id: deterministicNewId('somespace:bar:111'),
  migrationVersion: { 'bar': '8.0.0' },
  references: [{ type: 'foo', id: deterministicNewId('somespace:foo:222') }],
  referencesMigrationVersion: '8.0.0',
  namespaces: ['somespace'],
  originId: '111',
}

In addition, an Alias is created with ID 'namespace:bar:111', which points to the new object ID.

---

Scenario 2: "baz" object

---

Existing object:

{
  type: 'baz',
  id: '333',
  migrationVersion: { 'baz': '7.9.0' },
  references: [{ type: 'foo', id: '222' }, { type: 'bar', id: '111' }],
  namespace: 'somespace',
}

Transforms: migration-7.10.0-baz --> reference-8.0.0-foo --> reference-8.0.0-bar --> migration-8.0.0-baz

Output object:

{
  type: 'baz',
  id: '333',
  migrationVersion: { 'baz': '8.0.0' },
  references: [
    { type: 'foo', id: deterministicNewId('somespace:foo:222') },
    { type: 'bar', id: deterministicNewId('somespace:bar:111') }
  ],
  referenceMigrationVersion: '8.0.0'
  namespace: 'somespace',
}

---

3. Aliases

3.1. Usage

A new API will be exposed to retrieve objects use aliases. Instead of SavedObjectsClient.get, consumers should use SavedObjectsClient.resolve. For example, if a consumer calls client.resolve('bar', '111'), it will attempt to find an object with the ID or a matching alias with that ID.

If an object and alias exist for the same namespace+type+id combination, a new "alias conflict" error will occur. That must be resolved by the end user -- either the object must be deleted, or the alias must be disabled/deleted. Some diagrams that describe this scenario are below.

(Diagrams)

---

---

Since "alias conflict" errors can occur, at a minimum users need to be able to directly read and delete object aliases. A user interface will also be needed to facilitate this. We can add two pages:

• Stack management / Saved objects / Aliases: New tab in the existing "Saved objects" page. Renders a table that lists all aliases with the following information: type, ID, target title, last resolved, and error.
  • The ID is a link that takes you to another page for that particular alias.
  • The target title is a link that takes you to another page for that particular object (defined by getTitle and getInAppUrl when that object type is registered).
  • The last resolved column displays when this alias was last resolved (when the target object was last accessed through this alias).
  • The error column displays any "not found" or "alias conflict" error for this alias.
• Stack management / Saved objects / Aliases / <Alias ID>: Renders a page that shows the same information for that row in the aforementioned table. This will be needed so we can deep link to this page from other Kibana apps when an "alias conflict" error is encountered.

3.2. Technical Details

How aliases will be stored

Aliases will be stored as separate saved objects.

(Example)

{
  type: 'dashboard',
  id: 'cf3b7012-4abf-4d17-9874-8b55bf7fc844',
  attributes: { ... },
  namespaces: ['somespace']
}
{
  type: 'alias',
  id: 'somespace:dashboard:old-dashboard-id',
  attributes: {
    targetType: 'dashboard',
    targetId: 'cf3b7012-4abf-4d17-9874-8b55bf7fc844',
    lastResolved: '2020-10-22T20:09:50.587Z'
  },
  namespaces: ['somespace']
}

We also considered storing aliases in an array inside of each existing object, but decided on this approach instead. Benefits of this approach:

• Less effort to implement
• Better performance to retrieve a specific alias by its ID (using get instead of search)
• No changes to “root fields” for saved objects
• Alias raw ID ensures that there will never be multiple aliases with the same target namespace+type+ID
• If we ever want to resolve aliases in bulk operations, this will be much easier to work with

How aliases will be resolved

The action of "resolving" an alias attempts to find the specified object (namespace + type + ID). There are four possible outcomes when resolving an alias:

Aliases will only be resolved by using the new SavedObjectsClient.resolve API.

We also considered automatically resolving aliases in all SOC APIs, but decided on this approach instead. Benefits of this approach:

• Much less effort to implement
• Less effort for consumers that will not be interested in resolving aliases
• If an “alias conflict” exists, it has a less severe impact on the user experience

How users can interact with aliases

TBD Decision: end users can only read and delete aliases.

(Two approaches were considered -- click to expand)

In the first approach, end users can only read and delete aliases.

In the second approach, end users can create, read, update, and delete aliases.

Benefits of each approach are detailed below:

Approach 1	Approach 2
* Less effort to implement * We can add create/update functionality later without any rework	* More useful, really makes aliases into a first-class feature

Regardless of which approach is used, alias authorization would be determined by the user's Kibana privileges the same way that it is handled for other saved objects.

Time permitting: users will also be able to disable aliases. This may come in handy if they're not sure whether the alias is being used or not.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[kobelb]

Something that recently occurred to me is that I've generally presented index-patterns as an example of a saved-object which would be supported starting in Phase 2. index-patterns don't have references to any saved-objects. However, saved-objects do have references to index-patterns.

When originally writing #27004, this seemed fine. It required that the migration be written such that when we regenerate the IDs for these index-patterns, the saved-objects which reference the index-patterns are updated. However, given what we've learned in #59598, not all saved-objects are using references correctly right now.

[jportner]

@elastic/kibana-security @elastic/kibana-platform @kobelb @stacey-gammon @arisonl I've spent some time discussing this feature with Larry, Brandon, and Rudolf, and I worked out some of the details with a POC. I think I'm ready to finalize the requirements and the technical approach so I can build it. All of your input would be appreciated.

I didn't think this really necessitated a full blown Architecture Review meeting, but I'd be happy to schedule one if need be (or hop on separate call/s to walk through and discuss the details).

@stacey-gammon @arisonl In particular I'm looking for feedback from you on Section 3 Aliases --
We could pretty eaisly extend this feature to allow any user to create custom aliases for any object. However, my understanding is that there's some concern about overlaps with "short URLs". If we aren't ready to make a decision on that yet, we could simply use "Approach 1" for this issue and revisit in the future if we decide to do so.

NOTE: The original description for Phase 2 included converting Index Patterns into multi-namespace object types. That is currently blocked based on the issues at the top of the description. However, we can still build the entirety of Phase 2 without being blocked.

[rudolf]

When would a plugin prefer to use SavedObjectsClient::get over SavedObjectsClient::resolve? I'm concerned that we have two methods that are very similar, but choosing the wrong one could resolve in bugs with a high impact on users.

How do aliases work with the Saved Objects HTTP API? Unless we resolve ID's for all operations I think this will break any existing integrations (where a hardcoded Saved Object ID is being used)?

I might not understand the use-cases we have in mind, but my first reaction is that we should not expose aliases to users as a first-class concept. Although a plugin might wish to implement functionality to create "links" that point to other objects I don't think we should bring this concept into the SavedObjectsRepository. It's not a feature a storage/persistence layer should be concerned with, its part of the plugin's domain. This is better modelled in a higher layer like we currently do with URL saved objects.

[rudolf]

Do we intend to add a reference from a saved object to it's alias so that the alias is also exported?

[jportner]

I had conversations offline with @stacey-gammon and @rudolf. Summary:

How users can interact with Aliases: There is not much real benefit to allowing users to create and edit Aliases. Many URLs in Kibana are quite long (Dashboard, Visualization, etc.) because of our current reliance on storing state in the URL. These users are better served by existing short URL functionality. In a nutshell: we shouldn't add functionality "just because we can", because then we're committing to supporting it long term. As such, we'll keep Alias functionality to a minimum.

Consensus appears to be that Aliases are a necessary stop-gap in converting existing saved single-namespace objects to multi-namespace types. In a nutshell:

1. We need to change object IDs when we convert them
2. By changing object IDs, we break existing URLs (and any 3rd-party integrations that use those URLs)
3. The introduction of Aliases allow consumers to optionally continue to support existing URLs that use pre-conversion object IDs

We will limit the total impact of Aliases by:

• Only creating Aliases when objects are converted to multi-namespace types
• Only creating Aliases in non-Default spaces
• Only resolving Aliases in one API (SavedObjectsClient.resolve)
• Requiring consumers to "opt-in" to supporting Aliases by changing calls from get to resolve

The resolve API should only be used by consumers on existing public-facing routes. They will also need to change any code that makes subsequent API calls to use the object ID that was returned from the resolve API (as it may have resolved to a different object ID).

Nice to have / to be discussed:

• It would be great if we can be more proactive about informing users what aliases are / discourage their use. For example, when accessing a dashboard via its alias, show a warning message at the top of the screen saying something like "You're using an old URL to access this dashboard. This will go away soon. Click here to use the new URL."
• It may behoove us to delay conversion of existing types until they can all be done at the same time. This would likely reduce confusion and cognitive load for end users.

Do we intend to add a reference from a saved object to it's alias so that the alias is also exported?

In the spirit of limiting Alias usage/impact: no.

[arisonl]

My two cents from a user perspective, after chatting with @jportner:

• On aliases overall: Aliases should be transparent to users as much as possible. There doesn't seem to be much benefit for users to interact with aliases in a first class (saved object) citizen manner. In a sense this is evident from the fact that aliases re-introduce a flavour of the very same problem that this broader work is designed to solve. The underlying model is already complex enough and we want to abstract it for the end user to the extent that this is possible.
• On alias conflicts: In principle we would want a user action to be requested at the latest time in order to avoid disrupting users that will not face the conflict. Here however it seems that users set up for a conflict mainly when they share to a space an object that has been previously copied in that same space (in which case external references would not be resolvable). So it makes sense to question if this is a legit use case in the first place. Why would someone do this and do they really want both the shared object and the copied object or is it a misuse? (Maybe they copied the object in order to sync it, in the absence of a proper share previously, they created an external deeplink to it and now that sharing became available, they are trying to leverage it. In this case they should probably do a little clean up before, delete the old copy, then share and update their external deeplink). If we don't have a legit use case, this situation might deserve guidance/management sooner rather than later, for example in the form of a warning at the point of the attempted share, e.g.: "You are attempting to share an object to a space where a copy of that object already exists. Please make sure you apply best practices: [link]. If you choose to proceed, any links to the pre-existing object will need to be updated”.

[rudolf]

It may behoove us to delay conversion of existing types until they can all be done at the same time. This would likely reduce confusion and cognitive load for end users.

Since rewriting ID's will break any Saved Object API integrations this will be a breaking change for users. So the plan as outlined in this issue can only be released in 8.x

Having an integration fail every time a new Saved Object type gets converted across several minors will be a very painful experience. Users will more likely adopt the breaking change if it means they can use tagging across all their plugins instead of just a handful.

Although we can explore resolving ID's in the Saved Objects API, I would motivate that we postpone releasing this until 8.x. This will also allow us to remove "psuedo copy" which will further simplify the user experience and prevent users from getting into sticky situations.

[jportner]

Since rewriting ID's will break any Saved Object API integrations this will be a breaking change for users. So the plan as outlined in this issue can only be released in 8.x

Having an integration fail every time a new Saved Object type gets converted across several minors will be a very painful experience. Users will more likely adopt the breaking change if it means they can use tagging across all their plugins instead of just a handful.

I agree that having conversions spread out across minors (and potentially causing integrations to fail across minors) is undesirable, and we should avoid it by lumping all conversions together in a single release. We had good intentions when we planned to convert index patterns first, but that's not even possible for the foreseeable future anyway, so we might as well wait IMO.

However, "rewriting ID's will break any Saved Object API integrations": I feel like that statement is disingenuous. It might break some existing integrations that depend upon old IDs in non-Default Spaces, but it isn't guaranteed to break any/all integrations. The introduction of Aliases is specifically intended to avoid breaking external integrations which depend upon Kibana app URLs (such as iframed dashboards/visualizations, for example). These saved objects APIs are experimental and we can change them in a potentially-breaking way in a minor release, though we should have very good reason to do so.

Although we can explore resolving ID's in the Saved Objects API, I would motivate that we postpone releasing this until 8.x. This will also allow us to remove "psuedo copy" which will further simplify the user experience and prevent users from getting into sticky situations.

If we don't convert existing saved objects during 7.x, we won't be able to obtain accurate telemetry data to determine if people still depend upon "pseudo copy" before 8.0. We have our assumptions that such usage will taper off when all saved objects are shareable, but we aren't sure. There may be other use cases for pseudo copy that we aren't yet considering.

[rudolf]

These saved objects APIs are experimental and we can change them in a potentially-breaking way in a minor release, though we should have very good reason to do so.

Agreed. We should rather break these experimental API's than introduce technical debt which we might never be able to remove or otherwise prevent us from delivering value to users. Still, unless we know we can deliver significant value to users in 7.x, which would mean several plugins adopting multiple namespaces this might not be justified.

If we don't convert existing saved objects during 7.x, we won't be able to obtain accurate telemetry data to determine if people still depend upon "pseudo copy" before 8.0.

As I see it, the reason we want to remove "psuedo copy" is because in a multi-namespace world this feature is fundamentally broken. Keeping it will create confusion and difficult to resolve conflicts for users (not to mention the technical complexity it adds to our code base). If we discover workflows that depend on this we should rather find better alternatives.

One use case we already know of is import and export and more specifically selective import/export. The problem is that import/export was written for a single namespace world. When you export a multi-namespace object you loose all the namespace data associated with that object. There is no way to restore an object's namespaces to a previous state. Keeping "psuedo copy" will just put a plaster over the fact that our exports are no longer true backups of the data inside Kibana.

[jportner]

I met with @arisonl @mdefazio @kobelb @bmcconaghy and @legrego today to discuss the UI/UX for Aliases. Here are the salient points and decisions (please keep me honest if I fudged or forgot anything, folks!):

• We don't need to create Aliases for all converted object types, only those which expose an ID in some external URL. Accordingly, we will expose an option for this in object type registration so we can minimize the impact of Aliases in general.
• "Aliases" is a bit of a confusing term for end users, and our original intent for this feature has been reduced in scope. We should use a more descriptive term when referring to these in the UI. Tentatively we can call these "legacy URLs", which is more intuitive, as that's the only way end users will encounter these objects.
• When a user accesses a saved object using its alias, we should redirect them to the new URL. This will hopefully help reduce the proliferation of these legacy URLs.
• When an "Alias conflict" is encountered, we should not throw an error -- this would not be resolvable by all users (for instance, those who have read-only access). Instead, we should return the most correct result (the object with an ID that is an exact match), and let the consumer know that a legacy URL exists for this object.
• While we can't guarantee that Kibana will never get into a state where an "Alias conflict" is encountered, we can attempt to prevent this from occurring. The most common way that this situation would occur is if a user shared a previously-copied object from the Default space to another space where an alias of the same ID already exists. So, we should change the Share to Space flyout to prevent a user from doing this in the first place. It's not bulletproof, but it should reduce confusion.
• In light of all of the above, and our hope that experiencing these "Alias conflict" situations will be rare: we don't necessarily need a dedicated UI to manage saved object aliases. We can release Phase 2 without this, but we should collect telemetry data and keep an ear out for user feedback so we can decide if we need to add this in the future.
• We'll provide a dedicated HTTP API to delete aliases as a break-the-glass scenario for admins.
• We'll plan to get rid of Aliases entirely in the 9.0 release.

To summarize, here are the only ways that end users will interact with Aliases:

1. Access an object using its Alias ("legacy URL"): the user will experience a seamless client-side redirect to the new URL, and will see a warning letting them know this happened. Consumers will need to implement this.
2. Access an object and encounter an "Alias conflict": the user will see a warning letting them know another object exists that used to be accessed at this URL, along with a link to access said object at its new URL. Consumers will need to implement this.
3. Share an object to a space which would cause an "Alias conflict": the user will see a warning explaining that a legacy URL exists which points to a different object in that space. The user can either cancel the operation, or they can delete the legacy URL. I will implement this.

[rudolf]

@joshdover wondered if we could introduce a resolve query paramater to all HTTP Saved Object API's that defaults to true in 7.x. We could then change our public Saved Objects client to always set resolve=false so that Kibana code doesn't depend on resolve and doesn't have any performance degradation.

This will significantly reduce the impact on users using our HTTP API. This will probably degrade performance but then the solution would be for users to adapt their code and change ?resolve=false. In 8.x we can remove the resolve query parameter entirely.

Overall I feel like this has a low impact on the code base while significantly lowering the impact on users. Could this work or am I missing something critical?

[jportner]

Overall I feel like this has a low impact on the code base while significantly lowering the impact on users. Could this work or am I missing something critical?

I solicited feedback internally before finalizing the "Technical details" section and we agreed not to allow resolving on multiple/all SO APIs. That creates a bigger testing burden for a feature that we ultimately don't want to continue supporting forever. Not to mention that consumers need to understand the resolution "outcome" which required that we change the shape of the response. Where the get API response is this:

{
  "body": [Object]
}

the resolve API response is this:

{
  "body": {
    saved_object: [Object],
    outcome: "exactMatch" // or "aliasMatch" or "conflict"
  }
}

[kobelb]

@rudolf and @joshdover, I don't think that we should be adding a resolve query parameter for all of the APIs at this time.

Our biggest backward compatibility concern is continuing to support the existing URLs. URLs are a stable contract that we're obligated to support because they're GA and not experimental or beta. Users may have created bookmarks to a specific Dashboard, Visualization, etc. or put links in a Markdown visualization to enable intra-app navigation. During minor version updates, we should continue to support URLs regardless of the amount of effort, maintenance burden or technical debt we incur doing so as breaking this contract would have a significant impact on a large number of users.

Regenerating IDs, with the current plan, will create a breaking change for consumers of the saved-object APIs, but I think we should treat this differently than URLs. The breaking change that we're planning to implement will only be for programmatic usage of these APIs for saved-objects outside of the default space using a static ID. This breaking change won't affect all usages of the saved-object APIs, only certain usages. We already have far fewer consumers of these APIs in general, which is further reduced by the conditions in which these breaking changes are relevant. Additionally, the saved-object APIs are currently marked as experimental, so while I do think we should be avoiding introducing breaking changes out of consideration for our users, we do have the leeway to do so when necessary. However, it's my understanding that @jportner is planning to implement telemetry to determine the usage of these APIs, and if we find out from this telemetry that we do have a ton of programmatic consumers outside of the default space, we should likely revisit this decision.

[jportner]

However, it's my understanding that @jportner is planning to implement telemetry to determine the usage of these APIs

Yes! Issue: #85621

[pgayvallet]

I don't think that we should be adding a resolve query parameter for all of the APIs at this time.

Our biggest backward compatibility concern is continuing to support the existing URLs. URLs are a stable contract that we're obligated to support because they're GA and not experimental or beta. Users may have created bookmarks to a specific Dashboard, Visualization, etc. or put links in a Markdown visualization to enable intra-app navigation

So, following this logic, it is the type owner's responsibility to change all its calls to resolve instead of using get when they'll change their type from single to multi namespace?

I may be wrong, but this seems like way more error-prone, and overall 'dangerous' than allowing to do that 'under the hood' for our base SO apis?

[jportner]

So, following this logic, it is the type owner's responsibility to change all its calls to resolve instead of using get when they'll change their type from single to multi namespace?

Yes, exactly.

I may be wrong, but this seems like way more error-prone, and overall 'dangerous' than allowing to do that 'under the hood' for our base SO apis?

Well:

• When we implemented Kibana Feature Privileges, we did it on behalf of consumers. That turned out to be a mess, as they did not fully grasp the changes that we made, and inadvertently things wound up breaking because of it. We want to avoid another situation like that moving forward, which is one reason why we think it's best to force the consumer (type owner) to make this change as part of the conversion process.
• The truth is, this cannot be done fully transparently. The consumer must check the outcome of the resolve operation exactMatch, aliasMatch, or conflict) and then display additional UI elements (Sharing saved-objects in multiple spaces: phase 2.5 #85791) in the case of the two latter outcomes.
• I would argue that adding resolutions for all of our base SO APIs is more dangerous, as it would proliferate consumers' reliance on legacy URL aliases that we want to eliminate in a future release. As of now, this should only affect a small number of consumers, those with existing single-namespace saved objects that have external URLs which allow object IDs to be passed in.