Sharing saved-objects in multiple spaces

[jinmu03]

Requirements

Currently, Kibana only supports the ability to copy a saved-object to multiple-spaces. When copying a saved-object, all copies are separate saved-objects and the modification to any copy is not reflected in the other copies. This behavior is ideal in some situations, and problematic for others. This proposal is focused on adding the ability to share saved-objects in multiple spaces, so changes are automatically reflected in all spaces in which the saved-object has been shared.

The ability to share saved-objects in multiple spaces will alleviate two primary drawbacks to only supporting copies:

1. Difficult to keep copies in sync
2. Performance and storage impact of requiring a copy

When a user copies a saved-object to multiple spaces, the updates to the saved-objects are isolated from each other. This is problematic when the user wishes to update all copies. The user is currently required to manually update all copies utilizing the “Copy to Space” functionality that was introduced in 7.4, or cobble together some other solution. This is a common complaint for index-patterns. When an index-pattern’s mappings are refreshed or a scripted-field is added, if the intent is for this change to be made in all copies it’s a manual and quite painful process.

Additionally, some usages of saved-objects have a significant impact on the performance and storage requirements of both Elasticsearch and Kibana. In these situations, when a saved-object is copied the copy itself will consume as many resources as the original saved-object. An example of this is ML. When ML starts using Kibana’s saved-objects to provide more granular access to their jobs and other entities, a copy is a potentially dangerous operation. An Elasticsearch ML job consumes a potentially large amount of cluster resources to run, and as such a job should be copied with care.

When a saved-object is shared to a space, the saved-object will be modifiable in any of the spaces in which it has been shared. Since the saved-object can be modified in any space its been shared, this introduces some complexities for saved-objects which have references to other saved-objects. Before a saved-object can be shared to a space, all direct and transitive references to saved-objects must be shared. For example, dashboards have references to visualizations, which have a reference to an index-pattern. When a dashboard is shared to a space, all referenced visualizations and in-turn index-patterns must be shared before the dashboard is shared.

Additionally, when a saved-object is either created or updated and references new saved-objects, the references will be checked to ensure they exist in at least all spaces which the saved-object exists in. This will prevent saved-objects from being updated or created with broken references.

In the following situation, Bob only has access to the marketing space and creates a dashboard and two visualizations.

Alice has access to both the Marketing and Sales spaces, and shares the dashboard and visualizations to the Sales space.

Both Alice and Bob are able to edit Dashboard A; however, both are constrained to only adding visualizations which exist in both the Marketing and Sales space. This limits Bob in the types of edits he can make, but it doesn’t completely remove his ability to make edits.

Further elaborating upon the previous situation, if Alice was to unshare Visualization A from the Marketing space, we could potentially prevent her operation from occurring because it would make Dashboard A have a broken reference from the Marketing space. However, we already have a similar situation when we allow a Visualization to be deleted when it is referenced by a Dashboard within the same space.

This proposal relies on high-level referential integrity checking by default on create/update logic to ensure that references aren’t being broken. Programatically, developers will be able to circumvent this protection; however, it will be enabled by default. Broken references are generally beneficial to avoid, but in the case where they are broken it’s not a cataclysmic event.

From an authorization perspective, not many changes are required. When a saved-object is shared to a new space, the user will need to be authorized to write the saved-object in both the source and target spaces. However, when a user edits a saved-object, they will only need to be authorized to write the saved-object in their current space. For example, in the following situation Bob is only authorized to edit dashboards and visualizations in the Marketing space.

Bob will therefore be able to edit the dashboards and visualizations which are shared in the Marketing and Sales spaces. However, once object-level security is implemented the user will be able to specify a per-object ACL to make the shared saved-object read-only.

Implementation Phases #

The ability to share a saved-objects in multiple spaces will be implemented in three primary phases. Not all saved-objects of the specific "category" will automatically be able to be shared, and this will require effort from application and solution teams before a saved-object can be shared. For example, with Phase 1 not all new saved-object types will automatically be "shareable".

Phase 1: New saved-object types without references

Release: 7.8 #54043

This will allow ML to begin taking advantage of the ability to share saved-objects in multiple spaces. ML jobs haven't transitioned to take advantage of saved-objects, and this will allow us to reduce the complexity that migrations introduce

Phase 1.5: Management

Release: 7.10 #58139

This will support import/export for "shareable" types (saved object types marked as multiNamespace) without references. It will also support existing operations in the Saved Objects Management Screen (server side and client side) and add a UI for sharing a saved object to other space(s).

Ancillary task: Sharing to all spaces

Release: 7.10 #69808

As originally envisioned, sharing saved objects would only support explicitly-named spaces. However, recent use cases have surfaced that would benefit from the ability to share to all current and future spaces. This will support such use cases.

Phase 2: Existing saved-object types without references.

Release: 7.12 #54837

This will allow index-patterns to be shared in multiple-spaces. It requires that migrations and aliases are implemented, but doesn't require that the referential integrity checking be implemented.

Update: this feature is ready to merge in 7.11, but we decided to push it back to 7.12 to minimize risk of migration changes, which are already significantly changing in the 7.11 release.

Phase 2.5: Shared UI components

Release: 7.12 #85791

This will provide shared UI components for consumers to leverage when converting objects to multi=namespace types.

Phase 3: All saved-object types with references

Release: 7.14 #67380

Everything can now begin taking advantage of being shared in multiple spaces. As previously stated, given the level of effort that it will take to customize the various applications to inform the user that the saved-object is shared in multiple spaces and allow direct and transitive references to be shared, this will be taken advantage of incrementally by the application and solution teams.

Phase 3.5: UI for sharing with references

Release: 7.14 #100424

Phase 3 added server-side support for fetching an object and all of its references, and for sharing multiple objects. Due to the size of those changes, this phase was split into a separate PR. It includes: changing the UI for sharing an object with all of its references, updating the look and feel of the Share flyout to match a newer design mockup, and adding an additional step in the Share flyout to check for alias conflicts and prevent them from happening.

⚠️ Rollout Plan ⚠️

As of 2020/12/08: The original plan was to convert "index-pattern" saved objects to become shareable with the release of Phase 2, and allow plugins to convert other existing object types in a phased approach. It later became apparent that index-pattern objects cannot be converted this soon, because we need to account for inbound references (and some other object types do not use references using the standard method). There was some perceivable value to the goal of converting index-patterns first, but ultimately allowing some existing objects to be converted sooner than others is likely to cause user confusion and additional cognitive load. In the interest of simplifying things for the end-user, we decided to wait to convert all existing object types in a "big bang" release. Hopefully this will be a minor release before 8.0, but that remains to be seen. Update: this conversion will happen in the 8.0 release. See the meta-issue #100489 for details.

Technical Solution

When implementing Spaces, we prefixed a saved object’s Elasticsearch document ID with the namespace and added a singular namespace field. Prefixing the Elasticsearch document ID with the namespace allowed us to copy saved-objects from one Space to another without having to worry about conflicts, as the Elasticsearch document IDs are unique. Since the end-goal is to have saved-objects belong to multiple namespaces , we can no longer prefix the document ID with the namespace. The following technical solution expands upon the method in which we’re proposing to remove namespace from the Elasticsearch document ID, change the namespace field to namespaces and maintain backwards compatibility with existing Kibana URLs.

Brief Summary

This is the current document structure:

{
  _id: NAMESPACE:TYPE:ID;
  type: string;
  namespace: string;
}

We’d like to remove the NAMESPACE, from the _id and change namespace to namespaces. This would leave us with this new document structure:

{
  _id: TYPE:ID;
  type: string;
  namespaces: string[];
}

The issue is that we’re likely going to have quite a few _id collisions when doing so. Kibana currently supports the ability to have multiple saved-objects with the same ID in different namespaces. For example:

{
  _id: ‘marketing:index-pattern:1’;
  type: ‘index-pattern’;
  namespace: ‘marketing’;
}

And

{
  _id: sales:index-pattern:1’;
  type: ‘index-pattern’;
  namespace: sales;
}

These two documents will exist if we copy an index-pattern from the sales space to the marketing space, or vice-versa. If we remove the NAMESPACE from the _id for both of these documents, we’d end up with two documents with an _id of index-pattern:1.When we detect this collision during a migration, we have the option of regenerating an ID. However, we’ve been primarily concerned that doing so is going to break existing URLs in Kibana.

To prevent us from breaking URLs in Kibana and address another pain-point which users have experienced, we’d like to introduce the ability for end-users to manage saved-object aliases. Saved-object aliases are used when translating from an identifier in an application’s front-end route to the referenced saved-object. Prior to this change, front-end routes would include the actual saved-object ID, for example: http://localhost:5601/app/kibana#/dashboard/edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b. However, with the introduction of saved-object aliases, we’d allow the user to create a custom alias which references the actual saved-object ID. This would allow a front-end route like http://localhost:5601/app/kibana#/dashboard/foo to be used instead of the aforementioned route which includes the explicit saved-object ID.

Now that we have saved-object aliases, when we detect a collision during the migration, we regenerate a new ID and create an alias document, similar to the following:

{
  _id: alias:sales:index-pattern:1’;
  type: ‘alias’;
  namespaces: [‘sales’];
  ‘alias’: {
    type: ‘index-pattern’,
    id: ‘2’
  }
}

We also need to keep track of the old and new IDs. For the sake of the migration itself, an in-memory map is sufficient; however, to better support imports when an ID had to be regenerated, we’ll be persisting these in a regenerated-id saved-object.

This leaves us with the following documents to model the previous situation where we had an index-pattern with ID of 1 in both the sales and marketing namespaces.

{
  _id: ‘index-pattern:1’,
  type: ‘index-pattern’,
  namespaces: [‘marketing’]
}

{
  _id: ‘index-pattern:2’,
  type: ‘index-pattern’,
  namespaces: [‘sales’]
}

{
  _id: ‘alias:sales:index-pattern:1’,
  type: ‘alias’,
  namespaces: [‘sales’].
  ‘alias’: {
    type: ‘index-pattern’,
    id: ‘2’
  }
}

{
  _id: ‘regenerated-id:1’,
  type: ‘regenerated-id’,
  namespaces: [‘sales’],
  ‘regenerated-id’: {
    ‘old’: 1,
    ‘new’: 2
  }
}

Then, we no longer rely directly upon SavedObjectsClient::get when retrieving the saved object based on the identifier in the URL. Instead, we add a new SavedObjectsClient::resolve, which will return a saved-object using the type and identifier. The resolve method will look for an alias which matches the type and identifier in addition to a normal saved-object which matches the type and identifier. If we find only an alias, we return the referenced saved-object. If we only find an actual saved-object, we return the actual saved-object. If we find both, we throw an error and require the user to remedy this using Saved Object management. This way, we don’t alter the behavior of SavedObjectClient::get, and it always returns saved-objects based on their ID.

Migration Pseudocode

The following assumes we’ve topologically sorted all saved objects by saved-object types:

const savedObjectIds = new Set();
const inMemoryRegeneratedIds = new Map();

for (const savedObject of topologicallySortedSavedObjects) {
  if (savedObjectIds.has(`${savedObject.type}:${savedObject.id}`)) {
      const id = new Guid();

      shortUrls.create({
          url: savedObject.id,
          namespace: savedObject.namespace,
          type: savedObject.type,
          id: id
      });

      inMemoryRegeneratedIds.add(`${savedObject.type}:${savedObject.id}`, id);
      persistentRegeneratedIds.create({ // these are used later during import
          type: savedObject.type,
          old: savedObject.id,
          new: id,
      });
      
      savedObject.id = id;
  }

  for (const reference of savedObject.references) {
      if (inMemoryRegeneratedIds.has(`${reference.type}:${reference.id}`) {
          reference.id = inMemoryRegeneratedIds.get(`${reference.type}:${reference.id}`);
      }
  }

  savedObjectIds.add(savedObject.id);
}

SavedObjectsClient::resolve Pseudocode

const { type, identifier } = request.params;
const namespace = getNamespace(request.url);

const actualSavedObject = await savedObjectsClient.get(type, identifier);
const alias = await savedObjectsClient.get(alias, `${namespace}:${type}:${identifier}`);

if (!actualSavedObject && !alias) {
 throw new NotFoundError();
}

if (actualSavedObject && alias) {
 throw new GeneralError(`You've shadowed a saved-object, you must delete the alias.`);
}

if (actualSavedObject) {
 return actualSavedObject;
}

if (alias) {
 return await savedObjectsClient.get(customUrl.attributes.type, customUrl.attributes.id);
}

SavedObjectsRepository Operations

The following logic changes will be implemented within the SavedObjectsRepository, which exists within OSS Kibana. Currently, the SavedObjectsRepository ensures that saved-objects in different namespaces are isolated from each other and no conflicts exist. This is by virtue of the namespace being part of the Elasticsearch document ID. Additionally, the SavedObjectsRepository ensures that saved-objects which are being retrieved using the get, bulkGet and find methods all respect the namespace that is specified, and saved-objects in other namespaces are not retrieved.

With the changes being proposed, we can no longer ensure that saved-objects in different namespaces are always isolated from each other and no conflicts exist. Instead, we will be modifying the logic within the SavedObjectsRepository to reflect the new reality that conflicts may exist between saved-objects in namespaces.

Create and BulkCreate

• Perform a pre-create check to ensure all referenced saved-objects exist within the current namespace
• If the id isn’t specified
  • Create a new ID and a new saved-object is created
• If the id is specified and overwrite is false,
  • Try to insert as-is. If there’s a conflict, throw the conflict error
• If the id is specified and overwrite is true
  • Perform a pre-create check to ensure that we aren’t overwriting a saved-object which exists in a different namespace.
    • Execute a GET or _bulk_get to retrieve any existing saved-objects which already exist
    • If the saved-objects already exist, ensure the current namespace is already in the saved-object’s namespaces.
      • If it is, execute an update excluding the namespaces field, preserving the existing namespaces
      • If it is not, return a 409 Conflict
  • If the saved-object doesn’t already exist, execute a create

Update

• Perform a pre-update check to ensure that we’re updating a saved-object which exists in the current namespace and all newly referenced saved-objects exist within the current namespace.
  • GET the existing saved-object
  • If the current namespace isn’t in the namespaces, we throw a 409 Conflict error
  • For any newly referenced saved-objects perform a _bulk_get. If their namespaces don't include the updated saved-objects namespaces, throw an error.
  • Otherwise, proceed with the update

Get and BulkGet

• Perform the normal get or bulkGet
• Filter out any responses which don’t include the current namespace

Find

• Change the query generated from namespace to namespaces

Delete

• Perform a pre-delete check to ensure that the saved-object we’re deleting exists in the current namespace
  • GET the existing saved-object
  • If the saved-object exists in the current namespace, delete the saved-object
  • Otherwise, throw a 404 Not Found

DeleteByNamespace

Note: This is the internal method which is used to delete a space and all of the saved-objects in that space. It likely should be renamed based on the changes in logic I'm proposing.

• Perform a _delete_by_query for all saved-objects which only exist in the current namespace
• Perform a _update_by_query to remove the current namespace from all saved-objects which exist in multiple namespaces

Other Saved Object Operations

Export

• Nothing should need to be changed ehre

Import

Note: The following is a gross over-simplification and needs further thought and investigation

• When validating references
  • If a reference doesn’t exist, consult the regenerated-id saved-objects and use the new ID
• When creating the objects
  • If bulkCreate returns a conflict, there are now multiple remediations
    • Overwrite
      • Only possible when the user has overwrite false
    • Generate new ID
      • When regenerating the ID, we’d also update all references to use the new ID
      • Create a new regenerated-id saved-object
    • Use previously generated new ID
      • Only possible when there’s a relevant regenerated-id saved-object. There could be multiple, if we let there be...
      • When choosing this, we’d also have to update all references to use the new ID

[elasticmachine]

Pinging @elastic/kibana-security

[Mekk]

I raised #35380 for scripted fields in particular. Those are sometimes uneasy to write, require patching, and having to update them in 17 spaces made by various people is just painful.

Some loose ideas:

• what about inheriting new space from another one (so everything created on the „base” space is shared everywhere, but not the other way around – this would make it possible to maintain „exemplary/initial/official” spaces with curated scripted fields, patterns, common queries etc from which people would derive their private/team subspaces)

• what about explicit „import from another space” (or mayhaps „link to object from another space”) - on individual objects level, for example import just those two sripted fields defs

• what about „global” scripted fields available to all spaces based on index(es) those global fields target?

[Matheu-Chambers]

HI, Just thought I'd get in on this too... I've not much to add from what has already been said, really. My situation is that I'm creating dashboards for different users, but that use the same index-pattern, so having that available across spaces would be ideal...

thanks

[Mekk]

I abandoned using scripted fields (in part because of this problem), but I miss some way of „inheriting exemplary space” more and more – I maintain curated space with generally useful queries, visualizations, and dashboards, I encourage people to create their own spaces so they can freely create and save their own objects, but once they do it, they loose all the base (and even if they export/import them, they no longer get them updated after base changes, and no longer see new items). In the end people mostly switch between spaces, but this is confusing.

So, I think it would be best to „inherit”, so queries, dashboards, visualizations, visualizations and index patters can be seen and used in the child space.

Of course changes made in the child space should not impact the base (probably it would suffice to reject an attempt to save updated query or dashboard if it is imported, save as will work…, or mayhaps in such a case new object should be created in child space shielding the original)

[kobelb]

For those following along, the description has been updated with the requirements and general technical approach

[geheim01]

Is it working? I'm pretty interesed in having a visualization in two spaces, synced at the same time. So pretty much what you wrote. But this thread reads more like an idea of how to do it than a proper implementation. Is there any best practice how to do it?

Many thanks

[arisonl]

@ModernMozart Thanks for your interest. This feature is at an early stage of development and will be available in future releases. Would be great to know more about your use case, if you could share.

[XavierRamosORGADATA]

Yes, having dashboards and visualizations synced across different spaces would be awesome!

[Mekk]

My use case, just in case ;-)

1. I, powerful and kibana-skillful Mekk, maintain curated "Main" space. There I have 10 crucial index patterns, 24 useful queries, 11 dashboards, some timelions, a few painless fields, etc. I also have some useful settings like a bit different list of time ranges for Discover, much bigger row width and more compact datetime format. From time to time (usually a few times a month) I add sth new, or tweak some element.

2. New users come regularly. When some Paul or Jenny arrives and asks how to start with Kibana, I want him or her to have own space (after all they are going to start saving own queries fairly quickly, and attempts to share single space ended up with horrible mess of 250 chaotically named queries nobody could use anymore). But I also want him or her to see my queries and dashboards, as they are handy. I am also tired with showing how to clickety-create 10 index patterns.
   So I dream about being able to say „click Create Space, choose inherit from Main, and you will be done, then start from trying queries you will find in Discover/Open”.

3. I'd also like them to see my modifications. New query for new kind of data. New index pattern. New column in old query. Updated dashboard. I dream about them seeing those without the need to perform any action. There are many of them, those users. They miss mails, they misclick, they don't care. And they need not even know that I fixed dashoard definition because it had wrong formula. And I really don't want to check whether they see sth wrong because they still have unpatched one.

4. Well, as Paula started to write interesting dashboards used by testers, it would be nice to let her curate Tests space. My queries are helpful for them too, so it would be nice if Tests inherited from Main, while Joe, Philip, Marie (and Paula too) could create personal spaces inheriting from Tests (so they could see both items created in Main, and in Tests). Best if those who already created space could patch it to change inheritance without loosing own saved items. Or maybe add new inheritance without removing previous one. I am not quite sure whether it is better to have inheritance chain, or multi-ihneritance. Both

     Main <---- Tests <---- Philip

     Main<----Philip
                  |
                 \/
                Tests

would probably do.

5. Getting changes upstream (like getting to Main nice dashboard Adam wrote in his own space) is less critical, here export/import may be sufficient (not that I'd complain if there were some glue to click it directly and detect and resolve possible conflicts). But it is more important to have this dashboard automatically visible by everybody who inherited from Main once it arrives.

[kobelb]

Hey @Mekk, these use-cases that you've provided will be possible once we add the ability to share saved-objects in multiple space in addition to object-level security. Object-level security will allow you to on a per saved-object level specify which users should be able to read, write or share the saved-object. This will allow you to either create an individual space for each of these users, share your dashboards, visualization and index-patterns to those per-user spaces in a read-only manner.

[Mekk]

From my point of view object-level security is not needed, and may in fact defeat the use case. Sharing and importing whole space would be by far sufficient – I wish to share whole space, all in this space, and nothing less. The only security needed is that few people can edit this Main space, while everybody can import/inherit from that space (use those objects), but without the right to modify.
I'd prefer to avoid the need to explicitly grant anything to anybody on object level (I sense very irritating hundred clicks whenever either new user, or new item appears). And I'm 100% sure I wouldn't like people to cherry-pick items (or need to select them in any way). Whole point of my use-case is that users can use everything what was prepared in curated Main space without any effort, including seeing changes.

(I suppose object-level security is more intended towards people sharing single space. IMHO this is worse than having separate spaces as such space ends up polluted and being difficult to use.)

[kobelb]

@Mekk do you currently model your use-case by granting these users read-only access to the curated "Main" space and then all access to their individual spaces? This would allow them to copy the dashboards, visualization, etc. from the Main space into their own individual spaces if they ever want to start making modifications, or else they can just defer to the saved-objects in the Main space.

[alexfrancoeur]

@jportner @arisonl have we delivered all foundational functionality planned for shareable SO's? As I understand it, other teams can start introducing experiences and features built around this. If that's the case, should we close this out? Or are we waiting for folks to start leveraging this functionality?

[jportner]

@jportner @arisonl have we delivered all foundational functionality planned for shareable SO's?

Yes!

As I understand it, other teams can start introducing experiences and features built around this.

Yes!

If that's the case, should we close this out? Or are we waiting for folks to start leveraging this functionality?

We still aim to introduce some developer documentation so folks can leverage this functionality: #127034

Aside from that, I think we can close this issue since we aren't trying to be prescriptive about when consumers leverage this functionality. I'll let @arisonl weigh in though.

Edit: Spoke to @legrego, we'll close this for now and we can reopen if we need to!