-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sharing saved-objects in multiple spaces #27004
Comments
Pinging @elastic/kibana-security |
I raised #35380 for scripted fields in particular. Those are sometimes uneasy to write, require patching, and having to update them in 17 spaces made by various people is just painful. Some loose ideas:
|
HI, Just thought I'd get in on this too... I've not much to add from what has already been said, really. My situation is that I'm creating dashboards for different users, but that use the same index-pattern, so having that available across spaces would be ideal... thanks |
I abandoned using scripted fields (in part because of this problem), but I miss some way of „inheriting exemplary space” more and more – I maintain curated space with generally useful queries, visualizations, and dashboards, I encourage people to create their own spaces so they can freely create and save their own objects, but once they do it, they loose all the base (and even if they export/import them, they no longer get them updated after base changes, and no longer see new items). In the end people mostly switch between spaces, but this is confusing. So, I think it would be best to „inherit”, so queries, dashboards, visualizations, visualizations and index patters can be seen and used in the child space. Of course changes made in the child space should not impact the base (probably it would suffice to reject an attempt to save updated query or dashboard if it is imported, save as will work…, or mayhaps in such a case new object should be created in child space shielding the original) |
For those following along, the description has been updated with the requirements and general technical approach |
Is it working? I'm pretty interesed in having a visualization in two spaces, synced at the same time. So pretty much what you wrote. But this thread reads more like an idea of how to do it than a proper implementation. Is there any best practice how to do it? Many thanks |
@ModernMozart Thanks for your interest. This feature is at an early stage of development and will be available in future releases. Would be great to know more about your use case, if you could share. |
Yes, having dashboards and visualizations synced across different spaces would be awesome! |
My use case, just in case ;-)
would probably do.
|
Hey @Mekk, these use-cases that you've provided will be possible once we add the ability to share saved-objects in multiple space in addition to object-level security. Object-level security will allow you to on a per saved-object level specify which users should be able to read, write or share the saved-object. This will allow you to either create an individual space for each of these users, share your dashboards, visualization and index-patterns to those per-user spaces in a read-only manner. |
From my point of view object-level security is not needed, and may in fact defeat the use case. Sharing and importing whole space would be by far sufficient – I wish to share whole space, all in this space, and nothing less. The only security needed is that few people can edit this Main space, while everybody can import/inherit from that space (use those objects), but without the right to modify. (I suppose object-level security is more intended towards people sharing single space. IMHO this is worse than having separate spaces as such space ends up polluted and being difficult to use.) |
@Mekk do you currently model your use-case by granting these users read-only access to the curated "Main" space and then all access to their individual spaces? This would allow them to copy the dashboards, visualization, etc. from the Main space into their own individual spaces if they ever want to start making modifications, or else they can just defer to the saved-objects in the Main space. |
@jportner @arisonl have we delivered all foundational functionality planned for shareable SO's? As I understand it, other teams can start introducing experiences and features built around this. If that's the case, should we close this out? Or are we waiting for folks to start leveraging this functionality? |
Yes!
Yes!
We still aim to introduce some developer documentation so folks can leverage this functionality: #127034 Aside from that, I think we can close this issue since we aren't trying to be prescriptive about when consumers leverage this functionality. I'll let @arisonl weigh in though. Edit: Spoke to @legrego, we'll close this for now and we can reopen if we need to! |
Requirements
Currently, Kibana only supports the ability to copy a saved-object to multiple-spaces. When copying a saved-object, all copies are separate saved-objects and the modification to any copy is not reflected in the other copies. This behavior is ideal in some situations, and problematic for others. This proposal is focused on adding the ability to share saved-objects in multiple spaces, so changes are automatically reflected in all spaces in which the saved-object has been shared.
The ability to share saved-objects in multiple spaces will alleviate two primary drawbacks to only supporting copies:
When a user copies a saved-object to multiple spaces, the updates to the saved-objects are isolated from each other. This is problematic when the user wishes to update all copies. The user is currently required to manually update all copies utilizing the “Copy to Space” functionality that was introduced in 7.4, or cobble together some other solution. This is a common complaint for index-patterns. When an index-pattern’s mappings are refreshed or a scripted-field is added, if the intent is for this change to be made in all copies it’s a manual and quite painful process.
Additionally, some usages of saved-objects have a significant impact on the performance and storage requirements of both Elasticsearch and Kibana. In these situations, when a saved-object is copied the copy itself will consume as many resources as the original saved-object. An example of this is ML. When ML starts using Kibana’s saved-objects to provide more granular access to their jobs and other entities, a copy is a potentially dangerous operation. An Elasticsearch ML job consumes a potentially large amount of cluster resources to run, and as such a job should be copied with care.
When a saved-object is shared to a space, the saved-object will be modifiable in any of the spaces in which it has been shared. Since the saved-object can be modified in any space its been shared, this introduces some complexities for saved-objects which have references to other saved-objects. Before a saved-object can be shared to a space, all direct and transitive references to saved-objects must be shared. For example, dashboards have references to visualizations, which have a reference to an index-pattern. When a dashboard is shared to a space, all referenced visualizations and in-turn index-patterns must be shared before the dashboard is shared.
Additionally, when a saved-object is either created or updated and references new saved-objects, the references will be checked to ensure they exist in at least all spaces which the saved-object exists in. This will prevent saved-objects from being updated or created with broken references.
In the following situation, Bob only has access to the marketing space and creates a dashboard and two visualizations.
Alice has access to both the Marketing and Sales spaces, and shares the dashboard and visualizations to the Sales space.
Both Alice and Bob are able to edit Dashboard A; however, both are constrained to only adding visualizations which exist in both the Marketing and Sales space. This limits Bob in the types of edits he can make, but it doesn’t completely remove his ability to make edits.
Further elaborating upon the previous situation, if Alice was to unshare Visualization A from the Marketing space, we could potentially prevent her operation from occurring because it would make Dashboard A have a broken reference from the Marketing space. However, we already have a similar situation when we allow a Visualization to be deleted when it is referenced by a Dashboard within the same space.
This proposal relies on high-level referential integrity checking by default on create/update logic to ensure that references aren’t being broken. Programatically, developers will be able to circumvent this protection; however, it will be enabled by default. Broken references are generally beneficial to avoid, but in the case where they are broken it’s not a cataclysmic event.
From an authorization perspective, not many changes are required. When a saved-object is shared to a new space, the user will need to be authorized to write the saved-object in both the source and target spaces. However, when a user edits a saved-object, they will only need to be authorized to write the saved-object in their current space. For example, in the following situation Bob is only authorized to edit dashboards and visualizations in the Marketing space.
Bob will therefore be able to edit the dashboards and visualizations which are shared in the Marketing and Sales spaces. However, once object-level security is implemented the user will be able to specify a per-object ACL to make the shared saved-object read-only.
Implementation Phases #
The ability to share a saved-objects in multiple spaces will be implemented in three primary phases. Not all saved-objects of the specific "category" will automatically be able to be shared, and this will require effort from application and solution teams before a saved-object can be shared. For example, with Phase 1 not all new saved-object types will automatically be "shareable".
Phase 1: New saved-object types without references
Release: 7.8 #54043
This will allow ML to begin taking advantage of the ability to share saved-objects in multiple spaces. ML jobs haven't transitioned to take advantage of saved-objects, and this will allow us to reduce the complexity that migrations introduce
Phase 1.5: Management
Release: 7.10 #58139
This will support import/export for "shareable" types (saved object types marked as
multiNamespace
) without references. It will also support existing operations in the Saved Objects Management Screen (server side and client side) and add a UI for sharing a saved object to other space(s).Ancillary task: Sharing to all spaces
Release: 7.10 #69808
As originally envisioned, sharing saved objects would only support explicitly-named spaces. However, recent use cases have surfaced that would benefit from the ability to share to all current and future spaces. This will support such use cases.
Phase 2: Existing saved-object types without references.
Release: 7.12 #54837
This will allow index-patterns to be shared in multiple-spaces. It requires that migrations and aliases are implemented, but doesn't require that the referential integrity checking be implemented.
Update: this feature is ready to merge in 7.11, but we decided to push it back to 7.12 to minimize risk of migration changes, which are already significantly changing in the 7.11 release.
Phase 2.5: Shared UI components
Release: 7.12 #85791
This will provide shared UI components for consumers to leverage when converting objects to multi=namespace types.
Phase 3: All saved-object types with references
Release: 7.14 #67380
Everything can now begin taking advantage of being shared in multiple spaces. As previously stated, given the level of effort that it will take to customize the various applications to inform the user that the saved-object is shared in multiple spaces and allow direct and transitive references to be shared, this will be taken advantage of incrementally by the application and solution teams.
Phase 3.5: UI for sharing with references
Release: 7.14 #100424
Phase 3 added server-side support for fetching an object and all of its references, and for sharing multiple objects. Due to the size of those changes, this phase was split into a separate PR. It includes: changing the UI for sharing an object with all of its references, updating the look and feel of the Share flyout to match a newer design mockup, and adding an additional step in the Share flyout to check for alias conflicts and prevent them from happening.
As of 2020/12/08: The original plan was to convert "index-pattern" saved objects to become shareable with the release of Phase 2, and allow plugins to convert other existing object types in a phased approach. It later became apparent that index-pattern objects cannot be converted this soon, because we need to account for inbound references (and some other object types do not use references using the standard method). There was some perceivable value to the goal of converting index-patterns first, but ultimately allowing some existing objects to be converted sooner than others is likely to cause user confusion and additional cognitive load. In the interest of simplifying things for the end-user, we decided to wait to convert all existing object types in a "big bang" release.
Hopefully this will be a minor release before 8.0, but that remains to be seen.Update: this conversion will happen in the 8.0 release. See the meta-issue #100489 for details.Technical Solution
When implementing Spaces, we prefixed a saved object’s Elasticsearch document ID with the
namespace
and added a singularnamespace
field. Prefixing the Elasticsearch document ID with thenamespace
allowed us to copy saved-objects from one Space to another without having to worry about conflicts, as the Elasticsearch document IDs are unique. Since the end-goal is to have saved-objects belong to multiplenamespaces
, we can no longer prefix the document ID with thenamespace
. The following technical solution expands upon the method in which we’re proposing to removenamespace
from the Elasticsearch document ID, change thenamespace
field tonamespaces
and maintain backwards compatibility with existing Kibana URLs.Brief Summary
This is the current document structure:
We’d like to remove the
NAMESPACE
, from the_id
and changenamespace
tonamespaces
. This would leave us with this new document structure:The issue is that we’re likely going to have quite a few
_id
collisions when doing so. Kibana currently supports the ability to have multiple saved-objects with the sameID
in different namespaces. For example:And
These two documents will exist if we copy an index-pattern from the sales space to the marketing space, or vice-versa. If we remove the
NAMESPACE
from the_id
for both of these documents, we’d end up with two documents with an_id
ofindex-pattern:1
.When we detect this collision during a migration, we have the option of regenerating anID
. However, we’ve been primarily concerned that doing so is going to break existing URLs in Kibana.To prevent us from breaking URLs in Kibana and address another pain-point which users have experienced, we’d like to introduce the ability for end-users to manage saved-object aliases. Saved-object aliases are used when translating from an identifier in an application’s front-end route to the referenced saved-object. Prior to this change, front-end routes would include the actual saved-object ID, for example:
http://localhost:5601/app/kibana#/dashboard/edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b
. However, with the introduction of saved-object aliases, we’d allow the user to create a custom alias which references the actual saved-object ID. This would allow a front-end route likehttp://localhost:5601/app/kibana#/dashboard/foo
to be used instead of the aforementioned route which includes the explicit saved-object ID.Now that we have saved-object aliases, when we detect a collision during the migration, we regenerate a new
ID
and create analias
document, similar to the following:We also need to keep track of the old and new IDs. For the sake of the migration itself, an in-memory map is sufficient; however, to better support imports when an ID had to be regenerated, we’ll be persisting these in a
regenerated-id
saved-object.This leaves us with the following documents to model the previous situation where we had an
index-pattern
withID
of1
in both thesales
andmarketing
namespaces
.Then, we no longer rely directly upon
SavedObjectsClient::get
when retrieving the saved object based on the identifier in the URL. Instead, we add a newSavedObjectsClient::resolve
, which will return a saved-object using thetype
andidentifier
. Theresolve
method will look for analias
which matches thetype
andidentifier
in addition to a normal saved-object which matches the type and identifier. If we find only analias
, we return the referenced saved-object. If we only find an actual saved-object, we return the actual saved-object. If we find both, we throw an error and require the user to remedy this using Saved Object management. This way, we don’t alter the behavior ofSavedObjectClient::get
, and it always returns saved-objects based on theirID
.Migration Pseudocode
The following assumes we’ve topologically sorted all saved objects by saved-object types:
SavedObjectsClient::resolve Pseudocode
SavedObjectsRepository Operations
The following logic changes will be implemented within the SavedObjectsRepository, which exists within OSS Kibana. Currently, the SavedObjectsRepository ensures that saved-objects in different namespaces are isolated from each other and no conflicts exist. This is by virtue of the
namespace
being part of the Elasticsearch document ID. Additionally, the SavedObjectsRepository ensures that saved-objects which are being retrieved using theget
,bulkGet
andfind
methods all respect thenamespace
that is specified, and saved-objects in other namespaces are not retrieved.With the changes being proposed, we can no longer ensure that saved-objects in different namespaces are always isolated from each other and no conflicts exist. Instead, we will be modifying the logic within the SavedObjectsRepository to reflect the new reality that conflicts may exist between saved-objects in namespaces.
Create and BulkCreate
namespace
id
isn’t specifiedid
is specified andoverwrite
isfalse
,id
is specified andoverwrite
istrue
GET
or_bulk_get
to retrieve any existing saved-objects which already existnamespace
is already in the saved-object’snamespaces
.409 Conflict
Update
namespace
.GET
the existing saved-objectnamespace
isn’t in thenamespaces
, we throw a409 Conflict
error_bulk_get
. If theirnamespaces
don't include the updated saved-objectsnamespaces
, throw an error.Get and BulkGet
get
orbulkGet
Find
namespace
tonamespaces
Delete
GET
the existing saved-objectnamespace
, delete the saved-object404 Not Found
DeleteByNamespace
Note: This is the internal method which is used to delete a space and all of the saved-objects in that space. It likely should be renamed based on the changes in logic I'm proposing.
_delete_by_query
for all saved-objects which only exist in the currentnamespace
_update_by_query
to remove the currentnamespace
from all saved-objects which exist in multiple namespacesOther Saved Object Operations
Export
Import
Note: The following is a gross over-simplification and needs further thought and investigation
regenerated-id
saved-objects and use the new IDbulkCreate
returns a conflict, there are now multiple remediationsoverwrite
falseregenerated-id
saved-objectregenerated-id
saved-object. There could be multiple, if we let there be...The text was updated successfully, but these errors were encountered: