[Saved Objects] Support type changing and document splitting

[jonathan-buttner]

While designing a new feature for our Cases plugin I came up with a use case to change a saved object's type field during a migration and/or splitting an existing saved object into multiple saved objects. Here is the object I'm referring to: https://github.com/elastic/kibana/blob/master/x-pack/plugins/case/server/saved_object_types/cases.ts#L12

For my specific use case I was thinking of extracting some of the fields that are more inline with a metadata saved object like description, title, tags, etc into a new saved object and then as part of the migration removing those fields from the existing case saved object. It'd be great if during a kibana upgrade we could use the saved object migration functionality to split a document into multiple documents and even change a saved object's name.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

@rudolf I think we may already have an open issue for this one, but I can't find it. Do you have more luck that me?

[rudolf]

We have #26602 but it was closed since it was no longer a requirement.

[rudolf]

Because migrations need to be deterministic we will have to ensure that the split documents always get the same id's on all the Kibana nodes that connects to the same Elasticsearch instance. We could generate a UUIDv5 from the UUID of the object that's being split, the migration version and a counter (effectively SHA1(sourceId + migrationVersion + counter)).

@jonathan-buttner While splitting documents during a migration makes sense in general, can you elaborate on the reason for wanting to split the case metadata into it's own type? I think denormalizing data by keeping it in a single object usually simplifies the implementation, then you don't need to load two objects when doing a read and you also don't need to remember to delete both objects when doing a delete.

[jonathan-buttner]

Because migrations need to be deterministic we will have to ensure that the split documents always get the same id's on all the Kibana nodes that connects to the same Elasticsearch instance. We could generate a UUIDv5 from the UUID of the object that's being split, the migration version and a counter (effectively SHA1(sourceId + migrationVersion + counter)).

Ah I see, I didn't realize that the UUID needs to be the same on all Kibana nodes. That makes sense.

@jonathan-buttner While splitting documents during a migration makes sense in general, can you elaborate on the reason for wanting to split the case metadata into it's own type? I think denormalizing data by keeping it in a single object usually simplifies the implementation, then you don't need to load two objects when doing a read and you also don't need to remember to delete both objects when doing a delete.

Yeah good point about the denormalizing. The reason we were considering splitting the metadata into its own type is because we want to support a one-to-many relationship between the base metadata information (title, tags, connector fields) and user specific information (status, closed by, closed at) in a case. The use case is that a user can attach the results of a detection rule firing. When a rule fires, signals are generated and we want to add that information to a case automatically. We thought it would be helpful if we could group all the cases generated by the same detection rule together in some fashion.

I guess another option is we could add a parent field to the case saved object as a pointer back to the root case created when the detection rule is initially created. This field would be unpopulated when a case is created outside of a detection rule. This would avoid having to create a new type of saved object but would duplicate a lot of information between the cases.

We have more details in this issue: https://github.com/elastic/security-team/issues/705

[rudolf]

If the "metadata" doesn't change often, it might be simpler to have it duplicated, but I think you're probably in the best position to make this design tradeoff.

Splitting an object during migrations is technically possible and not a huge amount of effort for us, but it is a risky change. So if you choose a design that requires this we'll just need to discuss timelines.

Changing a saved object type feels a bit more complex and I'm not sure about the effort to make this possible. If e.g. cases is renamed into case we will have to apply the cases migration, and then apply any migrations for the case type.

[joshdover]

For my specific use case I was thinking of extracting some of the fields that are more inline with a metadata saved object like description, title, tags, etc into a new saved object and then as part of the migration removing those fields from the existing case saved object.

Would like to point out that splitting up objects some other implications, specifically around SO exports and Object-Level Security (OLS) authorization (#39259).

For exports, you will need to implement the onExport hook (#87807) in order to include the metadata objects when cases are exported. I believe this is already the situation case-comments though, so probably not a whole lot of extra work.

We have not yet settled on how privilege inheritance will work with OLS, but this will need to be considered with these metadata objects (as well as for case-comments).

@jonathan-buttner do you have an idea of when this would be needed? How necessary is this change? Just need to know for planning the Core roadmap.

[rudolf]

Changing a saved object type feels a bit more complex and I'm not sure about the effort to make this possible. If e.g. cases is renamed into case we will have to apply the cases migration, and then apply any migrations for the case type.

This seems to be implemented in the code already, but we don't have integration tests verifying this behaviour. So it's potentially as easy as just adding more tests.

[rudolf]

I'm closing this issue as this is no longer a requirement for the cases plugin. The problem and proposed solution still make sense, so should this become a blocker for anyone feel free to re-open.