Research approach for RAC when making rules and connectors share-capable

[chrisronline]

Related to #100067

In 8.0, we plan to run a migration that removes the namespace from the saved-object serialized ID. To do so, new rule and connector IDs will be generated. With this issue, we should research the impacts this will have on the RAC "alerts as data" when documents reference old rule or connector IDs, draft a proposal and work / discuss with the @elastic/kibana-security if we need anything further from them to support this.

This work is related to #100666 so we should be able to share/discuss options and workflows together.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Is it already possible to reference rules and connectors using the new SO ids (which don't include namespace)?

Since alerts-as-data documents are not released yet (so these are not existing objects which would require some sort of migration), what we could do is we could use the new SO ids when writing alerts, so there’s nothing to break and migrate in the future (data-wise). We would need to be able to:

• Get new rule ids by their existing ids for N rules. Examples:
  • When writing alerts from the executor function, we know the rule with its existing id; we need to get the new id and store it in a property of each alert object.
  • When fetching alerts of 1-N rules by their existing ids, we need to get N new ids by N existing ids, and then filter alerts-as-data indices by the new ids.
• Get old rule ids by their new ids for N rules (vice versa). Examples:
  • When displaying an alert (which includes a new id of the rule which produced it), in order to find its rule, we need to get its old id by the new one.
  • When aggregating over alerts containing new ids, we find the new ids which we convert to the old ones to fetch the rules.

[jportner]

Is it already possible to reference rules and connectors using the new SO ids (which don't include namespace)?

Unfortunately this is not possible. We plan to use uuidv5 to deterministically regenerate existing object IDs in the 8.0 release:

kibana/src/core/server/saved_objects/migrations/core/document_migrator.ts

Line 856 in 0561648

return uuidv5(`${namespace}:${type}:${id}`, uuidv5.DNS); // the uuidv5 namespace constant (uuidv5.DNS) is arbitrary

However, that uses SHA1 hashing, there is no way to invert that.

---

Edited response below for clarity

If you have a regular ES document that needs to store a link to a saved object, I see two options.

Option 1 (preferable)

1. Store the saved object ID in your regular ES document
2. Later, fetch the ES document in Kibana
3. Call the SavedObjectsClient.resolve() API with the ID to fetch the actual saved object (this will work with its old ID and its new ID)
4. Return the saved object

Option 2

If for some reason Option 1 is not tenable, you could predict what the new saved object ID will be, and start recording both the old (current) legacyId and the new id. You could use the legacyId for the duration of 7.x, then you could flip a switch in 8.0 to start using the new id instead.

Note: just to be 100% clear, it is not necessary to do this if you have one saved object linked to another saved object. If this is the case, you just need to store one saved object ID in the other saved object's root-level references field, and the ID will automatically be updated in 8.0.

[chrisronline]

@banderror Is it possible to use the references part of saved objects to build out what you need? Like @jportner said, that will give you the best forward-looking compatibility.

[gmmorris]

Is it already possible to reference rules and connectors using the new SO ids (which don't include namespace)?

Unfortunately this is not possible. We plan to use uuidv5 to deterministically regenerate existing object IDs in the 8.0 release:

kibana/src/core/server/saved_objects/migrations/core/document_migrator.ts

Line 856 in 0561648

return uuidv5(`${namespace}:${type}:${id}`, uuidv5.DNS); // the uuidv5 namespace constant (uuidv5.DNS) is arbitrary

However, that uses SHA1 hashing, there is no way to invert that.
Edited response below for clarity

If you have a regular ES document that needs to store a link to a saved object, I see two options.

Option 1 (preferable)

1. Store the saved object ID in your regular ES document
2. Later, fetch the ES document in Kibana
3. Call the SavedObjectsClient.resolve() API with the ID to fetch the actual saved object (this will work with its old ID and its new ID)
4. Return the saved object

Option 2

If for some reason Option 1 is not tenable, you could predict what the new saved object ID will be, and start recording both the old (current) legacyId and the new id. You could use the legacyId for the duration of 7.x, then you could flip a switch in 8.0 to start using the new id instead.

Note: just to be 100% clear, it is not necessary to do this if you have one saved object linked to another saved object. If this is the case, you just need to store one saved object ID in the other saved object's root-level references field, and the ID will automatically be updated in 8.0.

Thanks @jportner , I think some of the confusion comes from a lack of familiarity across the domains in question.

We're not talking about SOs here, just ES documents, but these could end up being huge indices, potentially millions of documents in ES by the time a customer upgrades from 7.16.x to 8.x.
For that reason, anything requiring a migration is not an option and so I don't think option 1 would work.

Option 2 could work for alerts-as-data I think, right @banderror ?
Could Kibana Security provide a service that can hand the Alerts-as-data service the predicted ID? We'd ideally have the same mechanism provide the predicted ID and the migrated one to avoid the algorithms diverting somehow in the future.

[banderror]

Sorry for missing your replies @jportner @chrisronline

@gmmorris is right, we're talking about slightly different things here.

RAC "alerts as data" documents are not Saved Objects, they are regular ES documents stored in separate indices. There can be potentially millions of them. They will contain fields like (rule.id and kibana.alert.rule.id) which are in fact "foreign keys" pointing to the corresponding Saved Objects. Right now they store existing ("legacy") SO ids. These indices are not released yet, but will likely be released in 7.15.

We will have exactly the same problem with RAC "evaluation" documents (same concept, same schema, different indices, not released yet).

And we will have similar problem with documents stored in event_log, with the difference that these documents are already released.

@banderror Is it possible to use the references part of saved objects to build out what you need? Like @jportner said, that will give you the best forward-looking compatibility.

So, unfortunately no, because they are not SOs.

...potentially millions of documents in ES by the time a customer upgrades from 7.16.x to 8.x. For that reason, anything requiring a migration is not an option

Yes, absolutely, I agree with that. We should avoid data migration, and instead prepare our code and RAC documents for this change beforehand.

I tried to propose a way to handle that (#107161 (comment)), maybe articulated not very clearly... What I meant is what Gidi formulated better than me:

Could Kibana Security provide a service that can hand the Alerts-as-data service the predicted ID? We'd ideally have the same mechanism provide the predicted ID and the migrated one to avoid the algorithms diverting somehow in the future.

Right now (ideally before 7.15 FF), we'd need an API exposed from Kibana Core that would return:

• an array of new ("predicted") SO ids given an array of old ("legacy") ids (I guess this would be a must have)
• and vice versa (I guess we'd need it if we don't include legacyId in the documents)

We shouldn't need to fetch those SOs to use this API.

Do we have such an API or is it possible to create and expose it?

Option 2 could work for alerts-as-data I think, right @banderror ?

I think so, sounds good to me. I was thinking to store only new ("predicted") SO ids in rule.id and kibana.alert.rule.id, but storing both the new id and the old ("legacy") id in an additional field and then querying by this field until 8.0 makes sense to me as well.

It's important to check the details though within the RAC group.

[gmmorris]

Brilliant, thanks @banderror

I think so, sounds good to me. I was thinking to store only new ("predicted") SO ids in rule.id and kibana.alert.rule.id, but storing both the new id and the old ("legacy") id in an additional field and then querying by this field until 8.0 makes sense to me as well.

You'll need both I think as the new IDs won't kick in until 8.0

[gmmorris]

We're looking into this problem in Event Log though and I think we'll solve it by querying for both the new and the old IDs when querying the Event Log.
cc @pmuellr

[jportner]

Right now (ideally before 7.15 FF), we'd need an API exposed from Kibana Core that would return:

• an array of new ("predicted") SO ids given an array of old ("legacy") ids (I guess this would be a must have)
• and vice versa (I guess we'd need it if we don't include legacyId in the documents)

We shouldn't need to fetch those SOs to use this API.

Do we have such an API or is it possible to create and expose it?

OK, there have been a couple of other use cases for such a Core API, I think it's quite reasonable and I can have a PR up for it today.
It's worth noting that there is no way to find a legacy ID given a new ID because we rely on uuidv5 to regenerate object IDs (that uses SHA-1 under the hood, it's a one-way hash).

For the API you'll need three inputs:

1. Object namespace (space ID)
2. Object type
3. Object ID

I'll reply here when I have a PR up.
Edit: opened an issue to track, #107744

[jasonrhodes]

I like the idea of storing both, with the legacy one clearly indicated as legacy in its field name(s)

It would be fantastic if whatever we are using to get the current ID to store could also just return both IDs so we can index both of them, and not worry too much about making a separate call for this. But having an API to give us the new ID based on these criteria is a huge step in the right direction, so thank you for moving on that.

[marshallmain]

The original issue regarding sharing saved objects describes an "alias" document that maps an old ID to a new ID. Is that still going to be built as part of the SO ID migration? It seems like a very important feature to be able to convert back and forth between old and new SO IDs.

If so, we could use that in 8.0+ to find the old ID given a new ID and build queries that search for both. We'd probably also want to store an additional version number on the alert next SO ID is legacy or 8.0+.

[jportner]

The original issue regarding sharing saved objects describes an "alias" document that maps an old ID to a new ID. Is that still going to be built as part of the SO ID migration?

Yes, there is a "legacy-url-alias" saved object type ("alias" for short). Each time an object is converted, if its ID changes, a new alias will be created that contains the old ID with a pointer to the new ID.

It seems like a very important feature to be able to convert back and forth between old and new SO IDs.

If so, we could use that in 8.0+ to find the old ID given a new ID and build queries that search for both. We'd probably also want to store an additional version number on the alert next SO ID is legacy or 8.0+.

We really, really don't want consumers relying on the old object IDs anymore. And we really don't want to support aliases long-term. There is a significant amount of technical debt that we have introduced to convert these saved object types and add/support aliases, and we want to eventually be able to pay it down.

Legacy URL aliases are only intended to be used internally by the SavedObjectsClient.resolve() function. We collect a bunch of usage data to measure how often this is used and how often the different outcomes are encountered. The ultimate goal is so that we can eventually identify when it is acceptable to remove aliases completely.

[jportner]

I'll reply here when I have a PR up.

PR: #107767

[gmmorris]

thanks for the quick turn around @jportner :elasticheart: