Research approach for event log when making rules and connectors share-capable

[mikecote]

Event log research issue for #100067.

In 8.0, we plan to run a migration that removes the namespace from the saved-object serialized ID. To do so, new rule and connector IDs will be generated. With this issue, we should research the impacts this will have on the event log when documents reference old rule or connector IDs, draft a proposal and work / discuss with the @elastic/kibana-security if we need anything further from them to support this.

Note: We should also consider the data indices from RAC when coming up with a proposal. The implementation may be similar where one saved object relates to one or many documents. Here is a ticket for their work: #107161

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

cc @tsg @spong @XavierM - wanted to give you a heads up on an 8.0 change to the rule / connector saved objects. The main issue is #100067, but we've opened this issue to track how other references to these SO's will be affected. I assume that's at least cases and the new alerts-as-data indices.

[mikecote]

Marking as blocked until the research in #100067 confirms this is still necessary and with what requirements.

[chrisronline]

The event log stores saved object ids in two ways:

• Structured through a specific field
• Unstructured through the message field

Since the event log is probably our largest index, it doesn't seem feasible to do anything about the unstructured usage, as attempting to read each document individually, parse the ids out of the string, run the .resolve() API against it, then update the document will be too expensive at scale.

The structured usage is more feasible, but still doesn't seem possible at scale to run a single sweep migration across the entire index. The update_by_query api is a nice option to perform bulk updates, but I'm not sure we can use it as we'd need to know the resolved ID before running the query which we wouldn't know without running the .resolve() (It's worth noting that it might be possible to predetermine the resolved id but the calculation is most likely not possible in a painless script).

Now, assuming it's not feasible to mass update/migrate the entire event log at startup, we might have other options.

Outside of any external index searching, the only use case for the index is three separate internal calls to a single API endpoint:

• https://github.com/elastic/kibana/blob/master/x-pack/plugins/alerting/server/rules_client/rules_client.ts#L455
• https://github.com/elastic/kibana/blob/master/x-pack/plugins/event_log/server/routes/find_by_ids.ts#L55
• https://github.com/elastic/kibana/blob/master/x-pack/plugins/event_log/server/routes/find.ts#L49

It seems feasible to ensure those calls still work, even if provided an unresolved saved object id.

The RulesClient usage is called from this endpoint which is called from our rules UI. It's worth noting that once #107070 is done, any page loads with unresolved saved objects ids (typically from a bookmark) will properly resolve to the new saved object id and it seems the UI uses the resolved id instead of the route parameter which means we may not need to even change that code path.

Overall, it seems we should focus on resolving structured ids at read time versus performing any kind of bulk migration of the entire index. If we wanted to pursue that though, perhaps it could be in the form of a one-off script to run versus anything that runs in Kibana automatically.

For future writes, I don't we need to do much because #107070 should entail converting all .get() calls to use .resolve(), as well as finding all persisted unresolved saved objects ids in other documents and properly resolving them (like done in the PoC PR)

[chrisronline]

Note: When resolving the saved object from the structured location, be sure to use the namespace field appropriately

[chrisronline]

As a result of the research, I opened:

• [Event log] Update findEventsBySavedObjectIds to resolve SO ids #107423
• [Event log] Update public API routes to resolve SO ids #107424

[pmuellr]

I just happened to think - I wonder if we would want to resolve the id's of the SO's that are being returned from the event log queries. For example, a query could return events for a single rule, whose id changed over time. So the results would contain different SO id values, but they're really the same rule.

Seems like a nice-to-have, if nothing else, just to provide a consistent response. But also sounds expensive, since we'd have to resolve each SO id available in the query results.

So currently thinking no, we don't want to do that now - but we probably want to document this behavior in the event log "docs".

[chrisronline]

Thanks for the thoughts @pmuellr.

Do you think we should close both issues then as not something we're going to do? Or do you think there is any value in resolving the ids for the findEventsBySavedObjectIds use case?

[chrisronline]

I'm reopening this as the original research needs to be adjusted based on some recent conversations between our team and the security team.

[chrisronline]

We recently opened #108077 to address concerns around using the event log with both legacy and migrated saved object IDs. As a result, I don't know if we need to do anything here as a result. The event log, as is, should work if both sets of IDs are provided. I don't think we're providing much support for resolving IDs in public HTTP APIs so I don't think there is any work there either.

@pmuellr WDYT?

[pmuellr]

I think #108077 covers the event log aspects, so ya, think we're done.

[chrisronline]

++!