Reduce the impact of legacy-url-aliases introduced by sharing to multiple spaces

[rudolf]

To enable sharing to multiple spaces #27004 the id's of saved objects had to be changed. A new saved object type, legacy-url-aliases was introduced to ensure that URLs to these old ids still work even though the underlying saved object's id was changed.

This means when users upgrade from 7.x to 8.x the amount of saved objects nearly doubles. This has the following impact:

• reduces the performance of saved objects APIs Investigate risk of performance regression from share-capable saved object types #113743
• severely impacts the performance of saved objects bulkDelete Reduce performance impact of deleting legacy url aliases #141136
• theoretically doubles the downtime during upgrade migrations Scale upgrade migrations to millions of saved objects #144035

This issue is to discuss ways to reduce the impact and eventually remove these legacy-url-aliases.

As a first step we should make it easy for users to delete legacy-url-aliases that aren't used. We could allow them to delete legacy-url-aliases that have never been used, or haven't been used in the last X months.

For the long term health of Kibana and our users it feels like we should deprecate these URLs and:

• automatically delete legacy-url-aliases that haven't been used for e.g. 12 months
• after X (3?) years remove support for them completely

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[lukeelmers]

For the long term health of Kibana and our users it feels like we should deprecate these URLs and:

• automatically delete legacy-url-aliases that haven't been used for e.g. 12 months
• after X (3?) years remove support for them completely

The question of how long legacy url aliases would need to stay around was never resolved -- even in the initial implementation, we knew it was an open question. But now that we have run into some real-world scenarios where it can cause scaling issues (specifically for users who are running upgrades with millions of saved objects), I think it's worth revisiting.

Part of this discussion is around how much notice is reasonable to give users before breaking URLs, something we have historically gone to great lengths to avoid doing.

And the other part of the discussion is more of a product-focused one: what type of ux do we envision to streamline removal of unused legacy aliases? Do we build a tool like Rudolf suggests for identifying unused aliases and removing them? Do we give people a choice of performing a "cleanup" of legacy aliases, while explaining the tradeoffs of doing so? cc @arisonl @alexh97

[legrego]

It's worth noting that legacy url aliases have unfortunately found a benefit in the post-8.0 era. They were originally intended to preserve bookmarks, but they also serve as a way to support "weak links" from one SO to another when these assets are imported or copied between spaces. Example: A markdown visualization within a dashboard which links to another dashboard.

• Importing saved objects can break "weak links" #123550
• [Saved objects] Create URL aliases during "Copy on save" #176552

My hope was that we'd be able to remove these legacy url aliases sooner rather than later, but as of now it's the only tool we have to support weak links for the import & copy-to-space use cases.

[rudolf]

Flagging this with breaking. Even if it does not have to be a traditional breaking change changing the behaviour to e.g. delete unused aliases feels like the kind of thing we want to be sure users understand and probably better to do in a major.

[pgayvallet]

We discussed about this issue during our platform services meeting, and we came with the idea/proposal of enabling the feature behind a configuration flag.

The reasoning was that:

1. This proposal's main purpose was to address issues around the volume of saved objects in our system indices, and this is no longer really a risk for most of our customers, so strictly speaking, the enhancement don't really feels mandatory anymore.
2. By letting customers choose if they want to enable this, we still have an escape hatch for the few customers this may be an issue with, but at the same time we avoid potentially breaking things for customers that didn't really need / want this cleanup in the first place.
3. By having this enabled only via a configuration flag, we dodge the breaking change label (and are even free to implement the enhancement whenever we want, not precisely for 9.0)

Functionally, it would be about adding two config settings (one to enable the feature, and one to define the retention period), and then implement the corresponding cleanup task / logic.

cc @rudolf (well, when you're back) does that make sense to you?