[Archive Migration] batch 2 of removing es_archives/empty_kibana

[LeeDr]

Summary

Part of Meta issue: #102552

This PR removes the use of es_archives/empty_kibana and replaced it with calls to kibanaServer.savedObjects.cleanStandardList. Similar to this PR #138189

WARNING: I should note that there is a difference between
esArchiver.load('x-pack/test/functional/es_archives/empty_kibana'); (which I think deletes the entire .kibana index) and
kibanaServer.savedObjects.cleanStandardList(); (which can only delete savedObjects of types supported by the Saved Object API in the DEFAULT SPACE). There might be some types of saved objects which are not deleted. The full set we delete are here https://github.com/elastic/kibana/blob/main/packages/kbn-test/src/kbn_client/kbn_client_saved_objects.ts#L225 And saved objects in other spaces wouldn't be deleted. I mostly used find/replace to generate this PR and didn't review all the tests in detail. But the tests passed! The fact that our tests are broken up into so many separate configs probably helps with the isolation of tests.
I find that generally tests that created a space also already had an after method that deletes the space (and everything in it).

[LeeDr]

There seem to be some tests here which are creating spaces and not cleaning them up at the end, which causes a failure when the next test tries to create a space which already exists. I don't know these tests but it seems like these should be matching pairs of setupSpacesAndUsers and tearDown? This could be a case where es_archives/empty_kibana would have wiped out everything in the .kibana index including spaces but cleanStandardList doesn't do that.

main/kibana - (remove_empty_kibana_2) > grep -r setupSpacesAndUsers x-pack/test/alerting_api_integration/security_and_spaces/group2/* | wc -l
      13
main/kibana - (remove_empty_kibana_2) > 
main/kibana - (remove_empty_kibana_2) > grep -r tearDown x-pack/test/alerting_api_integration/security_and_spaces/group2/* | wc -l           
       7

[LeeDr]

@elasticmachine merge upstream

[kibanamachine]

merge conflict between base and head

[LeeDr]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Fleet changes

[LeeDr]

I just want to call this out for reviewers. The issue here is that some tests are still unloading an es_archive which contains spaces. This method was failing if trying to delete a space which has already been removed by unloading the es_archive. It appears that this is primarily used as a tearDown step and not any tests.

[wayneseymour]

Yeah my eyebrow raised on this one at first, but I concur. Good call.

[pmuellr]

ResponseOps changes LGTM

[pmuellr]

Though the ResponseOps changes LGTM, I peeked at cleanStandardList() implementation, and we may have a problem with this:

kibana/packages/kbn-test/src/kbn_client/kbn_client_saved_objects.ts

Lines 223 to 230 in 81d7a6f

	public async cleanStandardList(options?: { space?: string }) {
	// add types here
	const types = [
	'url',
	'index-pattern',
	'action',
	'query',
	'alert',

You're not supposed to delete alerting rule SO's (type: alert) directly, as we have task SO's (in the task manager index) that are closely associated with the rule SO's. If you only delete the rule SO, but not the task manager SO, the rule will still attempt to run, as that determination is from the task manager SO, but the rule will fail since there is no rule SO.

While this is likely quite survivable today, since typically nothing bad happens except for excessive logging, we have seen some scenarios in the field of this affecting task manager health (it starts to complain about the failure rate if you delete a lot like this), so it would really be best NOT to delete the rule SO's, just by themselves. It will also cause these "zombie" tasks to delay execution of rules that are currently under test, generally making them run longer, or worse case fail with their own sorts of "timeouts".

I feel like perhaps we can have a way to do alternate delete processing for some of the types, so instead of deleting the rule SO via SO APIs, we'd delete it via the alerting APIs, which delete both SO docs.

[LeeDr]

I feel like perhaps we can have a way to do alternate delete processing for some of the types, so instead of deleting the rule SO via SO APIs, we'd delete it via the alerting APIs, which delete both SO docs.

If rules are defined within a space, does deleting the space take care of the task manager index SOs?

For rules not in a space, it sounds like we would have to use api/alerting/rules/_find and then DELETE api/alerting/rule/ for each id that was found. And it could be paginated... Is there already a cleanup method somewhere in the repo for that?

[pheyos]

ML changes LGTM

[efegurkan]

Enterprise Search changes look good.

[tsullivan]

LGTM

[parkiino]

onboarding lifecycle management changes lgtm

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: bc026ff

Metrics [docs]

✅ unchanged

History

• 💛 Build #65163 was flaky 4a38e01
• 💚 Build #65109 succeeded 4c85217
• 💔 Build #64972 failed 7733c90
• 💔 Build #64942 failed 5f3d706
• 💔 Build #64678 failed d51b4d5
• 💔 Build #64109 failed 4c2e16c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation