[Security Solution] Ensure alerts are scheduled when rule times out

[madirey]

Summary

Fixes: #121559

Related:
#120506

NOTE: Tests are very difficult to write for this case. There are tests in the alerting framework for the constructs used. Below is the method I used for manual testing, in detail.

To test:

1. Start an http server on localhost:
   sudo python3 -m http.server 5605 or use netcat.

2. Create a rule that creates alerts and then times out. Easiest way I have found to accomplish this is below:

If you create more than 2 alerts, the first 2 alerts will be created and then the task will stall, and eventually time out.

3. Check rule status after time out. You should see alerts created and rule notifications should fire for those 2 alerts (in example below, POST is not supported by the simple Python webserver, but you can see that the notification action did fire)

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[madirey]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 8d93216
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

History

• 💔 Build #33054 failed eed917f
• 💔 Build #32680 failed 9af5b64

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[marshallmain]

The changes here LGTM as is. There are some other areas we may want to address re: cancellation in addition though.

We should remove the experimentalFeatures.securityRulesCancelEnabled flag and choose an appropriate value (possibly undefined) for ruleTaskTimeout based on the outcome of https://github.com/elastic/security-team/issues/3415. This can be a follow up PR but we should target 8.2 for that work if possible.

Threat match rules have a separate function buildExecutionIntervalValidator that implements timeout functionality as well. If we can remove that and rely on the search timeout instead that would unify the rule types more. We'll need to modify some of the logic in the threat match executor so it breaks out of the loop when searchAfterBulkCreate returns an error though.

[marshallmain]

If we try to schedule throttled notifications after a rule is cancelled, the search executed inside this function will also be cancelled and we won't be able to schedule the actions. We may need the alerting framework to provide a secondary "un-cancellable" client that we can use during the actions scheduling process.

[madirey]

Ah, good point!

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 128276 or prevent reminders by adding the backport:skip label.