Alerts that timeout during execution don't respect their schedule

[mikecote]

I did some local testing today to see how alerts behave after switching over to use Task Manager's schedule. I noticed when testing the timeout behaviour that the task is scheduled to run immediately after timing out when it should run at the next schedule (see #39349).

I did the following changes locally to reproduce this issue in a timely manner mikecote@43683af. The main pieces are to get an alert executor to timeout and to have an alert schedule much larger than the timeout window. That way you can see the next execution happen right after timing out.

To run my example you can do the following:

node scripts/functional_tests_server.js --config=test/alerting_api_integration/security_and_spaces/config.ts
node scripts/functional_test_runner.js --config=test/alerting_api_integration/security_and_spaces/config.ts

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

It seems like the retryAt should be the next scheduled run or the timeout window (whichever is greater) but that may break the timeout logic..