[ResponseOps] retry rule runs when retryable errors occur

[pmuellr]

Currently when a rule executor throws an error, we don't retry the execution. However, there are cases where this would probably be a good thing to enable. An example is a transient networking error like a Socket Hang Up. If we know the rule is "safe" to run again, and detect a transient error, a retry after a short delay has a good chance of executing successfully.

Lots of things to consider:

• is the rule writing data anywhere, such that a retry would overwrite that data or otherwise result in confusing / corrupt data being written
• is the rule sensitive to it's interval, runAt time, etc, such that it would calculate inappropriate time intervals on a retry

Probably also applies to the ES index connector, but ... harder, as by definition it is writing data - we don't want to write it twice! It's also the case that we could instrument the ES index connector itself for retry support, and not have to worry about it at the "connector level".

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[mikecote]

Linking with #50215 (comment) for running rules ad-hoc. Once one of these pieces of work is done, we should be able to automatically enable the other issue using the same solution..