[ResponseOps] Alert creation delay based on user definition

[doakalexi]

Resolves #173009

Summary

This PR:

• Changes the field name from notification_delay to alert_delay
• Updates the alerts client and rule registry to index new alert docs on a delay
• Updates the framework code to delay the creation of an alert

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

To verify

• Use Dev Tools to create a rule with the alertDelay

POST kbn:/api/alerting/rule
{
  "params": {
    "searchType": "esQuery",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [
      -1
    ],
    "thresholdComparator": ">",
    "size": 100,
    "esQuery": """{
    "query":{
      "match_all" : {}
    }
  }""",
    "aggType": "count",
    "groupBy": "all",
    "termSize": 5,
    "excludeHitsFromPreviousRun": false,
    "sourceFields": [],
    "index": [
      ".kibana-event-log*"
    ],
    "timeField": "@timestamp"
  },
  "consumer": "stackAlerts",
  "schedule": {
    "interval": "1m"
  },
  "tags": [],
  "name": "test",
  "rule_type_id": ".es-query",
  "actions": [
    {
      "group": "query matched",
      "id": "${ACTION_ID}",
      "params": {
        "level": "info",
        "message": """Elasticsearch query rule '{{rule.name}}' is active:

- Value: {{context.value}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}"""
      },
      "frequency": {
        "notify_when": "onActionGroupChange",
        "throttle": null,
        "summary": false
      }
    }
  ],
  "alert_delay": {
    "active": 3
  }
}

• Verify that the alert will not be created until it has matched the delay threshold.
• Verify that the delay does not affect recovered alerts

[doakalexi]

/ci

[doakalexi]

/ci

[doakalexi]

/ci

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

I guess it's safe to assume that no rules will have been created with the notificationDelay config in serverless since it was API only? Otherwise they will be skipped due to validation error

[2024-02-01T11:12:21.408-05:00][WARN ][plugins.taskManager] Task (alerting:.es-query/85750824-12ec-4954-acfb-ec9b65899fc0) has a validation error in its indirect params: [notificationDelay]: definition for this key is missing

[doakalexi]

I guess it's safe to assume that no rules will have been created with the notificationDelay config in serverless since it was API only? Otherwise they will be skipped due to validation error

[2024-02-01T11:12:21.408-05:00][WARN ][plugins.taskManager] Task (alerting:.es-query/85750824-12ec-4954-acfb-ec9b65899fc0) has a validation error in its indirect params: [notificationDelay]: definition for this key is missing

Yes, it was API only. I don't think anyone should have added a notification delay in this case.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: c3fdfec
• Storybooks Preview

Failed CI Steps

• AI Assistant - Security Solution Cypress Tests #1
• Explore - Security Solution Cypress Tests #2

Metrics [docs]

✅ unchanged

History

• 💚 Build #190627 succeeded ec1d0d9
• 💔 Build #190606 failed 26b6a56
• 💛 Build #190538 was flaky 7082d03
• 💚 Build #190323 succeeded 134fae2
• 💔 Build #190312 failed 468f94d
• 💔 Build #190067 failed 9f3575a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[ymao1]

LGTM! Verified using an ES query rule type and a custom threshold rule type and alert delay works as described. Also tried upgrading to this PR branch from main with existing rules and verified that they continue running successfully.