[Alerting][Docs] Adds Alerting & Task Manager Scalability Guidance & Health Monitoring

docs/api/task-manager/health.asciidoc

@@ -0,0 +1,133 @@
+[[task-manager-api-health]]
+=== Get Task Manager health API
+++++
+<titleabbrev>Get Task Manager health</titleabbrev>
+++++
+
+Retrieve the health status of the {kib} Task Manager.
+
+[[task-manager-api-health-request]]
+==== Request
+
+`GET <kibana host>:<port>/api/task_manager/_health`
+
+[[task-manager-api-health-codes]]
+==== Response code
+
+`200`::
+    Indicates a successful call.
+
+[[task-manager-api-health-example]]
+==== Example
+
+Retrieve the health status of the {kib} Task Manager:
+
+[source,sh]
+--------------------------------------------------
+$ curl -X GET api/task_manager/_health
+--------------------------------------------------
+// KIBANA
+
+The API returns the following:
+
+[source,sh]
+--------------------------------------------------
+{
+  "id": "15415ecf-cdb0-4fef-950a-f824bd277fe4",
+  "timestamp": "2021-02-16T11:38:10.077Z",
+  "status": "OK",
+  "last_update": "2021-02-16T11:38:09.934Z",
+  "stats": {
+    "configuration": {
+      "timestamp": "2021-02-16T11:29:05.055Z",
+      "value": {
+        "request_capacity": 1000,
+        "max_poll_inactivity_cycles": 10,
+        "monitored_aggregated_stats_refresh_rate": 60000,
+        "monitored_stats_running_average_window": 50,
+        "monitored_task_execution_thresholds": {
+          "default": {
+            "error_threshold": 90,
+            "warn_threshold": 80
+          },
+          "custom": {}
+        },
+        "poll_interval": 3000,
+        "max_workers": 10
+      },
+      "status": "OK"
+    },
+    "runtime": {
+      "timestamp": "2021-02-16T11:38:09.934Z",
+      "value": {
+        "polling": {
+          "last_successful_poll": "2021-02-16T11:38:09.934Z",
+          "last_polling_delay": "2021-02-16T11:29:05.053Z",
+          "duration": {
+            "p50": 0,
+            "p90": 0,
+            "p95": 0,
+            "p99": 0
+          },
+          "claim_conflicts": {
+            "p50": 0,
+            "p90": 0,
+            "p95": 0,
+            "p99": 0
+          },
+          "claim_mismatches": {
+            "p50": 0,
+            "p90": 0,
+            "p95": 0,
+            "p99": 0
+          },
+          "result_frequency_percent_as_number": {
+            "Failed": 0,
+            "NoAvailableWorkers": 0,
+            "NoTasksClaimed": 0,
+            "RanOutOfCapacity": 0,
+            "RunningAtCapacity": 0,
+            "PoolFilled": 0
+          }
+        },
+        "drift": {
+          "p50": 0,
+          "p90": 0,
+          "p95": 0,
+          "p99": 0
+        },
+        "load": {
+          "p50": 0,
+          "p90": 0,
+          "p95": 0,
+          "p99": 0
+        },
+        "execution": {
+          "duration": {},
+          "result_frequency_percent_as_number": {}
+        }
+      },
+      "status": "OK"
+    },
+    "workload": {
+      "timestamp": "2021-02-16T11:38:05.826Z",
+      "value": {
+        "count": 26,
+        "task_types": {},
+        "schedule": [],
+        "overdue": 0,
+        "estimated_schedule_density": []
+      },
+      "status": "OK"
+    }
+  }
+}
+--------------------------------------------------
+
+The health API response is described in <<making-sense-of-task-manager-health-stats>>.
+
+The health monitoring API exposes three sections:
+
+* `configuration` is described in detail under <<task-manager-health-evaluate-the-configuration>>
+* `workload` is described in detail under <<task-manager-health-evaluate-the-workload>>
+* `runtime` is described in detail under <<task-manager-health-evaluate-the-runtime>>

docs/developer/plugin-list.asciidoc

@@ -525,6 +525,7 @@ routes, etc.
 
 |{kib-repo}blob/{branch}/x-pack/plugins/task_manager/README.md[taskManager]
 |The task manager is a generic system for running background tasks.
+Documentation: https://www.elastic.co/guide/en/kibana/master/task-manager.html
 
 
 |{kib-repo}blob/{branch}/x-pack/plugins/telemetry_collection_xpack/README.md[telemetryCollectionXpack]

docs/settings/task-manager-settings.asciidoc

@@ -27,6 +27,18 @@ Task Manager runs background tasks by polling for work on an interval.  You can
 
   | `xpack.task_manager.max_workers`
   | The maximum number of tasks that this Kibana instance will run simultaneously.  Defaults to 10.
-
+|===
+
+[float]
+[[task-manager-health-settings]]
+==== Task Manager Health settings 
+
+Settings that configure the <<task-manager-health-monitoring>> endpoint.
+
+[cols="2*<"]
+|===
+| `xpack.task_manager.`
+`monitored_task_execution_thresholds`
+  | Configures the threshold of failed task executions at which point the `warn` or `error` health status will be set under each task type execution status (under `stats.runtime.value.excution.result_frequency_percent_as_number[${task type}].status`). This setting allows configuration of both the default level and a custom task type specific level. By default, this setting is configured to mark the health of every task type as `warning` when it exceeds 80% failed executions, and as `error` at 90%. Custom configurations allow you to reduce this threshold to catch failures sooner for task types you might consider critical, such as alerting tasks. This value can be set to any number between 0 to 100, and a threshold is hit when the value *exceeds* this number. This means that you can avoid setting the status to `error` by setting the threshold at 100, or hit `error` the moment any task failes by setting the threshold to 0 (as it will exceed 0 once a single failer occurs).
 
 |===

docs/setup/settings.asciidoc

@@ -683,5 +683,5 @@ include::{kib-repo-dir}/settings/reporting-settings.asciidoc[]
 include::secure-settings.asciidoc[]
 include::{kib-repo-dir}/settings/security-settings.asciidoc[]
 include::{kib-repo-dir}/settings/spaces-settings.asciidoc[]
-include::{kib-repo-dir}/settings/telemetry-settings.asciidoc[]
 include::{kib-repo-dir}/settings/task-manager-settings.asciidoc[]
+include::{kib-repo-dir}/settings/telemetry-settings.asciidoc[]

docs/user/alerting/alerting-production-considerations.asciidoc

@@ -2,34 +2,45 @@
 [[alerting-production-considerations]]
 == Production considerations
 
-{kib} alerting run both alert checks and actions as persistent background tasks managed by the Kibana Task Manager. This has two major benefits:
+{kib} Alerting run both alert checks and actions as persistent background tasks managed by the {kib} Task Manager.
 
-* *Persistence*: all task state and scheduling is stored in {es}, so if {kib} is restarted, alerts and actions will pick up where they left off.  Task definitions for alerts and actions are stored in the index specified by `xpack.task_manager.index` (defaults to `.kibana_task_manager`).  It is important to have at least 1 replica of this index for production deployments, since if you lose this index all scheduled alerts and actions are also lost.
-* *Scaling*: multiple {kib} instances can read from and update the same task queue in {es}, allowing the alerting and action load to be distributed across instances. In cases where a {kib} instance no longer has capacity to run alert checks or actions, capacity can be increased by adding additional {kib} instances.
+When relying on Alerts and Actions as mission critical services, it is important to ensure the {kib} Task Manager <<task-manager-production-considerations>> are followed.
 
 [float]
+[[alerting-background-tasks]]
 === Running background alert checks and actions
 
-{kib} background tasks are managed by:
+{kib} uses <<task-manager>> to run Alerts and Actions as background tasks, distributed across all {kib} instances in the cluster.
 
-* Polling an {es} task index for overdue tasks at 3 second intervals.  This interval can be changed using the `xpack.task_manager.poll_interval` setting.
-* Tasks are then claiming them by updating them in the {es} index, using optimistic concurrency control to prevent conflicts. Each {kib} instance can run a maximum of 10 concurrent tasks, so a maximum of 10 tasks are claimed each interval. 
-* Tasks are run on the {kib} server. 
-* In the case of alerts which are recurring background checks, upon completion the task is scheduled again according to the <<defining-alerts-general-details, check interval>>.
+By default, each {kib} instance polls for work at 3 second intervals, and can run a maximum of 10 concurrent tasks.
+These tasks are then ran on the {kib} server.
+
+In the case of alerts which are recurring background checks, upon completion the task is scheduled again according to the <<defining-alerts-general-details, check interval>>.
+
+For more details on Task Manager, see <<task-manager-background-tasks>>.
 
 [IMPORTANT]
 ==============================================
-Because by default tasks are polled at 3 second intervals and only 10 tasks can run concurrently per {kib} instance, it is possible for alert and action tasks to be run late. This can happen if: 
+It is possible for alert and action tasks to be run late or at an inconsistent schedule.
+This is usually a symptom of the specific usage of the cluster in question.
 
-* Alerts use a small *check interval*. The lowest interval possible is 3 seconds, though intervals of 30 seconds or higher are recommended.
-* Many alerts or actions must be *run at once*. In this case pending tasks will queue in {es}, and be pulled 10 at a time from the queue at 3 second intervals. 
-* *Long running tasks* occupy slots for an extended time, leaving fewer slots for other tasks. 
-
-For details on the settings that can influence the performance and throughput of Task Manager, see <<task-manager-settings,`Task Manager Settings`>>.
+Such issues can be addressed by tweaking the {kib} <<task-manager>> or scaling the deployment to better suit your use case.
 
+For detailed guidance, see <<alerting-troubleshooting,`Alerting Troubleshooting`>>.
 ==============================================
 
 [float]
-=== Deployment considerations
+[[alerting-scaling-guidance]]
+=== Scaling Guidance
+
+As Alerts and Actions leverage <<task-manager>> to perform the majority of their work, scaling {kib} Alerting is possible by following the <<task-manager-scaling-guidance,Task Manager Scaling Guidance>>.
+
+When estimating the required task throughput, keep the following in mind:
+
+* Each Alert utilizes a single recurring task which is scheduled to run at the cadance defined by its <<defining-alerts-general-details, check interval>>
+* Each Action utilizes a single task, but since <<alerting-concepts-suppressing-duplicate-notifications, actions are taken per instance>>, alerts can end up generating a large number of non-recurring tasks.
+
+It is difficult to predict how much throughput is needed to ensure all Alerts and Actions are executed at consistent schedules.
+By counting Alerts as recurring tasks and Actions as non-recurring tasks, a rough throughput <<task-manager-rough-throughput-estimation,can be estimated>> as a _tasks per minute_ measurment.
 
-{es} and {kib} instances use the system clock to determine the current time. To ensure schedules are triggered when expected, you should synchronize the clocks of all nodes in the cluster using a time service such as http://www.ntp.org/[Network Time Protocol].
+Predicting the buffer required to account for Actions depends heavily on the Alert Types you use, the amount of Alert Instances they might detect and the number of actions you might choose to assign to action groups when defining your Alerts. With that in mind, we recommend regularly <<task-manager-health-monitoring,monitoring the health>> of your {kib} Task Managers instances.

docs/user/alerting/alerting-troubleshooting.asciidoc

@@ -0,0 +1,56 @@
+[role="xpack"]
+[[alerting-troubleshooting]]
+== Alerting Troubleshooting
+
+Use the information in this section to troubleshoot common problems and find answers for frequently asked questions.
+
+For issues that you cannot fix yourself … we’re here to help.
+If you are an existing Elastic customer with a support contract, please create a ticket in the
+https://support.elastic.co/customers/s/login/[Elastic Support portal].
+Or post in the https://discuss.elastic.co/[Elastic forum].
+
+
+[float]
+[[alerts-small-check-interval-run-late]]
+=== Alerts with Small Check Intervals Run Late
+
+*Symptoms*:
+
+Alerts are scheduled to run every 2 seconds but they seem to be running too late
+
+*Resolution*:
+
+As described under <<alerting-background-tasks>>, Alerts are ran as background tasks at a cadence defined by their *check interval*.
+When an Alert *check interval* is smaller than the Task Manager <<task-manager-settings,`poll_interval`>> the alert will inevitably run late.
+
+This issue can be addressed by either tweaking the <<task-manager-settings,{kib} Task Manager settings>> or increasing the *check interval* of the Alerts in question.
+
+For details on resolving this issue please see <<task-manager-health-scheduled-tasks-small-schedule-interval-run-late>>.
+
+
+[float]
+[[scheduled-alerts-run-late]]
+=== Alerts Run Late
+
+*Symptoms*:
+
+Scheduled Alerts run at an inconsistent cadence, often running late.
+
+Actions run long after the status of an Alert changes, notifying us of the change too late.
+
+*Resolution*:
+
+As described under <<alerting-background-tasks>>, Alerts and Actions are ran as background tasks ran by each {kib} instance at a default rate of 10 tasks every 3 seconds.
+
+This means that if many alerts or actions have been scheduled to run at the same time, pending tasks will queue in {es}. Each {kib} instance then polls for pending tasks at a rate of up to 10 tasks at a time, at 3 second intervals. As Alerts and Actions are backed by tasks, it is possible for pending tasks in the queue to exceed this capacity and run late as a result.
+
+For details on diagnosing the underlying causes of such delays, please see <<task-manager-health-tasks-run-late>>.
+
+Alerting and Action tasks can be identified amongst the variety of tasks in {kib} by their type.
+
+* Alert tasks always begin with `alerting:`. For example, the `alerting:.index-threshold` tasks back the <<alert-type-index-threshold, Index Threshold Stack Alert>>.
+* Action tasks always begin with `actions:`. For example, the `actions:.index` tasks back the <<index-action-type>>.
+
+When diagnosing Alerting related issue, this subset of the tasks in the system may be of particular interest.
+
+For broader details on monitoring and diagnosing task execution in Task Manager, see <<task-manager-health-monitoring>>.

docs/user/alerting/index.asciidoc

@@ -4,3 +4,4 @@ include::action-types.asciidoc[]
 include::alert-types.asciidoc[]
 include::geo-alert-types.asciidoc[]
 include::alerting-production-considerations.asciidoc[]
+include::alerting-troubleshooting.asciidoc[]

docs/user/index.asciidoc

@@ -43,6 +43,8 @@ include::reporting/index.asciidoc[]
 
 include::alerting/index.asciidoc[]
 
+include::task-manager/index.asciidoc[]
+
 include::api.asciidoc[]
 
 include::plugins.asciidoc[]

docs/user/task-manager/index.asciidoc

@@ -0,0 +1,4 @@
+include::task-manager.asciidoc[]
+include::task-manager-production-considerations.asciidoc[]
+include::task-manager-health-monitoring.asciidoc[]
+include::task-manager-troubleshooting.asciidoc[]