Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.12] [Alerting][Docs] Adds Alerting & Task Manager Scalability Guidance & Health Monitoring (#91171) #93603

Merged
merged 2 commits into from
Mar 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions docs/api/task-manager/health.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
[[task-manager-api-health]]
=== Get Task Manager health API
++++
<titleabbrev>Get Task Manager health</titleabbrev>
++++

Retrieve the health status of the {kib} Task Manager.

[[task-manager-api-health-request]]
==== Request

`GET <kibana host>:<port>/api/task_manager/_health`

[[task-manager-api-health-codes]]
==== Response code

`200`::
Indicates a successful call.

[[task-manager-api-health-example]]
==== Example

Retrieve the health status of the {kib} Task Manager:

[source,sh]
--------------------------------------------------
$ curl -X GET api/task_manager/_health
--------------------------------------------------
// KIBANA

The API returns the following:

[source,sh]
--------------------------------------------------
{
"id": "15415ecf-cdb0-4fef-950a-f824bd277fe4",
"timestamp": "2021-02-16T11:38:10.077Z",
"status": "OK",
"last_update": "2021-02-16T11:38:09.934Z",
"stats": {
"configuration": {
"timestamp": "2021-02-16T11:29:05.055Z",
"value": {
"request_capacity": 1000,
"max_poll_inactivity_cycles": 10,
"monitored_aggregated_stats_refresh_rate": 60000,
"monitored_stats_running_average_window": 50,
"monitored_task_execution_thresholds": {
"default": {
"error_threshold": 90,
"warn_threshold": 80
},
"custom": {}
},
"poll_interval": 3000,
"max_workers": 10
},
"status": "OK"
},
"runtime": {
"timestamp": "2021-02-16T11:38:09.934Z",
"value": {
"polling": {
"last_successful_poll": "2021-02-16T11:38:09.934Z",
"last_polling_delay": "2021-02-16T11:29:05.053Z",
"duration": {
"p50": 0,
"p90": 0,
"p95": 0,
"p99": 0
},
"claim_conflicts": {
"p50": 0,
"p90": 0,
"p95": 0,
"p99": 0
},
"claim_mismatches": {
"p50": 0,
"p90": 0,
"p95": 0,
"p99": 0
},
"result_frequency_percent_as_number": {
"Failed": 0,
"NoAvailableWorkers": 0,
"NoTasksClaimed": 0,
"RanOutOfCapacity": 0,
"RunningAtCapacity": 0,
"PoolFilled": 0
}
},
"drift": {
"p50": 0,
"p90": 0,
"p95": 0,
"p99": 0
},
"load": {
"p50": 0,
"p90": 0,
"p95": 0,
"p99": 0
},
"execution": {
"duration": {},
"result_frequency_percent_as_number": {}
}
},
"status": "OK"
},
"workload": {
"timestamp": "2021-02-16T11:38:05.826Z",
"value": {
"count": 26,
"task_types": {},
"schedule": [],
"overdue": 0,
"estimated_schedule_density": []
},
"status": "OK"
}
}
}
--------------------------------------------------

The health API response is described in <<making-sense-of-task-manager-health-stats>>.

The health monitoring API exposes three sections:

* `configuration` is described in detail under <<task-manager-health-evaluate-the-configuration>>
* `workload` is described in detail under <<task-manager-health-evaluate-the-workload>>
* `runtime` is described in detail under <<task-manager-health-evaluate-the-runtime>>
1 change: 1 addition & 0 deletions docs/developer/plugin-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -521,6 +521,7 @@ routes, etc.

|{kib-repo}blob/{branch}/x-pack/plugins/task_manager/README.md[taskManager]
|The task manager is a generic system for running background tasks.
Documentation: https://www.elastic.co/guide/en/kibana/master/task-manager-production-considerations.html


|{kib-repo}blob/{branch}/x-pack/plugins/telemetry_collection_xpack/README.md[telemetryCollectionXpack]
Expand Down
13 changes: 13 additions & 0 deletions docs/settings/task-manager-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,18 @@ Task Manager runs background tasks by polling for work on an interval. You can
| `xpack.task_manager.max_workers`
| The maximum number of tasks that this Kibana instance will run simultaneously. Defaults to 10.
Starting in 8.0, it will not be possible to set the value greater than 100.
|===

[float]
[[task-manager-health-settings]]
==== Task Manager Health settings

Settings that configure the <<task-manager-health-monitoring>> endpoint.

[cols="2*<"]
|===
| `xpack.task_manager.`
`monitored_task_execution_thresholds`
| Configures the threshold of failed task executions at which point the `warn` or `error` health status is set under each task type execution status (under `stats.runtime.value.excution.result_frequency_percent_as_number[${task type}].status`). This setting allows configuration of both the default level and a custom task type specific level. By default, this setting is configured to mark the health of every task type as `warning` when it exceeds 80% failed executions, and as `error` at 90%. Custom configurations allow you to reduce this threshold to catch failures sooner for task types that you might consider critical, such as alerting tasks. This value can be set to any number between 0 to 100, and a threshold is hit when the value *exceeds* this number. This means that you can avoid setting the status to `error` by setting the threshold at 100, or hit `error` the moment any task fails by setting the threshold to 0 (as it will exceed 0 once a single failure occurs).

|===
2 changes: 1 addition & 1 deletion docs/setup/settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -683,5 +683,5 @@ include::{kib-repo-dir}/settings/reporting-settings.asciidoc[]
include::secure-settings.asciidoc[]
include::{kib-repo-dir}/settings/security-settings.asciidoc[]
include::{kib-repo-dir}/settings/spaces-settings.asciidoc[]
include::{kib-repo-dir}/settings/telemetry-settings.asciidoc[]
include::{kib-repo-dir}/settings/task-manager-settings.asciidoc[]
include::{kib-repo-dir}/settings/telemetry-settings.asciidoc[]
8 changes: 8 additions & 0 deletions docs/user/alerting/alerting-getting-started.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,14 @@ If you are using an *on-premises* Elastic Stack deployment with <<using-kibana-w

* You must enable Transport Layer Security (TLS) for communication <<configuring-tls-kib-es, between {es} and {kib}>>. {kib} alerting uses <<api-keys, API keys>> to secure background alert checks and actions, and API keys require {ref}/configuring-tls.html#tls-http[TLS on the HTTP interface]. A proxy will not suffice.

[float]
[[alerting-setup-production]]
== Production considerations and scaling guidance

When relying on alerts and actions as mission critical services, make sure you follow the <<alerting-production-considerations,Alerting production considerations>>.

See <<alerting-scaling-guidance>> for more information on the scalability of {kib} alerting.

[float]
[[alerting-security]]
== Security
Expand Down
35 changes: 0 additions & 35 deletions docs/user/alerting/alerting-production-considerations.asciidoc

This file was deleted.

55 changes: 55 additions & 0 deletions docs/user/alerting/alerting-troubleshooting.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
[role="xpack"]
[[alerting-troubleshooting]]
== Alerting Troubleshooting

This page describes how to resolve common problems you might encounter with Alerting.
If your problem isn’t described here, please review open issues in the following GitHub repositories:

* https://github.com/elastic/kibana/issues[kibana] (https://github.com/elastic/kibana/issues?q=is%3Aopen+is%3Aissue+label%3AFeature%3AAlerting[Alerting issues])

Have a question? Contact us in the https://discuss.elastic.co/[discuss forum].

[float]
[[alerts-small-check-interval-run-late]]
=== Alerts with small check intervals run late

*Problem*:

Alerts with a small check interval, such as every two seconds, run later than scheduled.

*Resolution*:

Alerts run as background tasks at a cadence defined by their *check interval*.
When an Alert *check interval* is smaller than the Task Manager <<task-manager-settings,`poll_interval`>> the alert will run late.

Either tweak the <<task-manager-settings,{kib} Task Manager settings>> or increase the *check interval* of the alerts in question.

For more details, see <<task-manager-health-scheduled-tasks-small-schedule-interval-run-late>>.


[float]
[[scheduled-alerts-run-late]]
=== Alerts run late

*Problem*:

Scheduled alerts run at an inconsistent cadence, often running late.

Actions run long after the status of an alert changes, sending a notification of the change too late.

*Solution*:

Alerts and actions run as background tasks by each {kib} instance at a default rate of ten tasks every three seconds.

If many alerts or actions are scheduled to run at the same time, pending tasks will queue in {es}. Each {kib} instance then polls for pending tasks at a rate of up to ten tasks at a time, at three second intervals. Because alerts and actions are backed by tasks, it is possible for pending tasks in the queue to exceed this capacity and run late.

For details on diagnosing the underlying causes of such delays, see <<task-manager-health-tasks-run-late>>.

Alerting and action tasks are identified by their type.

* Alert tasks always begin with `alerting:`. For example, the `alerting:.index-threshold` tasks back the <<alert-type-index-threshold, index threshold stack alert>>.
* Action tasks always begin with `actions:`. For example, the `actions:.index` tasks back the <<index-action-type, index action>>.

When diagnosing issues related to Alerting, focus on the thats that begin with `alerting:` and `actions:`.

For more details on monitoring and diagnosing task execution in Task Manager, see <<task-manager-health-monitoring>>.
2 changes: 1 addition & 1 deletion docs/user/alerting/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ include::alerting-getting-started.asciidoc[]
include::defining-alerts.asciidoc[]
include::action-types.asciidoc[]
include::alert-types.asciidoc[]
include::alerting-production-considerations.asciidoc[]
include::alerting-troubleshooting.asciidoc[]
2 changes: 2 additions & 0 deletions docs/user/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ include::monitoring/monitoring-kibana.asciidoc[leveloffset=+2]

include::security/securing-kibana.asciidoc[]

include::production-considerations/index.asciidoc[]

include::discover.asciidoc[]

include::dashboard/dashboard.asciidoc[]
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
[role="xpack"]
[[alerting-production-considerations]]
== Alerting production considerations

++++
<titleabbrev>Alerting</titleabbrev>
++++

Alerting runs both alert checks and actions as persistent background tasks managed by the Task Manager.

When relying on alerts and actions as mission critical services, make sure you follow the <<task-manager-production-considerations, production considerations>> for Task Manager.

[float]
[[alerting-background-tasks]]
=== Running background alert checks and actions

{kib} uses background tasks to run alerts and actions, distributed across all {kib} instances in the cluster.

By default, each {kib} instance polls for work at three second intervals, and can run a maximum of ten concurrent tasks.
These tasks are then run on the {kib} server.

Alerts are recurring background tasks which are rescheduled according to the <<defining-alerts-general-details, check interval>> on completion.
Actions are non-recurring background tasks which are deleted on completion.

For more details on Task Manager, see <<task-manager-background-tasks>>.

[IMPORTANT]
==============================================
Alert and action tasks can run late or at an inconsistent schedule.
This is typically a symptom of the specific usage of the cluster in question.

You can address such issues by tweaking the <<task-manager-settings,Task Manager settings>> or scaling the deployment to better suit your use case.

For detailed guidance, see <<alerting-troubleshooting, Alerting Troubleshooting>>.
==============================================

[float]
[[alerting-scaling-guidance]]
=== Scaling Guidance

As alerts and actions leverage background tasks to perform the majority of work, scaling Alerting is possible by following the <<task-manager-scaling-guidance,Task Manager Scaling Guidance>>.

When estimating the required task throughput, keep the following in mind:

* Each alert uses a single recurring task that is scheduled to run at the cadence defined by its <<defining-alerts-general-details,check interval>>.
* Each action uses a single task. However, because <<alerting-concepts-suppressing-duplicate-notifications,actions are taken per instance>>, alerts can generate a large number of non-recurring tasks.

It is difficult to predict how much throughput is needed to ensure all alerts and actions are executed at consistent schedules.
By counting alerts as recurring tasks and actions as non-recurring tasks, a rough throughput <<task-manager-rough-throughput-estimation,can be estimated>> as a _tasks per minute_ measurement.

Predicting the buffer required to account for actions depends heavily on the alert types you use, the amount of alert Instances they might detect, and the number of actions you might choose to assign to action groups. With that in mind, regularly <<task-manager-health-monitoring,monitor the health>> of your Task Manager instances.
5 changes: 5 additions & 0 deletions docs/user/production-considerations/index.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
include::production.asciidoc[]
include::alerting-production-considerations.asciidoc[]
include::task-manager-production-considerations.asciidoc[]
include::task-manager-health-monitoring.asciidoc[]
include::task-manager-troubleshooting.asciidoc[]
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
[[production]]
== Use {kib} in a production environment
= Use {kib} in a production environment

++++
<titleabbrev>Production considerations</titleabbrev>
++++

* <<configuring-kibana-shield>>
* <<csp-strict-mode>>
Expand Down
Loading