Skip to content

Commit

Permalink
[DOCS] improving production documentation for task manager and alerti…
Browse files Browse the repository at this point in the history
…ng (elastic#87484)

* improving production documentation for task manager and alerting

* calling it Kibana Task Manager to be more explicit

* addressing PR feedback
  • Loading branch information
bmcconaghy committed Jan 6, 2021
1 parent 014c92a commit c64f37b
Show file tree
Hide file tree
Showing 6 changed files with 44 additions and 9 deletions.
1 change: 1 addition & 0 deletions docs/settings/settings-xkb.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,6 @@ include::logs-ui-settings.asciidoc[]
include::ml-settings.asciidoc[]
include::reporting-settings.asciidoc[]
include::spaces-settings.asciidoc[]
include::task-manager-settings.asciidoc[]
include::i18n-settings.asciidoc[]
include::fleet-settings.asciidoc[]
32 changes: 32 additions & 0 deletions docs/settings/task-manager-settings.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
[role="xpack"]
[[task-manager-settings-kb]]
=== Task Manager settings in {kib}
++++
<titleabbrev>Task Manager settings</titleabbrev>
++++

Task Manager runs background tasks by polling for work on an interval. You can configure its behavior to tune for performance and throughput.

[float]
[[task-manager-settings]]
==== Task Manager settings

[cols="2*<"]
|===
| `xpack.task_manager.max_attempts`
| The maximum number of times a task will be attempted before being abandoned as failed. Defaults to 3.

| `xpack.task_manager.poll_interval`
| How often, in milliseconds, the task manager will look for more work. Defaults to 3000 and cannot be lower than 100.

| `xpack.task_manager.request_capacity`
| How many requests can Task Manager buffer before it rejects new requests. Defaults to 1000.

| `xpack.task_manager.index`
| The name of the index used to store task information. Defaults to `.kibana_task_manager`.

| `xpack.task_manager.max_workers`
| The maximum number of tasks that this Kibana instance will run simultaneously. Defaults to 10.


|===
2 changes: 1 addition & 1 deletion docs/user/alerting/alerting-getting-started.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Alert schedules are defined as an interval between subsequent checks, and can ra

[IMPORTANT]
==============================================
The intervals of alert checks in {kib} are approximate, their timing of their execution is affected by factors such as the frequency at which tasks are claimed and the task load on the system. See <<alerting-scale-performance>> for more information.
The intervals of alert checks in {kib} are approximate, their timing of their execution is affected by factors such as the frequency at which tasks are claimed and the task load on the system. See <<alerting-production-considerations>> for more information.
==============================================

[float]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,28 +1,30 @@
[role="xpack"]
[[alerting-scale-performance]]
== Scale and performance
[[alerting-production-considerations]]
== Production considerations

{kib} alerting run both alert checks and actions as persistent background tasks. This has two major benefits:
{kib} alerting run both alert checks and actions as persistent background tasks managed by the Kibana Task Manager. This has two major benefits:

* *Persistence*: all task state and scheduling is stored in {es}, so if {kib} is restarted, alerts and actions will pick up where they left off.
* *Persistence*: all task state and scheduling is stored in {es}, so if {kib} is restarted, alerts and actions will pick up where they left off. Task definitions for alerts and actions are stored in the index specified by `xpack.task_manager.index` (defaults to `.kibana_task_manager`). It is important to have at least 1 replica of this index for production deployments, since if you lose this index all scheduled alerts and actions are also lost.
* *Scaling*: multiple {kib} instances can read from and update the same task queue in {es}, allowing the alerting and action load to be distributed across instances. In cases where a {kib} instance no longer has capacity to run alert checks or actions, capacity can be increased by adding additional {kib} instances.

[float]
=== Running background alert checks and actions

{kib} background tasks are managed by:

* Polling an {es} task index for overdue tasks at 3 second intervals.
* Polling an {es} task index for overdue tasks at 3 second intervals. This interval can be changed using the `xpack.task_manager.poll_interval` setting.
* Tasks are then claiming them by updating them in the {es} index, using optimistic concurrency control to prevent conflicts. Each {kib} instance can run a maximum of 10 concurrent tasks, so a maximum of 10 tasks are claimed each interval.
* Tasks are run on the {kib} server.
* In the case of alerts which are recurring background checks, upon completion the task is scheduled again according to the <<defining-alerts-general-details, check interval>>.

[IMPORTANT]
==============================================
Because tasks are polled at 3 second intervals and only 10 tasks can run concurrently per {kib} instance, it is possible for alert and action tasks to be run late. This can happen if:
Because by default tasks are polled at 3 second intervals and only 10 tasks can run concurrently per {kib} instance, it is possible for alert and action tasks to be run late. This can happen if:
* Alerts use a small *check interval*. The lowest interval possible is 3 seconds, though intervals of 30 seconds or higher are recommended.
* Many alerts or actions must be *run at once*. In this case pending tasks will queue in {es}, and be pulled 10 at a time from the queue at 3 second intervals.
* *Long running tasks* occupy slots for an extended time, leaving fewer slots for other tasks.
For details on the settings that can influence the performance and throughput of Task Manager, see {task-manager-settings}.
==============================================
2 changes: 1 addition & 1 deletion docs/user/alerting/defining-alerts.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ image::images/alert-flyout-general-details.png[alt='All alerts have name, tags,

Name:: The name of the alert. While this name does not have to be unique, the name can be referenced in actions and also appears in the searchable alert listing in the management UI. A distinctive name can help identify and find an alert.
Tags:: A list of tag names that can be applied to an alert. Tags can help you organize and find alerts, because tags appear in the alert listing in the management UI which is searchable by tag.
Check every:: This value determines how frequently the alert conditions below are checked. Note that the timing of background alert checks are not guaranteed, particularly for intervals of less than 10 seconds. See <<alerting-scale-performance>> for more information.
Check every:: This value determines how frequently the alert conditions below are checked. Note that the timing of background alert checks are not guaranteed, particularly for intervals of less than 10 seconds. See <<alerting-production-considerations>> for more information.
Notify every:: This value limits how often actions are repeated when an alert instance remains active across alert checks. See <<alerting-concepts-suppressing-duplicate-notifications>> for more information.

[float]
Expand Down
2 changes: 1 addition & 1 deletion docs/user/alerting/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ include::alerting-getting-started.asciidoc[]
include::defining-alerts.asciidoc[]
include::action-types.asciidoc[]
include::alert-types.asciidoc[]
include::alerting-scale-performance.asciidoc[]
include::alerting-production-considerations.asciidoc[]

0 comments on commit c64f37b

Please sign in to comment.