[ML] Adds ML modules for Metrics UI Integration #76460

blaklaybul · 2020-09-01T23:14:44Z

Summary

Adds the files for a new metrics_ui_hosts and metrics_ui_k8s modules for use within the Metrics app, containing the job and datafeed configuration files that support the analyses designed for the metrics integration.

For each modules, this PR contains:

module manifest.json containing a query that uniquely defines when the module should appear in the ML app.
ML Job configurations for 4 jobs:
- ~~{hosts||k8s}_cpu_usage~~
- {hosts||k8s}_memory_usage
- {hosts||k8s}_network_in
- {hosts||k8s}_network_out
Datafeed configurations to accompany the jobs.
Logo

To Do:

provide job and module descriptions, finalize titles

phillipb · 2020-09-02T02:09:14Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_hosts/manifest.json

+      },
+    "jobs": [
+        {
+            "id": "hosts_cpu_usage",


Noticed that the host jobs have an id prefixed with hosts, but the k8s jobs don't. Should we be consistent here? Not sure we actually need the prefix.

i've included the k8s prefix on the kubernetes jobs, as we'll want to distinguish between them in ML job management.

blaklaybul · 2020-09-15T15:19:43Z

Custom URLs have been added for all jobs. The app/ prefix has been left off, so they will not work as is. We're awaiting a PR from @peteharverson that will include metrics in our isKibanaUrl check for custom URLs. Also, to accommodate the new URLs, kubernetes.pod.name has been replaced by kubernetes.pod.id in the terms aggs in datafeed_k8s_network_in.json and datafeed_k8s_network_out.json. This ensures that these field values are passed to the datafeed and can therefore be used as influencers in the URL creation.

lcawl · 2020-09-15T22:05:00Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_hosts/manifest.json

+{
+    "id": "metrics_ui_hosts",
+    "title": "Metrics Hosts",
+    "description": "Detect anomalous memory, cpu, and network behavior on hosts.",


Suggested change

"description": "Detect anomalous memory, cpu, and network behavior on hosts.",

"description": "Detect anomalous memory, CPU, and network behavior on hosts.",

updated - thanks!

lcawl · 2020-09-15T22:11:07Z

...ck/plugins/ml/server/models/data_recognizer/modules/metrics_ui_hosts/ml/hosts_cpu_usage.json

+      "hosts",
+      "metrics"
+    ],
+    "description": "Metrics: Hosts - Identify unusual spikes in cpu utilization across hosts.",


Suggested change

"description": "Metrics: Hosts - Identify unusual spikes in cpu utilization across hosts.",

"description": "Metrics: Hosts - Identify unusual spikes in CPU utilization across hosts.",

updated - thanks!

lcawl · 2020-09-15T22:12:42Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/manifest.json

+{
+    "id": "metrics_ui_k8s",
+    "title": "Metrics Kubernetes",
+    "description": "Detect anomalous memory, cpu, and network behavior on kubernetes pods.",


Suggested change

"description": "Detect anomalous memory, cpu, and network behavior on kubernetes pods.",

"description": "Detect anomalous memory, CPU, and network behavior on Kubernetes pods.",

updated - thanks!

lcawl · 2020-09-15T22:13:14Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_cpu_usage.json

+      "k8s",
+      "metrics"
+    ],
+    "description": "Metrics: Kubernetes - Identify unusual spikes in cpu utilization across kubernetes pods.",


Suggested change

"description": "Metrics: Kubernetes - Identify unusual spikes in cpu utilization across kubernetes pods.",

"description": "Metrics: Kubernetes - Identify unusual spikes in CPU utilization across Kubernetes pods.",

updated - thanks!

lcawl · 2020-09-15T22:13:32Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_memory_usage.json

+      "k8s",
+      "metrics"
+    ],
+    "description": "Metrics: Kubernetes - Identify unusual spikes in memory usage across kubernetes pods.",


Suggested change

"description": "Metrics: Kubernetes - Identify unusual spikes in memory usage across kubernetes pods.",

"description": "Metrics: Kubernetes - Identify unusual spikes in memory usage across Kubernetes pods.",

updated - thanks!

lcawl · 2020-09-15T22:14:22Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_network_in.json

@@ -0,0 +1,42 @@
+{
+  "job_type": "anomaly_detector",
+  "description": "Metrics: Kubernetes - Identify unusual spikes in inbound traffic across kubernetes pods.",


Suggested change

"description": "Metrics: Kubernetes - Identify unusual spikes in inbound traffic across kubernetes pods.",

"description": "Metrics: Kubernetes - Identify unusual spikes in inbound traffic across Kubernetes pods.",

updated - thanks!

lcawl · 2020-09-15T22:14:36Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_network_out.json

@@ -0,0 +1,42 @@
+{
+  "job_type": "anomaly_detector",
+  "description": "Metrics: Kubernetes - Identify unusual spikes in outbound traffic across kubernetes pods.",


Suggested change

"description": "Metrics: Kubernetes - Identify unusual spikes in outbound traffic across kubernetes pods.",

"description": "Metrics: Kubernetes - Identify unusual spikes in outbound traffic across Kubernetes pods.",

updated - thanks!

blaklaybul · 2020-09-15T22:48:58Z

@phillipb I wanted to lay out how you will need to override the job and datafeed configs based on the partition fields provided by the user (using cloud.project.id in the following examples). The flow will be slightly different for the hosts and k8s modules:

metrics-ui-hosts

job configurations

The analysis_config object in each job configuration will need to include "partition_field_name": "cloud.project.id" and cloud.project.id will need to be added to the list of influencers. So, for example, for the job hosts_network_in, the analysis_config will need appear as follows:

"analysis_config": {
      "bucket_span": "15m",
      "detectors": [
        {
          "detector_description": "max(bytes_in_derivative)",
          "function": "max",
          "field_name": "bytes_in_derivative",
          "parition_field_name": "cloud.project.id"
        }
      ],
      "influencers": [
        "host.name",
        "cloud.project.id"
        ],
      "summary_count_field_name": "doc_count"
    }

datafeed configurations

For the metrics-ui-hosts module, all datafeeds with the exception of datafeed_hosts_memory_usage use aggregations - for these, we will need to wrap the existing aggregation in a terms agg on the user-supplied partition field. This terms agg must have a name matching the user-supplied field. Using datafeed_hosts_network_in as an example, the aggregations object will need to appear as such:

{
    "aggregations": {
        "cloud.project.id": {
            "terms": {
                "field": "cloud.project.id"
            },
            "aggregations": {
                "host.name": {"terms": {"field": "host.name"},
                    "aggregations": {
                        "buckets": {
                            "date_histogram": {"field": "@timestamp","fixed_interval": "5m"},
                            "aggregations": {
                                "@timestamp": {"max": {"field": "@timestamp"}},
                                "bytes_in_max": {"max": {"field": "system.network.in.bytes"}},
                                "bytes_in_derivative": {"derivative": {"buckets_path": "bytes_in_max"}},
                                "positive_only":{
                                    "bucket_script": {
                                        "buckets_path": {"in_derivative": "bytes_in_derivative.value"},
                                        "script": "params.in_derivative > 0.0 ? params.in_derivative : 0.0"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

metrics-ui-k8s

job configurations

Since the metrics-ui-k8s module is shipping with a default partition field - kubernetes.namespace - the detectors in analysis_config already contain "partition_field_name": "kubernetes.namespace". If the user chooses to override the default, all you will need to do is replace kubernetes.namespace with the user supplied field in the detector and in the influencer list.

datafeed configurations

Only datafeed_k8s_network_in and datafeed_k8s_network_out contains aggregations in the metrics-ui-k8s module. Since we are supplying a default partition field, the outer aggregations are already in the configs. So for these datafeeds, all you will need to do is replace kubernetes.namespace with the user-supplied field name in the outer aggregation in 2 places - the name of the agg and the "field" value.

elasticmachine · 2020-09-16T00:01:03Z

Pinging @elastic/ml-ui (:ml)

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_cpu_usage.json

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_memory_usage.json

peteharverson · 2020-09-16T11:42:53Z

Worth noting here for reference, that there is an open issue that we currently do not support the plot of metric data in the Anomaly Explorer or Single Metric Viewer charts for detectors which use a derivative aggregation on a scripted field - #18464. This is because of the difficulties of reverse engineering the datafeed config aggregations back to a search to run on the source data to obtain the metric data for plotting in the charts. Currently the charts just display blank.

This will affect the four inbound / outbound traffic jobs.

Similarly, the hosts CPU usage job uses a bucket_script for the CPU metric in the datafeed, so again, the metric data in the anomaly charts will be blank.

peteharverson

Tested these two modules with the metrics-ui-full data set and overall it looks good. A couple of questions about descriptions for detectors, plus whether we want to remove the query section from the hosts module before merging.

peteharverson · 2020-09-16T13:28:15Z

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_hosts/manifest.json

+    "type": "Metricbeat Data",
+    "logoFile": "logo.json",
+    "defaultIndexPattern": "metricbeat-*",
+    "query": {


If the jobs in this module provide no value without the specific overrides we are expecting from the Metrics UI, then removing this query block is the way to hide it from the ML job wizards.

the query and defaultIndexPattern fields have been removed, as we do in the logs integration modules.

…ptions

blaklaybul · 2020-09-16T15:30:12Z

As per @sorantis 's request, the CPU jobs have been removed from both modules.

blaklaybul · 2020-09-16T16:07:19Z

@elasticmachine merge upstream

peteharverson

Latest edits LGTM

lcawl

Descriptions LGTM

blaklaybul · 2020-09-16T20:36:49Z

@elasticmachine merge upstream

peteharverson · 2020-09-17T08:13:14Z

@elasticmachine merge upstream

* adds metrics ml integration * renames jobs, updates datafeeds * adds allow_no_indices: true for datafeeds * updates module ids in manifest * adds custom urls * adds module and individual job descriptions * removes model plots * updates terms agg sizes * updates chunking config * removes query and default index pattern from manifest, updates descriptions Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* adds metrics ml integration * renames jobs, updates datafeeds * adds allow_no_indices: true for datafeeds * updates module ids in manifest * adds custom urls * adds module and individual job descriptions * removes model plots * updates terms agg sizes * updates chunking config * removes query and default index pattern from manifest, updates descriptions Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

kibanamachine · 2020-09-17T21:46:25Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: a4bb4fe

Build metrics

distributable file count

id	value	diff	baseline
default	45934	+16	45918

History

💚 Build #75102 succeeded a4bb4fe
💔 Build #74977 failed 9377511
💔 Build #74838 failed 5f01331
💚 Build #74856 succeeded 77532b0
💔 Build #74659 failed 1288b8f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…rok/new-patterns-component-use-array * 'master' of github.com:elastic/kibana: (140 commits) Add telemetry as an automatic privilege grant (elastic#77390) [Security Solutions][Cases] Cases Redesign (elastic#73247) Use Search API in TSVB (elastic#76274) [Mappings editor] Add support for constant_keyword field type (elastic#76564) [ML] Adds ML modules for Metrics UI Integration (elastic#76460) [Drilldowns] {{event.points}} in URL drilldown for VALUE_CLICK_TRIGGER (elastic#76771) Migrate status & stats APIs to KP + remove legacy status lib (elastic#76054) use App updater API instead of deprecated chrome.navLinks.update (elastic#77708) [CSM Dashboard] Remove points from line chart (elastic#77617) [APM] Trace timeline: Replace multi-fold function icons with new EuiIcon glyphs (elastic#77470) [Observability] Overview: Alerts section style improvements (elastic#77670) Bump the Node.js version used by Docker in CI (elastic#77714) Upgrade all minimist (sub)dependencies to version ^1.2.5 (elastic#60284) Remove unneeded forced package resolutions (elastic#77467) [ML] Add metrics app to check made for internal custom URLs (elastic#77627) Functional tests - add supertest for test_user (elastic#77584) [ML] Adding option to create AD jobs without starting the datafeed (elastic#77484) Bump node-fetch to 2.6.1 (elastic#77445) Bump sharkdown from v0.1.0 to v0.1.1 (elastic#77607) [APM]fixing y axis on transaction error rate to 100% (elastic#77609) ... # Conflicts: # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/manage_processor_form.container.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/manage_processor_form.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/drag_and_drop_text_list.scss # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/drag_and_drop_text_list.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/text_editor.scss # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/processors/grok.test.tsx

adds metrics ml integration

3a29942

blaklaybul added the :ml label Sep 1, 2020

phillipb reviewed Sep 2, 2020

View reviewed changes

phillipb mentioned this pull request Sep 4, 2020

[Metrics UI] Anomaly Detection setup flow for Metrics #76787

Merged

3 tasks

blaklaybul added 4 commits September 8, 2020 18:28

renames jobs, updates datafeeds

1d5e0d1

adds allow_no_indices: true for datafeeds

1a6198d

updates module ids in manifest

3418147

adds custom urls

bcd4620

adds module and individual job descriptions

edb65d3

lcawl reviewed Sep 15, 2020

View reviewed changes

blaklaybul added 3 commits September 15, 2020 19:03

removes model plots

2051546

updates terms agg sizes

1248e27

updates chunking config

1288b8f

blaklaybul marked this pull request as ready for review September 16, 2020 00:01

blaklaybul requested a review from a team as a code owner September 16, 2020 00:01

blaklaybul self-assigned this Sep 16, 2020

peteharverson reviewed Sep 16, 2020

View reviewed changes

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_cpu_usage.json Outdated Show resolved Hide resolved

x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_memory_usage.json Outdated Show resolved Hide resolved

peteharverson reviewed Sep 16, 2020

View reviewed changes

panbalag requested a review from szabosteve September 16, 2020 15:03

peteharverson mentioned this pull request Sep 16, 2020

[ML] Add metrics app to check made for internal custom URLs #77627

Merged

1 task

removes query and default index pattern from manifest, updates descri…

5f01331

…ptions

Merge branch 'master' into ml-metrics-integration-modules

77532b0

peteharverson added Feature:Anomaly Detection ML anomaly detection release_note:enhancement labels Sep 16, 2020

peteharverson added v7.10.0 v8.0.0 labels Sep 16, 2020

peteharverson approved these changes Sep 16, 2020

View reviewed changes

lcawl approved these changes Sep 16, 2020

View reviewed changes

Merge branch 'master' into ml-metrics-integration-modules

9377511

Merge branch 'master' into ml-metrics-integration-modules

a4bb4fe

blaklaybul merged commit 6d12c68 into elastic:master Sep 17, 2020

blaklaybul mentioned this pull request Sep 17, 2020

[7.x] [ML] Adds ML modules for Metrics UI Integration (#76460) #77759

Merged

szabosteve mentioned this pull request Sep 18, 2020

[DOCS] Adds Metrics AD configurations to OOTB jobs elastic/stack-docs#1366

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adds ML modules for Metrics UI Integration #76460

[ML] Adds ML modules for Metrics UI Integration #76460

blaklaybul commented Sep 1, 2020 •

edited

Loading

phillipb Sep 2, 2020

blaklaybul Sep 8, 2020

blaklaybul commented Sep 15, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

lcawl Sep 15, 2020

blaklaybul Sep 16, 2020

blaklaybul commented Sep 15, 2020 •

edited

Loading

elasticmachine commented Sep 16, 2020

peteharverson commented Sep 16, 2020 •

edited

Loading

peteharverson left a comment

peteharverson Sep 16, 2020

blaklaybul Sep 16, 2020

blaklaybul commented Sep 16, 2020

blaklaybul commented Sep 16, 2020

peteharverson left a comment

lcawl left a comment

blaklaybul commented Sep 16, 2020

peteharverson commented Sep 17, 2020

kibanamachine commented Sep 17, 2020

	"description": "Detect anomalous memory, cpu, and network behavior on hosts.",
	"description": "Detect anomalous memory, CPU, and network behavior on hosts.",

	"description": "Metrics: Hosts - Identify unusual spikes in cpu utilization across hosts.",
	"description": "Metrics: Hosts - Identify unusual spikes in CPU utilization across hosts.",

	"description": "Metrics: Kubernetes - Identify unusual spikes in cpu utilization across kubernetes pods.",
	"description": "Metrics: Kubernetes - Identify unusual spikes in CPU utilization across Kubernetes pods.",

[ML] Adds ML modules for Metrics UI Integration #76460

[ML] Adds ML modules for Metrics UI Integration #76460

Conversation

blaklaybul commented Sep 1, 2020 • edited Loading

Summary

To Do:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaklaybul commented Sep 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaklaybul commented Sep 15, 2020 • edited Loading

metrics-ui-hosts

job configurations

datafeed configurations

metrics-ui-k8s

job configurations

datafeed configurations

elasticmachine commented Sep 16, 2020

peteharverson commented Sep 16, 2020 • edited Loading

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaklaybul commented Sep 16, 2020

blaklaybul commented Sep 16, 2020

peteharverson left a comment

Choose a reason for hiding this comment

lcawl left a comment

Choose a reason for hiding this comment

blaklaybul commented Sep 16, 2020

peteharverson commented Sep 17, 2020

kibanamachine commented Sep 17, 2020

💚 Build Succeeded

Build metrics

distributable file count

History

blaklaybul commented Sep 1, 2020 •

edited

Loading

blaklaybul commented Sep 15, 2020 •

edited

Loading

peteharverson commented Sep 16, 2020 •

edited

Loading