-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ML] Adds ML modules for Metrics UI Integration #76460
[ML] Adds ML modules for Metrics UI Integration #76460
Conversation
}, | ||
"jobs": [ | ||
{ | ||
"id": "hosts_cpu_usage", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noticed that the host jobs have an id prefixed with hosts
, but the k8s jobs don't. Should we be consistent here? Not sure we actually need the prefix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i've included the k8s prefix on the kubernetes jobs, as we'll want to distinguish between them in ML job management.
Custom URLs have been added for all jobs. The |
{ | ||
"id": "metrics_ui_hosts", | ||
"title": "Metrics Hosts", | ||
"description": "Detect anomalous memory, cpu, and network behavior on hosts.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Detect anomalous memory, cpu, and network behavior on hosts.", | |
"description": "Detect anomalous memory, CPU, and network behavior on hosts.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
"hosts", | ||
"metrics" | ||
], | ||
"description": "Metrics: Hosts - Identify unusual spikes in cpu utilization across hosts.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Metrics: Hosts - Identify unusual spikes in cpu utilization across hosts.", | |
"description": "Metrics: Hosts - Identify unusual spikes in CPU utilization across hosts.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
{ | ||
"id": "metrics_ui_k8s", | ||
"title": "Metrics Kubernetes", | ||
"description": "Detect anomalous memory, cpu, and network behavior on kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Detect anomalous memory, cpu, and network behavior on kubernetes pods.", | |
"description": "Detect anomalous memory, CPU, and network behavior on Kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
"k8s", | ||
"metrics" | ||
], | ||
"description": "Metrics: Kubernetes - Identify unusual spikes in cpu utilization across kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Metrics: Kubernetes - Identify unusual spikes in cpu utilization across kubernetes pods.", | |
"description": "Metrics: Kubernetes - Identify unusual spikes in CPU utilization across Kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
"k8s", | ||
"metrics" | ||
], | ||
"description": "Metrics: Kubernetes - Identify unusual spikes in memory usage across kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Metrics: Kubernetes - Identify unusual spikes in memory usage across kubernetes pods.", | |
"description": "Metrics: Kubernetes - Identify unusual spikes in memory usage across Kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
@@ -0,0 +1,42 @@ | |||
{ | |||
"job_type": "anomaly_detector", | |||
"description": "Metrics: Kubernetes - Identify unusual spikes in inbound traffic across kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Metrics: Kubernetes - Identify unusual spikes in inbound traffic across kubernetes pods.", | |
"description": "Metrics: Kubernetes - Identify unusual spikes in inbound traffic across Kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
@@ -0,0 +1,42 @@ | |||
{ | |||
"job_type": "anomaly_detector", | |||
"description": "Metrics: Kubernetes - Identify unusual spikes in outbound traffic across kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"description": "Metrics: Kubernetes - Identify unusual spikes in outbound traffic across kubernetes pods.", | |
"description": "Metrics: Kubernetes - Identify unusual spikes in outbound traffic across Kubernetes pods.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated - thanks!
@phillipb I wanted to lay out how you will need to override the job and datafeed configs based on the partition fields provided by the user (using metrics-ui-hostsjob configurationsThe
datafeed configurationsFor the
metrics-ui-k8sjob configurationsSince the datafeed configurationsOnly |
Pinging @elastic/ml-ui (:ml) |
x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_cpu_usage.json
Outdated
Show resolved
Hide resolved
x-pack/plugins/ml/server/models/data_recognizer/modules/metrics_ui_k8s/ml/k8s_memory_usage.json
Outdated
Show resolved
Hide resolved
Worth noting here for reference, that there is an open issue that we currently do not support the plot of metric data in the Anomaly Explorer or Single Metric Viewer charts for detectors which use a derivative aggregation on a scripted field - #18464. This is because of the difficulties of reverse engineering the datafeed config aggregations back to a search to run on the source data to obtain the metric data for plotting in the charts. Currently the charts just display blank. This will affect the four inbound / outbound traffic jobs. Similarly, the hosts CPU usage job uses a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested these two modules with the metrics-ui-full
data set and overall it looks good. A couple of questions about descriptions for detectors, plus whether we want to remove the query
section from the hosts module before merging.
"type": "Metricbeat Data", | ||
"logoFile": "logo.json", | ||
"defaultIndexPattern": "metricbeat-*", | ||
"query": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the jobs in this module provide no value without the specific overrides we are expecting from the Metrics UI, then removing this query
block is the way to hide it from the ML job wizards.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the query
and defaultIndexPattern
fields have been removed, as we do in the logs integration modules.
As per @sorantis 's request, the CPU jobs have been removed from both modules. |
@elasticmachine merge upstream |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Latest edits LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Descriptions LGTM
@elasticmachine merge upstream |
@elasticmachine merge upstream |
* adds metrics ml integration * renames jobs, updates datafeeds * adds allow_no_indices: true for datafeeds * updates module ids in manifest * adds custom urls * adds module and individual job descriptions * removes model plots * updates terms agg sizes * updates chunking config * removes query and default index pattern from manifest, updates descriptions Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* adds metrics ml integration * renames jobs, updates datafeeds * adds allow_no_indices: true for datafeeds * updates module ids in manifest * adds custom urls * adds module and individual job descriptions * removes model plots * updates terms agg sizes * updates chunking config * removes query and default index pattern from manifest, updates descriptions Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
💚 Build SucceededBuild metricsdistributable file count
History
To update your PR or re-run it, just comment with: |
…rok/new-patterns-component-use-array * 'master' of github.com:elastic/kibana: (140 commits) Add telemetry as an automatic privilege grant (elastic#77390) [Security Solutions][Cases] Cases Redesign (elastic#73247) Use Search API in TSVB (elastic#76274) [Mappings editor] Add support for constant_keyword field type (elastic#76564) [ML] Adds ML modules for Metrics UI Integration (elastic#76460) [Drilldowns] {{event.points}} in URL drilldown for VALUE_CLICK_TRIGGER (elastic#76771) Migrate status & stats APIs to KP + remove legacy status lib (elastic#76054) use App updater API instead of deprecated chrome.navLinks.update (elastic#77708) [CSM Dashboard] Remove points from line chart (elastic#77617) [APM] Trace timeline: Replace multi-fold function icons with new EuiIcon glyphs (elastic#77470) [Observability] Overview: Alerts section style improvements (elastic#77670) Bump the Node.js version used by Docker in CI (elastic#77714) Upgrade all minimist (sub)dependencies to version ^1.2.5 (elastic#60284) Remove unneeded forced package resolutions (elastic#77467) [ML] Add metrics app to check made for internal custom URLs (elastic#77627) Functional tests - add supertest for test_user (elastic#77584) [ML] Adding option to create AD jobs without starting the datafeed (elastic#77484) Bump node-fetch to 2.6.1 (elastic#77445) Bump sharkdown from v0.1.0 to v0.1.1 (elastic#77607) [APM]fixing y axis on transaction error rate to 100% (elastic#77609) ... # Conflicts: # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/manage_processor_form.container.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/manage_processor_form.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/drag_and_drop_text_list.scss # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/drag_and_drop_text_list.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/text_editor.scss # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/processors/grok.test.tsx
Summary
Adds the files for a new
metrics_ui_hosts
andmetrics_ui_k8s
modules for use within the Metrics app, containing the job and datafeed configuration files that support the analyses designed for the metrics integration.For each modules, this PR contains:
{hosts||k8s}_cpu_usageTo Do: