Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Adds optimizations for Logs UI anomaly detection jobs #102191

Merged

Conversation

peteharverson
Copy link
Contributor

Summary

Adds two optimizations for the Logs UI anomaly detection jobs:

  • Sets stop_on_warn: true for the Log entries categories job
  • Sets both the Log entries categories job and the Log entries rate job to use a dedicated results index

The parameter stop_on_warn will automatically stop partitions from being analysed, if ML identifies that they are not suited for categorization.

Since both jobs are configured with model plot enabled, it is expected that their results index can be large, so it is best practice to use a dedicated results index.

Fixes #100936

@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@weltenwort weltenwort self-requested a review June 15, 2021 13:30
Copy link
Member

@weltenwort weltenwort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to work fine, thanks for helping us out! 👍

Could you please increment the custom_settings.job_revision in the log_entry_categories_count.json file, in which you updated the job config? That way the user will be notified of the changed job behabiour in the UI.

Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@weltenwort weltenwort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@peteharverson
Copy link
Contributor Author

@elasticmachine merge upstream

@peteharverson
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
infra 1.7MB 1.7MB +147.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @peteharverson

@peteharverson peteharverson merged commit 35e10ba into elastic:master Jun 16, 2021
@peteharverson peteharverson deleted the ml-logs-ui-job-enhancements branch June 16, 2021 10:31
peteharverson added a commit to peteharverson/kibana that referenced this pull request Jun 16, 2021
…02191)

* [ML] Adds optimizations for Logs UI anomaly detection jobs

* [ML] Increment version for log-entry-categories-count job

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
peteharverson added a commit that referenced this pull request Jun 16, 2021
…102318)

* [ML] Adds optimizations for Logs UI anomaly detection jobs

* [ML] Increment version for log-entry-categories-count job

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
majagrubic pushed a commit to majagrubic/kibana that referenced this pull request Jun 18, 2021
…02191)

* [ML] Adds optimizations for Logs UI anomaly detection jobs

* [ML] Increment version for log-entry-categories-count job

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimizations for Logs UI ML categorization job
5 participants