[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

ajosh0504 · 2021-06-09T21:18:18Z

Summary

This PR adds a security_auth module for use within the Security app. Detailed information, stats, and screenshots are here: https://github.com/elastic/mechagodzilla/issues/35

It contains 1 Module called security_auth consisting of:

ML Job configurations for 6 jobs:
- auth_high_count_logon_events_for_a_source_ip
- auth_high_count_logon_events
- auth_high_count_logon_fails
- auth_rare_hour_for_a_user
- auth_rare_source_ip_for_a_user
- auth_rare_user
Corresponding datafeed configurations
Logo
Descriptions coming soon

elasticmachine · 2021-06-09T21:18:20Z

Pinging @elastic/ml-ui (:ml)

randomuserid · 2021-06-09T21:42:53Z

I tested the module on a local dev instance

peteharverson · 2021-06-10T10:09:46Z

x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/manifest.json

@@ -0,0 +1,77 @@
+{
+  "id": "security_auth",


Looks like the expected response for the auditbeat data set used in the x-pack/test/api_integration/apis/ml/modules/recognize_module·ts test needs editing, to add in the ID of the new module security_auth which also now matches our test data set. This block here

{ testTitleSuffix: 'for auditbeat dataset', sourceDataArchive: 'x-pack/test/functional/es_archives/ml/module_auditbeat', indexPattern: 'ft_module_auditbeat', user: USER.ML_POWERUSER, expected: { responseCode: 200, moduleIds: ['auditbeat_process_hosts_ecs', 'security_linux', 'siem_auditbeat'], }, },

ajosh0504 · 2021-06-10T13:46:19Z

x-pack/test/api_integration/apis/ml/modules/recognize_module.ts

-        moduleIds: ['auditbeat_process_hosts_ecs', 'security_linux', 'siem_auditbeat'],
+        moduleIds: [
+          'auditbeat_process_hosts_ecs',
+          'security_auth',


@peteharverson I noticed that in the testing output, but security_auth has been added to the auditbeat testset and yet that test still fails.

@ajosh0504 From what I can see in the test log it looks like we're now expecting the security_auth module to be recognized in the ft_module_auditbeat index pattern, but it is not. If we think that it's ok it's not recognized there, we can remove it from the expected modules for this dataset.

@pheyos Any idea it might not be recognizing the module?

@ajosh0504 I'd need to double check the dataset. But the new module checks for "event.category": "authentication". And it it is not recognized in an index pattern, it means that the documents don't have this field or have a different value for this field. I'll take a closer look and report back here.

@ajosh0504 I've checked the ft_module_auditbeat dataset and it has "event.category": "audit-rule" (see screenshot), so it's ok that it doesn't match.

Yes event.category was added last year and this Auditbeat data may be from 2019 so that test will have to be skipped until we have newer data.

I've created a reminder issue to decide whether or not to update the dataset, see #101910.
For now it's fine to not have the security_auth in the list of expected modules for the "old" auditbeat dataset.

this test modules uses older Auditbeat data which predates the event.category field so the test has to be skipped per https://elastic.zoom.us/j/93000943632?pwd=TmpvNWhtYUNzMUc0c0N6Tlc2QlVPZz09

needs to be a single line

Some linters want spaces and some linters want no spaces. This linter wants spaces.

added description text

removed a wayward newline char

ajosh0504 · 2021-06-14T14:52:55Z

@elasticmachine merge upstream

blaklaybul

Jobs and datafeed configs look good, but I am a bit concerned about the low model memory limits. For auth_high_count_logon_events_for_a_source_ip we have a limit of 12mb. In other modules we've used 128, 256, or 512 mb for jobs with by or partition fields that could have high cardinality - can we bump these up a bit here?

ajosh0504 · 2021-06-14T16:17:01Z

@blaklaybul We'll update that. Is there a standard limit for low cardinality jobs as well? I have seen 16/32 mb in some of the jobs.

blaklaybul · 2021-06-14T16:36:34Z

@ajosh0504 for the existing jobs, we did thorough testing on live systems to arrive at the memory limits we ship with. Fields with a potential for higher cardinality warrant higher limits. source.ip can really get up there in cardinality! For example, this job uses a simple count detector, but we still have the limit set fairly high since the systems these are meant to be run on can be quite large.

I would suggest testing these new jobs on live system to get better memory estimates, or at least setting them
conservatively to > 128mb

randomuserid · 2021-06-14T16:38:58Z

@ajosh0504 for the existing jobs, we did thorough testing on live systems to arrive at the memory limits we ship with. Fields with a potential for higher cardinality warrant higher limits. source.ip can really get up there in cardinality! For example, this job uses a simple count detector, but we still have the limit set fairly high since the systems these are meant to be run on can be quite large.

I would suggest testing these new jobs on live system to get better memory estimates, or at least setting theme conservatively to > 128mb

They were tested on a medium sized prod cluster and we set the memory minimums to a multiple of what we saw there. We can increase it again though

blaklaybul · 2021-06-14T16:45:34Z

@randomuserid Ok, but 6mb seems fairly low for this job which uses source.ip as a by field and has a partition field of user.name. I would think that this config when run on a large system would require significantly more memory than that.

Using another example - high_count_by_destination_country uses a relatively simple config with a known number of potential by field values and we set the memory limit to 32mb.

raised memory limits to 128mb which is larger than the highest observed peak model bytes for the most memory hungry jobs in this event class.

randomuserid · 2021-06-14T16:56:02Z

@randomuserid Ok, but 6mb seems fairly low for this job which uses source.ip as a by field and has a partition field of user.name. I would think that this config when run on a large system would require significantly more memory than that.

Using another example - high_count_by_destination_country uses a relatively simple config with a known number of potential by field values and we set the memory limit to 32mb.

How's this: I increased each to 128MB which is a bit more then the largest observed peak model bytes for any job in this event class. That should be sufficient for most data sets.

blaklaybul

New mem limits LGTM

kibanamachine · 2021-06-14T18:59:28Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	6.9MB	6.9MB	+16.0B

History

💚 Build #130980 succeeded fa596d3
💚 Build #130662 succeeded f926eb3
💚 Build #130618 succeeded 7c5d712
💚 Build #130522 succeeded 39cde71
💔 Build #130514 failed 78b768b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ajosh0504

randomuserid

…eat, Winlogbeat, Filebeat and Logs) (elastic#101840) * Adding Security Authentication jobs in 7.14 * Renamed some jobs * Changing memory limits and linting change * Linting fix * Changed the order * Adding module to ml_modules.tsx * Update recognize_module.ts this test modules uses older Auditbeat data which predates the event.category field so the test has to be skipped per https://elastic.zoom.us/j/93000943632?pwd=TmpvNWhtYUNzMUc0c0N6Tlc2QlVPZz09 * Update recognize_module.ts needs to be a single line * Update recognize_module.ts Some linters want spaces and some linters want no spaces. This linter wants spaces. * descriptions added description text * Update auth_rare_hour_for_a_user.json removed a wayward newline char * Minor nitpicking * memory limits raised memory limits to 128mb which is larger than the highest observed peak model bytes for the most memory hungry jobs in this event class. Co-authored-by: Craig <mailredirector36@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-06-14T19:38:55Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…eat, Winlogbeat, Filebeat and Logs) (#101840) (#102127) * Adding Security Authentication jobs in 7.14 * Renamed some jobs * Changing memory limits and linting change * Linting fix * Changed the order * Adding module to ml_modules.tsx * Update recognize_module.ts this test modules uses older Auditbeat data which predates the event.category field so the test has to be skipped per https://elastic.zoom.us/j/93000943632?pwd=TmpvNWhtYUNzMUc0c0N6Tlc2QlVPZz09 * Update recognize_module.ts needs to be a single line * Update recognize_module.ts Some linters want spaces and some linters want no spaces. This linter wants spaces. * descriptions added description text * Update auth_rare_hour_for_a_user.json removed a wayward newline char * Minor nitpicking * memory limits raised memory limits to 128mb which is larger than the highest observed peak model bytes for the most memory hungry jobs in this event class. Co-authored-by: Craig <mailredirector36@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com> Co-authored-by: Craig <mailredirector36@gmail.com>

…eat, Winlogbeat, Filebeat and Logs) (elastic#101840) * Adding Security Authentication jobs in 7.14 * Renamed some jobs * Changing memory limits and linting change * Linting fix * Changed the order * Adding module to ml_modules.tsx * Update recognize_module.ts this test modules uses older Auditbeat data which predates the event.category field so the test has to be skipped per https://elastic.zoom.us/j/93000943632?pwd=TmpvNWhtYUNzMUc0c0N6Tlc2QlVPZz09 * Update recognize_module.ts needs to be a single line * Update recognize_module.ts Some linters want spaces and some linters want no spaces. This linter wants spaces. * descriptions added description text * Update auth_rare_hour_for_a_user.json removed a wayward newline char * Minor nitpicking * memory limits raised memory limits to 128mb which is larger than the highest observed peak model bytes for the most memory hungry jobs in this event class. Co-authored-by: Craig <mailredirector36@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Adding Security Authentication jobs in 7.14

a62f736

ajosh0504 added release_note:enhancement :ml v7.14.0 labels Jun 9, 2021

ajosh0504 requested review from randomuserid and a team June 9, 2021 21:18

ajosh0504 self-assigned this Jun 9, 2021

ajosh0504 added 2 commits June 9, 2021 14:20

Renamed some jobs

f664246

Changing memory limits and linting change

62f256e

randomuserid added release_note:feature Makes this part of the condensed release notes auto-backport Deprecated - use backport:version if exact versions are needed and removed release_note:enhancement labels Jun 9, 2021

ajosh0504 added 3 commits June 9, 2021 14:49

Linting fix

bc2fee5

Changed the order

38fa9ab

Adding module to ml_modules.tsx

4f837f5

ajosh0504 requested a review from a team as a code owner June 10, 2021 01:41

peteharverson reviewed Jun 10, 2021

View reviewed changes

ajosh0504 commented Jun 10, 2021

View reviewed changes

Update recognize_module.ts

ae27ef2

this test modules uses older Auditbeat data which predates the event.category field so the test has to be skipped per https://elastic.zoom.us/j/93000943632?pwd=TmpvNWhtYUNzMUc0c0N6Tlc2QlVPZz09

pheyos mentioned this pull request Jun 10, 2021

[ML] Functional tests - check if modules_auditbeat esArchive needs to be updated #101910

Open

Craig added 2 commits June 10, 2021 11:08

Update recognize_module.ts

78b768b

needs to be a single line

Update recognize_module.ts

39cde71

Some linters want spaces and some linters want no spaces. This linter wants spaces.

ajosh0504 mentioned this pull request Jun 10, 2021

[New Rule] Add detection rules for auth ML jobs elastic/detection-rules#1283

Merged

Craig and others added 3 commits June 10, 2021 15:03

descriptions

8c9d807

added description text

Update auth_rare_hour_for_a_user.json

7c5d712

removed a wayward newline char

Minor nitpicking

f926eb3

Merge branch 'master' into security_auth_jobs

fa596d3

blaklaybul suggested changes Jun 14, 2021

View reviewed changes

memory limits

f252962

raised memory limits to 128mb which is larger than the highest observed peak model bytes for the most memory hungry jobs in this event class.

blaklaybul approved these changes Jun 14, 2021

View reviewed changes

randomuserid approved these changes Jun 14, 2021

View reviewed changes

ajosh0504 merged commit 35f9625 into master Jun 14, 2021

kibanamachine mentioned this pull request Jun 14, 2021

[7.x] [ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) (#101840) #102127

Merged

lcawl mentioned this pull request Sep 10, 2021

[DOCS] Add security:authentication jobs elastic/stack-docs#1813

Merged

spalger deleted the security_auth_jobs branch May 8, 2022 22:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

ajosh0504 commented Jun 9, 2021 •

edited by randomuserid

Loading

elasticmachine commented Jun 9, 2021

randomuserid commented Jun 9, 2021

peteharverson Jun 10, 2021

ajosh0504 Jun 10, 2021

pheyos Jun 10, 2021

ajosh0504 Jun 10, 2021

pheyos Jun 10, 2021

pheyos Jun 10, 2021

randomuserid Jun 10, 2021

pheyos Jun 10, 2021

ajosh0504 commented Jun 14, 2021

blaklaybul left a comment

ajosh0504 commented Jun 14, 2021

blaklaybul commented Jun 14, 2021 •

edited

Loading

randomuserid commented Jun 14, 2021 •

edited

Loading

blaklaybul commented Jun 14, 2021

randomuserid commented Jun 14, 2021

blaklaybul left a comment

kibanamachine commented Jun 14, 2021

randomuserid left a comment

kibanamachine commented Jun 14, 2021

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

Conversation

ajosh0504 commented Jun 9, 2021 • edited by randomuserid Loading

Summary

elasticmachine commented Jun 9, 2021

randomuserid commented Jun 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajosh0504 commented Jun 14, 2021

blaklaybul left a comment

Choose a reason for hiding this comment

ajosh0504 commented Jun 14, 2021

blaklaybul commented Jun 14, 2021 • edited Loading

randomuserid commented Jun 14, 2021 • edited Loading

blaklaybul commented Jun 14, 2021

randomuserid commented Jun 14, 2021

blaklaybul left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 14, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

randomuserid left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 14, 2021

💚 Backport successful

ajosh0504 commented Jun 9, 2021 •

edited by randomuserid

Loading

blaklaybul commented Jun 14, 2021 •

edited

Loading

randomuserid commented Jun 14, 2021 •

edited

Loading