Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

Merged
merged 14 commits into from
Jun 14, 2021
Merged

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a62f736
Adding Security Authentication jobs in 7.14
ajosh0504 Jun 9, 2021
f664246
Renamed some jobs
ajosh0504 Jun 9, 2021
62f256e
Changing memory limits and linting change
ajosh0504 Jun 9, 2021
bc2fee5
Linting fix
ajosh0504 Jun 9, 2021
38fa9ab
Changed the order
ajosh0504 Jun 9, 2021
4f837f5
Adding module to ml_modules.tsx
ajosh0504 Jun 10, 2021
ae27ef2
Update recognize_module.ts
Jun 10, 2021
78b768b
Update recognize_module.ts
Jun 10, 2021
39cde71
Update recognize_module.ts
Jun 10, 2021
8c9d807
descriptions
Jun 10, 2021
7c5d712
Update auth_rare_hour_for_a_user.json
Jun 10, 2021
f926eb3
Minor nitpicking
ajosh0504 Jun 10, 2021
fa596d3
Merge branch 'master' into security_auth_jobs
kibanamachine Jun 14, 2021
f252962
memory limits
Jun 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/logo.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"icon": "logoSecurity"
}
77 changes: 77 additions & 0 deletions x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
{
"id": "security_auth",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the expected response for the auditbeat data set used in the x-pack/test/api_integration/apis/ml/modules/recognize_module·ts test needs editing, to add in the ID of the new module security_auth which also now matches our test data set. This block here

    {
      testTitleSuffix: 'for auditbeat dataset',
      sourceDataArchive: 'x-pack/test/functional/es_archives/ml/module_auditbeat',
      indexPattern: 'ft_module_auditbeat',
      user: USER.ML_POWERUSER,
      expected: {
        responseCode: 200,
        moduleIds: ['auditbeat_process_hosts_ecs', 'security_linux', 'siem_auditbeat'],
      },
    },

"title": "Security: Authentication",
"description": "Detect anomalous activity in your ECS-compatible authentication logs.",
"type": "auth data",
"logoFile": "logo.json",
"defaultIndexPattern": "auditbeat-*,logs-*,filebeat-*,winlogbeat-*",
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
}
]
}
},
"jobs": [
{
"id": "auth_high_count_logon_events_for_a_source_ip",
"file": "auth_high_count_logon_events_for_a_source_ip.json"
},
{
"id": "auth_high_count_logon_fails",
"file": "auth_high_count_logon_fails.json"
},
{
"id": "auth_high_count_logon_events",
"file": "auth_high_count_logon_events.json"
},
{
"id": "auth_rare_hour_for_a_user",
"file": "auth_rare_hour_for_a_user.json"
},
{
"id": "auth_rare_source_ip_for_a_user",
"file": "auth_rare_source_ip_for_a_user.json"
},
{
"id": "auth_rare_user",
"file": "auth_rare_user.json"
}
],
"datafeeds": [
{
"id": "datafeed-auth_high_count_logon_events_for_a_source_ip",
"file": "datafeed_auth_high_count_logon_events_for_a_source_ip.json",
"job_id": "auth_high_count_logon_events_for_a_source_ip"
},
{
"id": "datafeed-auth_high_count_logon_fails",
"file": "datafeed_auth_high_count_logon_fails.json",
"job_id": "auth_high_count_logon_fails"
},
{
"id": "datafeed-auth_high_count_logon_events",
"file": "datafeed_auth_high_count_logon_events.json",
"job_id": "auth_high_count_logon_events"
},
{
"id": "datafeed-auth_rare_hour_for_a_user",
"file": "datafeed_auth_rare_hour_for_a_user.json",
"job_id": "auth_rare_hour_for_a_user"
},
{
"id": "datafeed-auth_rare_source_ip_for_a_user",
"file": "datafeed_auth_rare_source_ip_for_a_user.json",
"job_id": "auth_rare_source_ip_for_a_user"
},
{
"id": "datafeed-auth_rare_user",
"file": "datafeed_auth_rare_user.json",
"job_id": "auth_rare_user"
}
]
}
29 changes: 29 additions & 0 deletions .../server/models/data_recognizer/modules/security_auth/ml/auth_high_count_logon_events.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"job_type": "anomaly_detector",
"description": "",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "high count of logon events",
"function": "high_non_zero_count",
"detector_index": 0
}
],
"influencers": []
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "2mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
34 changes: 34 additions & 0 deletions ...ata_recognizer/modules/security_auth/ml/auth_high_count_logon_events_for_a_source_ip.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"job_type": "anomaly_detector",
"description": "",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "high count of auth events for a source IP",
"function": "high_non_zero_count",
"by_field_name": "source.ip",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"winlog.event_data.LogonType",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "12mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
29 changes: 29 additions & 0 deletions ...l/server/models/data_recognizer/modules/security_auth/ml/auth_high_count_logon_fails.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"job_type": "anomaly_detector",
"description": "",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "high count of logon fails",
"function": "high_non_zero_count",
"detector_index": 0
}
],
"influencers": []
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "2mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
33 changes: 33 additions & 0 deletions .../ml/server/models/data_recognizer/modules/security_auth/ml/auth_rare_hour_for_a_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"job_type": "anomaly_detector",
"description": "",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "rare hour for a user",
"function": "time_of_day",
"by_field_name": "user.name",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "3mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
34 changes: 34 additions & 0 deletions ...erver/models/data_recognizer/modules/security_auth/ml/auth_rare_source_ip_for_a_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"job_type": "anomaly_detector",
"description": "",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "rare source IP for a user",
"function": "rare",
"by_field_name": "source.ip",
"partition_field_name": "user.name",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "6mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
33 changes: 33 additions & 0 deletions x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/ml/auth_rare_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"job_type": "anomaly_detector",
"description": "",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "rare user",
"function": "rare",
"by_field_name": "user.name",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "2mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
26 changes: 26 additions & 0 deletions ...odels/data_recognizer/modules/security_auth/ml/datafeed_auth_high_count_logon_events.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_high_count_logon_events",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "success"
}
}
]
}
}
}
26 changes: 26 additions & 0 deletions ...nizer/modules/security_auth/ml/datafeed_auth_high_count_logon_events_for_a_source_ip.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_high_count_logon_events_for_a_source_ip",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "success"
}
}
]
}
}
}
26 changes: 26 additions & 0 deletions ...models/data_recognizer/modules/security_auth/ml/datafeed_auth_high_count_logon_fails.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_high_count_logon_fails",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "failure"
}
}
]
}
}
}
26 changes: 26 additions & 0 deletions ...r/models/data_recognizer/modules/security_auth/ml/datafeed_auth_rare_hour_for_a_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_rare_hour_for_a_user",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "success"
}
}
]
}
}
}
Loading