Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

Merged
merged 14 commits into from
Jun 14, 2021
Merged

[ML] Adds Authentication module with six ML jobs for ECS data (Auditbeat, Winlogbeat, Filebeat and Logs) #101840

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a62f736
Adding Security Authentication jobs in 7.14
ajosh0504 Jun 9, 2021
f664246
Renamed some jobs
ajosh0504 Jun 9, 2021
62f256e
Changing memory limits and linting change
ajosh0504 Jun 9, 2021
bc2fee5
Linting fix
ajosh0504 Jun 9, 2021
38fa9ab
Changed the order
ajosh0504 Jun 9, 2021
4f837f5
Adding module to ml_modules.tsx
ajosh0504 Jun 10, 2021
ae27ef2
Update recognize_module.ts
Jun 10, 2021
78b768b
Update recognize_module.ts
Jun 10, 2021
39cde71
Update recognize_module.ts
Jun 10, 2021
8c9d807
descriptions
Jun 10, 2021
7c5d712
Update auth_rare_hour_for_a_user.json
Jun 10, 2021
f926eb3
Minor nitpicking
ajosh0504 Jun 10, 2021
fa596d3
Merge branch 'master' into security_auth_jobs
kibanamachine Jun 14, 2021
f252962
memory limits
Jun 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/logo.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"icon": "logoSecurity"
}
77 changes: 77 additions & 0 deletions x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
{
"id": "security_auth",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the expected response for the auditbeat data set used in the x-pack/test/api_integration/apis/ml/modules/recognize_module·ts test needs editing, to add in the ID of the new module security_auth which also now matches our test data set. This block here

    {
      testTitleSuffix: 'for auditbeat dataset',
      sourceDataArchive: 'x-pack/test/functional/es_archives/ml/module_auditbeat',
      indexPattern: 'ft_module_auditbeat',
      user: USER.ML_POWERUSER,
      expected: {
        responseCode: 200,
        moduleIds: ['auditbeat_process_hosts_ecs', 'security_linux', 'siem_auditbeat'],
      },
    },

"title": "Security: Authentication",
"description": "Detect anomalous activity in your ECS-compatible authentication logs.",
"type": "auth data",
"logoFile": "logo.json",
"defaultIndexPattern": "auditbeat-*,logs-*,filebeat-*,winlogbeat-*",
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
}
]
}
},
"jobs": [
{
"id": "auth_high_count_logon_events_for_a_source_ip",
"file": "auth_high_count_logon_events_for_a_source_ip.json"
},
{
"id": "auth_high_count_logon_fails",
"file": "auth_high_count_logon_fails.json"
},
{
"id": "auth_high_count_logon_events",
"file": "auth_high_count_logon_events.json"
},
{
"id": "auth_rare_hour_for_a_user",
"file": "auth_rare_hour_for_a_user.json"
},
{
"id": "auth_rare_source_ip_for_a_user",
"file": "auth_rare_source_ip_for_a_user.json"
},
{
"id": "auth_rare_user",
"file": "auth_rare_user.json"
}
],
"datafeeds": [
{
"id": "datafeed-auth_high_count_logon_events_for_a_source_ip",
"file": "datafeed_auth_high_count_logon_events_for_a_source_ip.json",
"job_id": "auth_high_count_logon_events_for_a_source_ip"
},
{
"id": "datafeed-auth_high_count_logon_fails",
"file": "datafeed_auth_high_count_logon_fails.json",
"job_id": "auth_high_count_logon_fails"
},
{
"id": "datafeed-auth_high_count_logon_events",
"file": "datafeed_auth_high_count_logon_events.json",
"job_id": "auth_high_count_logon_events"
},
{
"id": "datafeed-auth_rare_hour_for_a_user",
"file": "datafeed_auth_rare_hour_for_a_user.json",
"job_id": "auth_rare_hour_for_a_user"
},
{
"id": "datafeed-auth_rare_source_ip_for_a_user",
"file": "datafeed_auth_rare_source_ip_for_a_user.json",
"job_id": "auth_rare_source_ip_for_a_user"
},
{
"id": "datafeed-auth_rare_user",
"file": "datafeed_auth_rare_user.json",
"job_id": "auth_rare_user"
}
]
}
29 changes: 29 additions & 0 deletions .../server/models/data_recognizer/modules/security_auth/ml/auth_high_count_logon_events.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"job_type": "anomaly_detector",
"description": "Security: Authentication - looks for an unusually large spike in successful authentication events. This can be due to password spraying, user enumeration or brute force activity.",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "high count of logon events",
"function": "high_non_zero_count",
"detector_index": 0
}
],
"influencers": []
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "2mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
34 changes: 34 additions & 0 deletions ...ata_recognizer/modules/security_auth/ml/auth_high_count_logon_events_for_a_source_ip.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"job_type": "anomaly_detector",
"description": "Security: Authentication - looks for an unusually large spike in successful authentication events events from a particular source IP address. This can be due to password spraying, user enumeration or brute force activity.",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "high count of auth events for a source IP",
"function": "high_non_zero_count",
"by_field_name": "source.ip",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"winlog.event_data.LogonType",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "12mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
29 changes: 29 additions & 0 deletions ...l/server/models/data_recognizer/modules/security_auth/ml/auth_high_count_logon_fails.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
{
"job_type": "anomaly_detector",
"description": "Security: Authentication - looks for an unusually large spike in authentication failure events. This can be due to password spraying, user enumeration or brute force activity and may be a precursor to account takeover or credentialed access.",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "high count of logon fails",
"function": "high_non_zero_count",
"detector_index": 0
}
],
"influencers": []
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "2mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
33 changes: 33 additions & 0 deletions .../ml/server/models/data_recognizer/modules/security_auth/ml/auth_rare_hour_for_a_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"job_type": "anomaly_detector",
"description": "Security: Authentication - looks for a user logging in at a time of day that is unusual for the user. This can be due to credentialed access via a compromised account when the user and the threat actor are in different time zones. In addition, unauthorized user activity often takes place during non-business hours.",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "rare hour for a user",
"function": "time_of_day",
"by_field_name": "user.name",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "3mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
34 changes: 34 additions & 0 deletions ...erver/models/data_recognizer/modules/security_auth/ml/auth_rare_source_ip_for_a_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"job_type": "anomaly_detector",
"description": "Security: Authentication - looks for a user logging in from an IP address that is unusual for the user. This can be due to credentialed access via a compromised account when the user and the threat actor are in different locations. An unusual source IP address for a username could also be due to lateral movement when a compromised account is used to pivot between hosts.",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "rare source IP for a user",
"function": "rare",
"by_field_name": "source.ip",
"partition_field_name": "user.name",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "6mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
33 changes: 33 additions & 0 deletions x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/ml/auth_rare_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"job_type": "anomaly_detector",
"description": "Security: Authentication - looks for an unusual user name in the authentication logs. An unusual user name is one way of detecting credentialed access by means of a new or dormant user account. A user account that is normally inactive, because the user has left the organization, which becomes active, may be due to credentialed access using a compromised account password. Threat actors will sometimes also create new users as a means of persisting in a compromised web application.",
"groups": [
"security",
"authentication"
],
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "rare user",
"function": "rare",
"by_field_name": "user.name",
"detector_index": 0
}
],
"influencers": [
"source.ip",
"user.name"
]
},
"allow_lazy_open": true,
"analysis_limits": {
"model_memory_limit": "2mb"
},
"data_description": {
"time_field": "@timestamp"
},
"custom_settings": {
"created_by": "ml-module-security-auth"
}
}
26 changes: 26 additions & 0 deletions ...odels/data_recognizer/modules/security_auth/ml/datafeed_auth_high_count_logon_events.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_high_count_logon_events",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "success"
}
}
]
}
}
}
26 changes: 26 additions & 0 deletions ...nizer/modules/security_auth/ml/datafeed_auth_high_count_logon_events_for_a_source_ip.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_high_count_logon_events_for_a_source_ip",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "success"
}
}
]
}
}
}
26 changes: 26 additions & 0 deletions ...models/data_recognizer/modules/security_auth/ml/datafeed_auth_high_count_logon_fails.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_high_count_logon_fails",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "failure"
}
}
]
}
}
}
26 changes: 26 additions & 0 deletions ...r/models/data_recognizer/modules/security_auth/ml/datafeed_auth_rare_hour_for_a_user.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"job_id": "auth_rare_hour_for_a_user",
"indices": [
"auditbeat-*",
"logs-*",
"filebeat-*",
"winlogbeat-*"
],
"max_empty_searches": 10,
"query": {
"bool": {
"filter": [
{
"term": {
"event.category": "authentication"
}
},
{
"term": {
"event.outcome": "success"
}
}
]
}
}
}
Loading