Integrate threat intel feeds

[eirsep]

Description

---

Job Scheduler and threat intel feed Integration

• Integrates Job Scheduler Plugin in Security Analytics
• Adds framework in Security Analytics for configuring pre-package feeds.
• Adds support for job to constantly update system indices with feed data
• Configures 1 free feed - otx alienvault ip reputation ( a feed of malicious IPs)
• Adds support for IoC(indicator of Compromise) To ECS Field mapping for each log type which tells us fields where the given type of IoC may occur
• Feed data is stored in system indices. new data is populated in new system index and the old index is deleted.

---

Detector creation/updation flow changes

• Adds framework to convert threat intel IoCs data into doc level queries
• new boolean field added to Detector model - threat_intel_enabled. (Value is False by default). If set to true, we will create threat intel data based queries
• When feed data is updated, all detectors having threat intel data are also updated with queries built from the fresh feeds
• Doc level monitor created to execute sigma rules will have new queries.

---

Detector trigger:

• Detector trigger now supports new field detection_types which is a list with 2 possible values "rules" and "threat_intel"
• If trigger is configured and detection_types is passed with only "rules" in the list , we will create alerts when documents match sigma rules.
• If detection_types is passed with only "threat_intel" in the list , we will create alerts when documents match threat intel based queries.
• If detection_types is passed with both "rules" and "threat_intel" in the list , we will create alerts when documents match either sigma rules or threat intel based queries
• Detection types list param cannot be empty.

---

Detector findings and threat_intel_based queries:

• Findings created from matching threat intel based queries will contain the tags threat_intel, ioc_type:<ioc_type>, "field:<field>, feed_name:<feed_name>
• Threat intel based queries will have query id starting with the prefix threat_intel
• Threat intel based queries will have severity level high

---

Get Mapping VIew changes:

• We have added a new field threat_intel_field_aliases which returns a list of field aliases per ioc for each log type that the user needs to provide mappings for, from his source data.

Issues Resolved

Threat intel feeds in Security Analytics

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #669 (bed73ff) into main (294785f) will increase coverage by 1.92%.
The diff coverage is 43.11%.

@@             Coverage Diff              @@
##               main     #669      +/-   ##
============================================
+ Coverage     25.04%   26.96%   +1.92%     
- Complexity      946     1068     +122     
============================================
  Files           255      274      +19     
  Lines         11158    12345    +1187     
  Branches       1250     1342      +92     
============================================
+ Hits           2794     3329     +535     
- Misses         8111     8743     +632     
- Partials        253      273      +20     

Files	Coverage Δ
...yanalytics/settings/SecurityAnalyticsSettings.java	96.87% <100.00%> (+0.87%)	⬆️
...urityanalytics/threatIntel/common/TIFJobState.java	100.00% <100.00%> (ø)
...tics/transport/TransportGetMappingsViewAction.java	0.00% <ø> (ø)
...opensearch/securityanalytics/util/RuleIndices.java	0.00% <ø> (ø)
.../securityanalytics/action/GetDetectorResponse.java	18.91% <0.00%> (-0.53%)	⬇️
...ecurityanalytics/action/IndexDetectorResponse.java	57.14% <0.00%> (-1.69%)	⬇️
...ytics/threatintel/common/StashedThreadContext.java	83.33% <83.33%> (ø)
...analytics/threatIntel/action/PutTIFJobRequest.java	90.90% <90.90%> (ø)
...ecurityanalytics/threatIntel/common/Constants.java	0.00% <0.00%> (ø)
...ytics/transport/TransportSearchDetectorAction.java	0.00% <0.00%> (ø)
... and 24 more

[lezzago]

remove

[lezzago]

Why are we getting the feed id based on the first item in tifdList? Is it the same throughout?

[eirsep]

Currently yes,

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-669-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 884ddd04f63fb833153beaaeeb4ffd28ed5b395c
# Push it to GitHub
git push --set-upstream origin backport/backport-669-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-669-to-2.x.

[opensearch-trigger-bot]

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.11 2.11
# Navigate to the new working tree
cd .worktrees/backport-2.11
# Create a new branch
git switch --create backport/backport-669-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 884ddd04f63fb833153beaaeeb4ffd28ed5b395c
# Push it to GitHub
git push --set-upstream origin backport/backport-669-to-2.11
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport/backport-669-to-2.11.