Update CTI ECS 1.11 fields #113404

nkhristinin · 2021-09-29T12:45:02Z

Summary

Update CTI ECS 1.11 fields
Reaplcing threatintel.indicator with threat.indicator.

How to test

Check master and check that CTI works with ECS < 1.11

Be sure that you have threat intel data in the filebeat index. If you don't have, then try setup threat intel dashboard with data
Copy value from threatintel.indicator.file.hash.sha256 of first hit of you result. (Later refer as ${SHA256_HASH})

GET filebeat-*/_search
{
  "query": {
    "exists": {
      "field": "threatintel.indicator.file.hash.sha256"
    }
  }
}

Create test index

PUT test-filebeat
 {
   "mappings" : {
     "properties" : {
       "@timestamp" : {
         "type" : "date"
       }
     }
   }
 }

Create Custom Rule
Index patterns- test-filebeat
Custom query * : *
Let's create an event in our test-filebeat index

 POST test-filebeat/_doc
 {
   "@timestamp": "${YOUR_DATE}", 
   "file": {
     "hash": {
       "sha256":"${SHA256_HASH}"
     }
   }
 }

In our rule alerts table we should have an alert with enrichment.
Open alert, go to Threat intel tab and check that it has indicators field in Enriched with Threat Intelligence section.
For example indicator.file.hash.sha256 - ${SHA256_HASH}

image

Let's create Indicator Match Rule
Index patterns -test-filebeat
Custom query - *:*
Indicator index patterns - filebeat-*
Indicator index query @timestamp >= "now-30d"
Indicator mapping file.hash.sha256 MATCHES threatintel.indicator.file.hash.sha256
Additional look-back time - 24H (to catch alert from the previous step)
This rule should catch alerts from the previous step. Open alert, go to Threat intel tab and check that it has indicators field in Threat Match Detected section.

image

Check out the branch of this PR

Create one more event in test-filebeat index

 POST test-filebeat/_doc
 {
   "@timestamp": "${YOUR_DATE}", 
   "file": {
     "hash": {
       "sha256":"${SHA256_HASH}"
     }
   }
 }

You should see new alert in Indicator math rule, but you shouldn't have any indicators field on Threat intel -> Threat Match Detected section

image

Check your custom rule, all alert there shouldn't have any Threat Intel enrichments:

image

Let's create the event in filebeat which match the new ECS 1.11:

Create filebeat event (update ${YOUR_DATE} and ${SHA256_HASH})


POST filebeat-8.0.0/_doc
 {
          "agent" : {
            "name" : "Macbook",
            "id" : "8f59569e-00b5-416d-a19f-94026068ec38",
            "type" : "filebeat",
            "ephemeral_id" : "acb5e630-9d79-497b-9627-080320b30158",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "malwarebazaar"
          },
          "threat" : {
            "malwarebazaar" : {
              "intelligence" : {
                "downloads" : 0,
                "uploads" : 1
              },
              "tags" : [
                "BitRAT",
                "DEU",
                "exe",
                "geo",
                "RAT"
              ],
              "anonymous" : 0,
              "dhash_icon" : "f0b07090cb6468e2",
              "code_sign" : [ ]
            },
            "indicator" : {
              "geo" : {
                "country_iso_code" : "US"
              },
              "first_seen" : "2021-09-27T08:09:33.000Z",
              "file" : {
                "extension" : "exe",
                "size" : 23552,
                "mime_type" : "application/x-dosexec",
                "pe" : {
                  "imphash" : "f34d5f2d4577ed6d9ceec516c1f5a744"
                },
                "name" : "XEROX-ZAHLUNG27.092021.exe",
                "hash" : {
                  "sha1" : "dd0834cdb49932a99491bd59200510c023ae9a5e",
                  "sha384" : "0e4a5d29f00becdc792e94983e812b07aa4c8a636f8b87d37bbba5e0d54cd0f37f365b3b6a827002e71292b8c0aeba1d",
                  "sha256" : "${SHA256_HASH}",
                  "tlsh" : "T188B25476BFD80D55CCF70A3A1CD2BD344D3679CE56A18ABF6888E23E7F101A00969791",
                  "ssdeep" : "384:NkSYhMbVs30Q1TiTPoc3qA5WuDhO4+99y/CMzqTLwzfWntw+synvd3c:SMbVpScXf8c",
                  "md5" : "69c3b568206568200980e419fa392afd"
                },
                "elf" : { }
              },
              "provider" : "abuse_ch",
              "type" : "file"
            }
          },
          "tags" : [
            "threatintel-malwarebazaar",
            "forwarded"
          ],
          "input" : {
            "type" : "httpjson"
          },
          "@timestamp" : "${YOUR_DATE}",
          "ecs" : {
            "version" : "1.10.0"
          },
          "related" : {
            "hash" : [
              "69c3b568206568200980e419fa392afd",
              "2b9e8dc39755d37c2a78fd00f6fa57414205648e65ae6a6c76c102c7d78e2e86",
              "384:NkSYhMbVs30Q1TiTPoc3qA5WuDhO4+99y/CMzqTLwzfWntw+synvd3c:SMbVpScXf8c",
              "f34d5f2d4577ed6d9ceec516c1f5a744",
              "T188B25476BFD80D55CCF70A3A1CD2BD344D3679CE56A18ABF6888E23E7F101A00969791"
            ]
          },
          "service" : {
            "type" : "threatintel"
          },
          "event" : {
            "ingested" : "2021-09-27T08:09:47.206498Z",
            "created" : "2021-09-27T08:09:43.918Z",
            "kind" : "enrichment",
            "module" : "threatintel",
            "category" : "threatintel",
            "type" : "indicator",
            "dataset" : "threat.malwarebazaar"
          }
        }

Check your custom rule, all alerts there shouldn't have Threat Intel tab with data and there indicators field in Threat Match Detected section.

image

Let's update our Indicator Math rule like that:

image

Create one more event in test-filebeat index

 POST test-filebeat/_doc
 {
   "@timestamp": "${YOUR_DATE}", 
   "file": {
     "hash": {
       "sha256":"${SHA256_HASH}"
     }
   }
 }

Check the last alert for the Indicator Math rule, it should have 2 enrichments in Threat Match Detected section.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-09-30T18:11:53Z

Pinging @elastic/security-solution (Team: SecuritySolution)

nkhristinin · 2021-09-30T18:12:23Z

@elasticmachine merge upstream

kibanamachine · 2021-09-30T23:01:38Z

⏳ Build in-progress, with failures

continuous-integration/kibana-ci/pull-request
Commit: 2151556
Storybooks Preview
This comment will update when the build is complete

Failed CI Steps

Execute xpack-securitySolutionCypressChrome

History

💔 Build #156881 failed 7514ad6
💔 Build #156411 failed d45bfab

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ecezalp · 2021-10-04T13:11:30Z

@elasticmachine merge upstream

ecezalp

LGTM

I was able to confirm that once I check out the PR and create a new alert, there were no investigation time enrichments on it. Once I posted some data to my filebeat index with a matching sha256 hash under the "threat" field, I was able to see the enrichment as expected.

rylnd

Code looks good, although I have not tested this e2e as Ece did. There are some cypress failures around the enrichment e2e tests; I haven't verified whether these are legitimate failures or just outdated tests, but let me know if you need help debugging or fixing them!

x-pack/plugins/security_solution/common/cti/constants.ts

nkhristinin · 2021-10-06T09:43:43Z

@elasticmachine merge upstream

nkhristinin · 2021-10-11T11:50:21Z

@elasticmachine merge upstream

nkhristinin · 2021-10-12T09:58:52Z

@elasticmachine merge upstream

nkhristinin · 2021-10-13T13:15:48Z

@elasticmachine merge upstream

nkhristinin · 2021-10-13T20:47:51Z

@elasticmachine merge upstream

rylnd · 2021-10-13T22:11:19Z

x-pack/test/security_solution_cypress/es_archives/threat_indicator2/data.json

@@ -56,7 +56,7 @@
        "module": "threatintel",
        "category": "threat",
        "type": "indicator",
-        "dataset": "threatintel.abusemalware"
+        "dataset": "ti_abusech.malware"


It doesn't look like this archive was updated; I think that's at least part of the cause of Cypress failures.

kibanamachine · 2021-10-14T17:11:34Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.6MB	4.6MB	-162.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	104.1KB	104.1KB	-5.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-10-14T18:24:25Z

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 113404

* Update threatintel to threat * Remove CTI mappings * Update CTI_DATASET_KEY_MAP * Update default threat index * Change mapping to dataset * Fix tests * Fix tests * Fix test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/security_solution/server/plugin.ts

* Revert "Update CTI ECS 1.11 fields (#113404)"

* Update threatintel to threat * Remove CTI mappings * Update CTI_DATASET_KEY_MAP * Update default threat index * Change mapping to dataset * Fix tests * Fix tests * Fix test Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 1929f88)

Update threatintel to threat

d45bfab

nkhristinin requested review from rylnd and ecezalp September 30, 2021 18:07

nkhristinin added auto-backport Deprecated - use backport:version if exact versions are needed release_note:breaking Team: CTI Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.16.0 v8.0.0 labels Sep 30, 2021

nkhristinin marked this pull request as ready for review September 30, 2021 18:11

nkhristinin requested a review from a team as a code owner September 30, 2021 18:11

Merge branch 'master' into cti-field-update

7514ad6

nkhristinin force-pushed the cti-field-update branch from 1988946 to 7514ad6 Compare September 30, 2021 19:04

Remove CTI mappings

2151556

Merge branch 'master' into cti-field-update

382472e

ecezalp approved these changes Oct 4, 2021

View reviewed changes

rylnd approved these changes Oct 4, 2021

View reviewed changes

x-pack/plugins/security_solution/common/cti/constants.ts Outdated Show resolved Hide resolved

Merge branch 'master' into cti-field-update

9319728

kibanamachine and others added 3 commits October 11, 2021 07:50

Merge branch 'master' into cti-field-update

d420347

Update CTI_DATASET_KEY_MAP

bc1981d

Update default threat index

44dfae9

kibanamachine and others added 2 commits October 12, 2021 05:58

Merge branch 'master' into cti-field-update

7850d6d

Change mapping to dataset

bec5e14

kibanamachine and others added 2 commits October 13, 2021 09:15

Merge branch 'master' into cti-field-update

379a300

Fix tests

752c80a

Merge branch 'master' into cti-field-update

beb73f5

rylnd reviewed Oct 13, 2021

View reviewed changes

nkhristinin added 2 commits October 14, 2021 12:34

Fix tests

3c5a41e

Fix test

3019082

nkhristinin merged commit 1929f88 into elastic:master Oct 14, 2021

nkhristinin mentioned this pull request Oct 14, 2021

[7.x] Update CTI ECS 1.11 fields (#113404) #115088

Merged

ecezalp added a commit that referenced this pull request Oct 29, 2021

Revert "Update CTI ECS 1.11 fields (#113404)" (#116716)

bce1e80

* Revert "Update CTI ECS 1.11 fields (#113404)"

nkhristinin added v7.16.0 and removed v7.16.0 labels Nov 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update CTI ECS 1.11 fields #113404

Update CTI ECS 1.11 fields #113404

nkhristinin commented Sep 29, 2021 •

edited

Loading

elasticmachine commented Sep 30, 2021

nkhristinin commented Sep 30, 2021

kibanamachine commented Sep 30, 2021

ecezalp commented Oct 4, 2021

ecezalp left a comment

rylnd left a comment

nkhristinin commented Oct 6, 2021

nkhristinin commented Oct 11, 2021

nkhristinin commented Oct 12, 2021

nkhristinin commented Oct 13, 2021

nkhristinin commented Oct 13, 2021

rylnd Oct 13, 2021

kibanamachine commented Oct 14, 2021

kibanamachine commented Oct 14, 2021

Update CTI ECS 1.11 fields #113404

Update CTI ECS 1.11 fields #113404

Conversation

nkhristinin commented Sep 29, 2021 • edited Loading

Summary

How to test

Check master and check that CTI works with ECS < 1.11

Check out the branch of this PR

Checklist

For maintainers

elasticmachine commented Sep 30, 2021

nkhristinin commented Sep 30, 2021

kibanamachine commented Sep 30, 2021

⏳ Build in-progress, with failures

Failed CI Steps

History

ecezalp commented Oct 4, 2021

ecezalp left a comment

Choose a reason for hiding this comment

rylnd left a comment

Choose a reason for hiding this comment

nkhristinin commented Oct 6, 2021

nkhristinin commented Oct 11, 2021

nkhristinin commented Oct 12, 2021

nkhristinin commented Oct 13, 2021

nkhristinin commented Oct 13, 2021

rylnd Oct 13, 2021

Choose a reason for hiding this comment

kibanamachine commented Oct 14, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

kibanamachine commented Oct 14, 2021

💔 Backport failed

nkhristinin commented Sep 29, 2021 •

edited

Loading