Add support for threat.feed.name #120250

nkhristinin · 2021-12-02T17:44:06Z

Summary

We add supporting of threat.feed.name for alerts flyout and timeline view.

Context

Previously we were showing indicator. provider as source name (provider) of enrichment.

e.g

file.hash.sha256 ${hashValue} from AbuseCH

In new fleet TI integrations and filebeat 8.x the purpose of indicator. provider field was changed.

threat.indicator.provider - goes to the actual person/entity that submitted the indicator. As an example, this could be a threat researcher (“Cryptolaemus1”, “lrz_urlhaus”, “tammeto”, etc.)

And to be able to show previous source name was introduced threat.feed.name

Solution

Currently, we have 2 different ways of building enrichment on the backend:

for real-time investigation enrichments (query all fields from threat, so threat.feed just will be in enrichment automatically)
for IM rule (we build it manually, we need to pass feed into enrichment)

The frontend has logic about what to show for source name:

if enrichment has feed.name - show it

How to tests

Be sure that on alerts flyout and timeline we have the right source name

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios
(https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))

rylnd

Code looks good, but I don't think conflating provider and feed is a good approach here; we should find a way to display them as distinct concepts.

rylnd · 2021-12-02T18:07:59Z

x-pack/plugins/timelines/public/mock/mock_timeline_data.ts

@@ -1107,6 +1107,9 @@ export const mockTimelineData: TimelineItem[] = [
              field: ['source.ip'],
              type: ['ip'],
            },
+            feed: {
+              name: ['indicator_provider'],


A more realistic example here might help distinguish it from provider 😉

Agree, it's value used here, I think it make more sense

rylnd · 2021-12-02T18:15:20Z

x-pack/plugins/security_solution/public/common/components/event_details/cti_details/helpers.tsx

@@ -81,7 +82,9 @@ export const getEnrichmentIdentifiers = (enrichment: CtiEnrichment): CtiEnrichme
  field: getEnrichmentValue(enrichment, MATCHED_FIELD),
  value: getEnrichmentValue(enrichment, MATCHED_ATOMIC),
  type: getEnrichmentValue(enrichment, MATCHED_TYPE),
-  provider: getShimmedIndicatorValue(enrichment, PROVIDER),
+  provider:


I don't think we want to conflate these two values as they're different concepts. Feed name is the data feed containing the indicator, while the provider is the entity that discovered/generated the indicator.

So, do you suggest that we need to keep only FEED_NAME here?

It's basically will be the same as old behaviour

rylnd · 2021-12-02T18:15:59Z

...curity_solution/public/timelines/components/timeline/body/renderers/cti/threat_match_row.tsx

@@ -46,7 +47,7 @@ export const ThreatMatchRow = ({
    contextId,
    eventId,
    indicatorReference: get(data, REFERENCE)[0] as string | undefined,
-    indicatorProvider: get(data, PROVIDER)[0] as string | undefined,
+    indicatorProvider: (get(data, FEED_NAME)[0] ?? get(data, PROVIDER)[0]) as string | undefined,


Ditto on the conflation here.

rylnd · 2021-12-02T19:28:45Z

x-pack/plugins/security_solution/public/common/components/event_details/cti_details/helpers.tsx

@@ -81,7 +81,7 @@ export const getEnrichmentIdentifiers = (enrichment: CtiEnrichment): CtiEnrichme
  field: getEnrichmentValue(enrichment, MATCHED_FIELD),
  value: getEnrichmentValue(enrichment, MATCHED_ATOMIC),
  type: getEnrichmentValue(enrichment, MATCHED_TYPE),
-  provider: getShimmedIndicatorValue(enrichment, PROVIDER),
+  provider: getShimmedIndicatorValue(enrichment, FEED_NAME),


I think we should rename this to feedName to remain consistent. I see no reason not to keep returning provider as well:

Suggested change

provider: getShimmedIndicatorValue(enrichment, FEED_NAME),

feedName: getShimmedIndicatorValue(enrichment, FEED_NAME),

provider: getShimmedIndicatorValue(enrichment, PROVIDER),

rylnd · 2021-12-02T19:30:27Z

...curity_solution/public/timelines/components/timeline/body/renderers/cti/threat_match_row.tsx

@@ -46,7 +47,7 @@ export const ThreatMatchRow = ({
    contextId,
    eventId,
    indicatorReference: get(data, REFERENCE)[0] as string | undefined,
-    indicatorProvider: get(data, PROVIDER)[0] as string | undefined,
+    indicatorProvider: get(data, FEED_NAME)[0] as string | undefined,


If we're changing the semantic meaning of this prop we should change its name as well.

Suggested change

indicatorProvider: get(data, FEED_NAME)[0] as string | undefined,

feedName: get(data, FEED_NAME)[0] as string | undefined,

nkhristinin · 2021-12-03T17:08:16Z

@elasticmachine merge upstream

ecezalp · 2021-12-03T18:37:00Z

.../security_solution/public/common/components/event_details/cti_details/enrichment_summary.tsx

-            </EnrichmentFieldProvider>
+          {feedName && (
+            <EnrichmentFieldFeedName>
+              {i18n.PROVIDER_PREPOSITION} {feedName}


we could probably rename i18n.PROVIDER_PREPOSITION

ecezalp · 2021-12-03T18:39:58Z

...lugins/security_solution/public/common/components/event_details/cti_details/helpers.test.tsx

+        'matched.atomic': ['matched atomic'],
+        'matched.type': ['matched type'],
+        'feed.name': ['feed name'],
+        'indicator.provider': ['provider name'],


do we use the indicator.provider enrichment identifier in Kibana? trying to understand if we need to leave it here considering it's a mapped field just like most others and it would work on the timeline anyway. if it's not used on the row renderer UI, there might not be an urgent reason to keep it here

ecezalp · 2021-12-03T18:42:23Z

..._solution/server/lib/detection_engine/signals/threat_mapping/enrich_signal_threat_matches.ts

    return {
      indicator,
+      feed,


I thought we only wanted and mapped the threat.feed.name field for the alert, so I think we could consider just putting feed.name here instead of the feed obj

ecezalp · 2021-12-03T18:50:20Z

...y_solution/public/common/components/event_details/cti_details/enrichment_accordion_group.tsx

@@ -98,7 +98,7 @@ const EnrichmentAccordion: React.FC<{
      key={accordionId}
      initialIsOpen={true}
      arrowDisplay="right"
-      buttonContent={<EnrichmentButtonContent field={field} provider={provider} value={value} />}
+      buttonContent={<EnrichmentButtonContent field={field} feedName={feedName} value={value} />}


it looks like we are completely removing the provider string from the UI - I thought we wanted to display provider if there was no feedName for the purpose of identifying the source of a feed. I understand previously raised concerns around conflating information , but we also know that it's very unlikely for the users to have threat.feed.name fields populated at least in 8.0 (as threat.feed.name is not in the ECS specification). IMHO we can do something like feedName={feedName ?? provider} in the UI context chatted w/Ryland, it looks like this is something we could get product feedback on - since threat intel integrations from 8.0 are all designed to ship the threat.feed.name field, this is less of a concern

nkhristinin · 2021-12-06T07:27:48Z

@elasticmachine merge upstream

kibana-ci · 2021-12-06T15:41:06Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	2.7MB	2.7MB	+132.0B
`observability`	362.2KB	362.3KB	+132.0B
`securitySolution`	4.6MB	4.6MB	-65.0B
total			+199.0B

History

💔 Build #11223 failed b74d19b
💔 Build #11200 failed 34dec8c
💔 Build #11196 failed 8cb9c71
💔 Build #11033 failed 8c3b3a5
💔 Build #10955 failed e1ecea8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cavokz

LGTM

rylnd

LGTM

ecezalp

LGTM

* Add support for threat.feed.name * fix cy tests * Remove provider field * fix tests * Provider to feed.name * Fix tests * Fix tests * fix comments * Fix i18n * fix type * Fix types * fix tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…-chromium-before-compiling-pdf * 'main' of github.com:elastic/kibana: (121 commits) FTR should use the new kibana_system user (elastic#120436) [Lens] Temporarily exclude Mosaic/Waffle from the suggestions list (elastic#120488) [Reporting] Fix slow CSV with large max size bytes (elastic#120365) [CTI] Threat Intel Card on Overview page needs to accommodate Fleet TI integrations - (elastic#120459) [Dashboard] Added KibanaThemeProvider. (elastic#120122) add more rule_registry unit tests (elastic#120323) [Lens] update xpack.lens.pie.smallValuesWarningMessage text (elastic#120478) [Fleet] Simplified package policy create API, enriching default values (elastic#119739) mock `elastic-apm-node` in `@kbn/test` jest preset (elastic#120324) [RAC] Rename occurrences of alert_type to rule_type in Infra (elastic#120455) [Security Solutions] Removes tech debt of exporting all from linter rule for timeline plugin (elastic#120437) [Workplace Search] Add API Keys view to replace Access tokens (elastic#120147) [Security Solutions] Removes tech debt of exporting all from linter rule for cases plugin in the server section (elastic#120411) Add support for threat.feed.name (elastic#120250) [Rule Registry] Rewrite APM registry rules for Observability (elastic#117740) [docs] Fix artifact layout formatting (elastic#119386) [Workplace Search] Add a technical preview of connecting GitHub via GitHub apps (elastic#119764) add osquery notes for 7.16 (elastic#120407) chore(NA): splits types from code on @kbn/docs-utils (elastic#120533) [Reporting] Decouple screenshotting plugin from the reporting (elastic#120110) ... # Conflicts: # x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.test.ts # x-pack/plugins/reporting/server/browsers/chromium/driver_factory/index.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.test.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.ts # x-pack/plugins/reporting/server/test_helpers/create_mock_browserdriverfactory.ts

nkhristinin · 2021-12-07T10:57:43Z

How to test:

Intro

Run elasticsearch and kibana

Filebeat

Pull the latest changes of beats
Add filebeat data as described here

Create a custom test-index

PUT test-index
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "file": {
         "properties": {
           "hash": {
             "properties": {
               "md5": {
                 "type" : "keyword",
                  "ignore_above" : 1024
               }
             }
           }
         }
      }
  }
}
}

Create an IM rule

Index patterns: **test-index**
Custom query: ***:***
Indicator index patterns: **filebeat-***
Indicator mapping: **file.hash.md5 MATCHES threat.indicator.file.hash.md5**
Indicator index query: **@timestamp >= "now-30d"**
Runs every: **1m**
Additional look-back time: **5h**

Let's make a match

Find in FIlebeat the sha256 hash value and copy it

GET filebeat-8*/_search
{
  "query": {
    "exists": {
      "field": "threat.indicator.file.hash.md5"
    }
  },
  "_source": ["threat.indicator.file.hash.md5"]
}

Send the document into our test-index

POST test-index/_doc
{
  "@timestamp": ${TIMESTAMP_VALUE},
  "file": {
    "hash": {
      "md5": "${MD5_FROM_FILEBEAT}"
    }
  }
}

Go to your rule, you should see 1 alert.
Check the Threat Intel tab, it should have the feed name

Go to timeline from the alert row, you also should have the feed name:

* Add support for threat.feed.name * fix cy tests * Remove provider field * fix tests * Provider to feed.name * Fix tests * Fix tests * fix comments * Fix i18n * fix type * Fix types * fix tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* Add support for threat.feed.name * fix cy tests * Remove provider field * fix tests * Provider to feed.name * Fix tests * Fix tests * fix comments * Fix i18n * fix type * Fix types * fix tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

nkhristinin added 2 commits December 2, 2021 14:53

Add support for threat.feed.name

7eea4fc

fix cy tests

940c52f

nkhristinin requested review from rylnd and ecezalp December 2, 2021 17:44

nkhristinin requested review from a team as code owners December 2, 2021 17:44

nkhristinin added release_note:fix Team: CTI v8.0.0 labels Dec 2, 2021

rylnd requested changes Dec 2, 2021

View reviewed changes

nkhristinin added 2 commits December 2, 2021 19:43

Remove provider field

b73208e

fix tests

f523c5f

rylnd reviewed Dec 2, 2021

View reviewed changes

nkhristinin added 3 commits December 3, 2021 15:05

Provider to feed.name

30bd930

Fix tests

e1ecea8

Fix tests

c48b0d2

Merge branch 'main' into threat-feed-name

8c3b3a5

ecezalp reviewed Dec 3, 2021

View reviewed changes

kibanamachine and others added 6 commits December 6, 2021 02:27

Merge branch 'main' into threat-feed-name

e723473

fix comments

ca717df

Fix i18n

8cb9c71

fix type

34dec8c

Fix types

b74d19b

fix tests

6ec99ed

cavokz approved these changes Dec 6, 2021

View reviewed changes

rylnd approved these changes Dec 6, 2021

View reviewed changes

ecezalp self-requested a review December 6, 2021 18:12

ecezalp approved these changes Dec 6, 2021

View reviewed changes

andrew-goldstein approved these changes Dec 6, 2021

View reviewed changes

ecezalp merged commit a8405fe into elastic:main Dec 7, 2021

nkhristinin mentioned this pull request Dec 7, 2021

[8.0] Add support for threat.feed.name (#120250) #120584

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for threat.feed.name #120250

Add support for threat.feed.name #120250

nkhristinin commented Dec 2, 2021 •

edited

Loading

rylnd left a comment

rylnd Dec 2, 2021

nkhristinin Dec 2, 2021 •

edited

Loading

rylnd Dec 2, 2021

nkhristinin Dec 2, 2021

rylnd Dec 2, 2021

rylnd Dec 2, 2021

rylnd Dec 2, 2021

nkhristinin commented Dec 3, 2021

ecezalp Dec 3, 2021

ecezalp Dec 3, 2021

ecezalp Dec 3, 2021

rylnd Dec 3, 2021

ecezalp Dec 3, 2021 •

edited

Loading

nkhristinin commented Dec 6, 2021

kibana-ci commented Dec 6, 2021

cavokz left a comment •

edited

Loading

rylnd left a comment

ecezalp left a comment

nkhristinin commented Dec 7, 2021

	provider: getShimmedIndicatorValue(enrichment, FEED_NAME),
	feedName: getShimmedIndicatorValue(enrichment, FEED_NAME),
	provider: getShimmedIndicatorValue(enrichment, PROVIDER),

	indicatorProvider: get(data, FEED_NAME)[0] as string \| undefined,
	feedName: get(data, FEED_NAME)[0] as string \| undefined,

Add support for threat.feed.name #120250

Add support for threat.feed.name #120250

Conversation

nkhristinin commented Dec 2, 2021 • edited Loading

Summary

Context

Solution

How to tests

Checklist

rylnd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkhristinin Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkhristinin commented Dec 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecezalp Dec 3, 2021 • edited Loading

Choose a reason for hiding this comment

nkhristinin commented Dec 6, 2021

kibana-ci commented Dec 6, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

cavokz left a comment • edited Loading

Choose a reason for hiding this comment

rylnd left a comment

Choose a reason for hiding this comment

ecezalp left a comment

Choose a reason for hiding this comment

nkhristinin commented Dec 7, 2021

How to test:

Intro

Filebeat

Create a custom test-index

Create an IM rule

Let's make a match

nkhristinin commented Dec 2, 2021 •

edited

Loading

nkhristinin Dec 2, 2021 •

edited

Loading

ecezalp Dec 3, 2021 •

edited

Loading

cavokz left a comment •

edited

Loading