[AWS] Add the Security Hub data streams #3553

darshan-elastic · 2022-06-21T14:38:41Z

What does this PR do?

Added two data streams.
Added data collection logic to the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
Added dashboard.
Added test for pipeline for the data streams.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws directory.
Run the following command to run tests.

elastic-package test asset -d securityhub_findings,securityhub_insights
elastic-package test static -d securityhub_findings,securityhub_insights
elastic-package test system -d securityhub_findings,securityhub_insights
elastic-package test pipeline -d securityhub_findings,securityhub_insights

Screenshots

Relates System tests fail in packages using policy_templates elastic-package#899
Closes AWS Security Hub #3589

cla-checker-service · 2022-06-21T14:38:46Z

💚 CLA has been signed

elasticmachine · 2022-06-21T14:59:47Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-08-15T20:55:20.750+0000
Duration: 36 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	162
Skipped	2
Total	164

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-06-22T14:06:26Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-06-22T14:31:24Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`13/13`)	💚
Files	92.857% (`13/14`)	👎 -4.275
Classes	92.857% (`13/14`)	👎 -4.275
Methods	84.167% (`202/240`)	👎 -5.059
Lines	95.686% (`5190/5424`)	👍 4.944
Conditionals	100.0% (`0/0`)	💚

flash1293 · 2022-06-28T15:28:14Z

It seems like one panel on the dashboard is using a non-existing field name (maybe it got renamed):

Also, is there a special reason most of the "event distribution" charts are pies, but some of them are bar charts?

As there are a lot of similar panels and the dashboard gets pretty long, would it make sense to split it up into multiple dashboards and link it up via a markdown visualization so the user can "select" the charts they are interested in instead of scrolling through a long dashboard? This really depends on how it's meant to be used though.

andrewkroh · 2022-06-29T02:26:43Z

packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs

+      value_type: json
+  - set:
+      target: header.Authorization
+      value: '[[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate (now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate (now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate (now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate (now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.amazonaws.com\n" (sprintf "x-amz-date:%s\n\n" (formatDate (now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'


There are multiple calls to now in here that need to match the already set X-Amz-Date header down to the second. This is a race condition. Instead of calling formatDate now "20060102T150405Z" twice, can you replace that with an access to the header value? Like (index .header "X-Amz-Date") assuming the header map is available in this context.

@marc-gr @P1llus Have you come across this type of issue before? I was thinking it would useful to have a "now"-like function that returns the same time for all invocations within the context of a given request. This would allow all "now"s used in a request to be the same.

Or another option would to provide a generic scratch pad where you can store the current time once and then reference it later. A similar concept is used in consul-template https://github.com/hashicorp/consul-template/blob/main/docs/templating-language.md#scratch. The idea would be like

request.transforms: - set: target: scratch.RequestTime value: '[[ now ]]' - set: target: header.X-Amz-Date value: '[[formatDate (.scratch.RequestTime) "20060102T150405Z"]]'

This does not fully fix the problem but it does reduce the number of now invocations to one (credit to @efd6). You can declare variable in the template then use the variable instead of calling now. Like [[$now := now]] then replace the now calls with $now.

Do we have any documentation for this?

Assuming by "this" you mean declaring variables in templates, then you can look at https://pkg.go.dev/text/template#hdr-Variables. Variable declarations in templates is something built in to Go text/template.

andrewkroh · 2022-06-29T02:53:51Z

packages/aws/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 1.0.0
 name: aws
 title: AWS
-version: 1.16.6
+version: 1.17.0
 license: basic
 description: Collect logs and metrics from Amazon Web Services with Elastic Agent.
 type: integration


Leaving this comment here because it's close to the kibana.version. What are the Agent version requirements? Are there newly introduced httpjson template functions required for this?

Yes newly introduced httpjson template functions is required for this. So we have mentioned it in Readme that 8.4.0 version will be required.

It looks like this did not happen.

andrewkroh · 2022-06-29T17:43:10Z

packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs

+      target: header.Authorization
+  - set:
+      target: header.Authorization
+      value: '[[(sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate (now) "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate (now) "20060102"))) "{{aws_region}}")) "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n" (formatDate (now) "20060102T150405Z") "\n" (sprintf "%s/%s\n" (formatDate (now) "20060102") "{{aws_region}}/securityhub/aws4_request") (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.amazonaws.com\n" (sprintf "x-amz-date:%s\n\n" (formatDate (now) "20060102T150405Z")) "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]'


For reviewers, I formatted this for to help make sense of it.

[[ (sprintf "AWS4-HMAC-SHA256 Credential={{access_key_id}}/%s/{{aws_region}}/securityhub/aws4_request, SignedHeaders=host;x-amz-date, Signature=%s" (formatDate now "20060102" ) (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac "sha256" "AWS4{{secret_access_key}}" (formatDate now "20060102" ) ) ) "{{aws_region}}" ) ) "securityhub" ) ) "aws4_request" ) ) "AWS4-HMAC-SHA256\n" (formatDate now "20060102T150405Z" ) "\n" (sprintf "%s/%s\n" (formatDate now "20060102" ) "{{aws_region}}/securityhub/aws4_request" ) (hash "sha256" "POST\n" "/findings\n" "\n" "host:securityhub.{{aws_region}}.amazonaws.com\n" (sprintf "x-amz-date:%s\n\n" (formatDate now "20060102T150405Z" ) ) "host;x-amz-date\n" (hash "sha256" (sprintf "%s" .body ) ) ) ) ) ]]

andrewkroh

I think this needs a system test so that we can test indexing. With only pipeline tests we can't catch various mapping issues.

packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs

packages/aws/data_stream/securityhub_findings/manifest.yml

packages/aws/data_stream/securityhub_insights/elasticsearch/ingest_pipeline/default.yml

...a_stream/securityhub_findings/_dev/test/pipeline/test-securityhub-findings.log-expected.json

packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml

packages/aws/data_stream/securityhub_findings/sample_event.json

packages/aws/_dev/build/docs/securityhub.md

darshan-elastic · 2022-07-05T07:12:46Z

/test

darshan-elastic · 2022-07-05T16:43:45Z

@andrewkroh We are facing below issue when we are updating version to 8.3.0

So can we do the changes in Redshift data stream?

andrewkroh · 2022-07-07T00:28:15Z

We are facing below issue when we are updating version to 8.3.0

Let's back out the 8.3.0 ECS change and handle the upgrade separately as to not mix the redshift updates with the security hub additions.

darshan-elastic · 2022-07-13T18:22:11Z

/test

andrewkroh · 2022-08-02T11:50:02Z

@elastic/obs-cloud-monitoring, can you please take a look at this PR? As a codeowner for aws you are a required reviewer.

kaiyan-sheng

looks good to me

Add the securityhub data streams

25b0896

darshan-elastic force-pushed the aws-security-hub branch from faf7157 to 25b0896 Compare June 21, 2022 14:54

darshan-elastic and others added 2 commits June 22, 2022 11:23

Merge branch 'main' into aws-security-hub

134563c

Update the dashboard version to resolve CI failure

6360da5

vinit-chauhan added enhancement New feature or request Team:Security-External Integrations Integration:aws AWS labels Jun 22, 2022

darshan-elastic marked this pull request as ready for review June 22, 2022 14:06

darshan-elastic requested review from a team as code owners June 22, 2022 14:06

darshan-elastic self-assigned this Jun 23, 2022

andrewkroh reviewed Jun 29, 2022

View reviewed changes

packages/aws/_dev/build/docs/securityhub.md Outdated Show resolved Hide resolved

darshan-elastic and others added 3 commits July 4, 2022 22:21

Update as per review comments

c4164d7

Merge branch 'main' into aws-security-hub

28c8fb9

Update the dashboard version to resolve CI failure

822b7b9

darshan-elastic and others added 2 commits July 5, 2022 18:42

Add system test

ab4528f

Merge branch 'elastic:main' into aws-security-hub

f539b87

Update system test with new logs and certificate

77407a6

Skip System test and Update Data collection

d9714da

update manifest

eb06e5b

darshan-elastic force-pushed the aws-security-hub branch from 9a28dd2 to eb06e5b Compare July 13, 2022 18:44

darshan-elastic and others added 6 commits July 14, 2022 00:17

Merge branch 'main' into aws-security-hub

fa8777c

Add on_failure in date, convert and uri_parts processor

d0c1e24

Merge branch 'main' into aws-security-hub

ec19418

Merge branch 'main' into aws-security-hub

ef34c25

Merge branch 'main' into aws-security-hub

3f8fb39

Merge branch 'main' into aws-security-hub

48435d4

andrewkroh approved these changes Aug 2, 2022

View reviewed changes

andrewkroh added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Aug 2, 2022

kaiyan-sheng approved these changes Aug 9, 2022

View reviewed changes

darshan-elastic and others added 2 commits August 16, 2022 02:18

Merge branch 'main' into aws-security-hub

35d5695

Changed version to 1.21.0 to resolve conflicts

9fed399

andrewkroh merged commit e18a067 into elastic:main Aug 15, 2022

efd6 mentioned this pull request Sep 5, 2022

aws: README should indicate that filebeat 8.4 is a requirement #4123

Closed

jamiehynds mentioned this pull request Sep 6, 2022

AWS Security Hub #3589

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AWS] Add the Security Hub data streams #3553

[AWS] Add the Security Hub data streams #3553

darshan-elastic commented Jun 21, 2022 •

edited

Loading

cla-checker-service bot commented Jun 21, 2022 •

edited

Loading

elasticmachine commented Jun 21, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 22, 2022

elasticmachine commented Jun 22, 2022 •

edited

Loading

flash1293 commented Jun 28, 2022 •

edited

Loading

andrewkroh Jun 29, 2022

andrewkroh Jun 29, 2022

andrewkroh Jul 7, 2022

darshan-elastic Jul 11, 2022 •

edited

Loading

andrewkroh Jul 11, 2022

andrewkroh Jun 29, 2022

darshan-elastic Jul 4, 2022

efd6 Sep 5, 2022

andrewkroh Jun 29, 2022

andrewkroh left a comment

darshan-elastic commented Jul 5, 2022

darshan-elastic commented Jul 5, 2022 •

edited

Loading

andrewkroh commented Jul 7, 2022

darshan-elastic commented Jul 13, 2022

andrewkroh commented Aug 2, 2022 •

edited

Loading

kaiyan-sheng left a comment

[AWS] Add the Security Hub data streams #3553

[AWS] Add the Security Hub data streams #3553

Conversation

darshan-elastic commented Jun 21, 2022 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Screenshots

cla-checker-service bot commented Jun 21, 2022 • edited Loading

elasticmachine commented Jun 21, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jun 22, 2022

elasticmachine commented Jun 22, 2022 • edited Loading

🌐 Coverage report

flash1293 commented Jun 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darshan-elastic Jul 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

darshan-elastic commented Jul 5, 2022

darshan-elastic commented Jul 5, 2022 • edited Loading

andrewkroh commented Jul 7, 2022

darshan-elastic commented Jul 13, 2022

andrewkroh commented Aug 2, 2022 • edited Loading

kaiyan-sheng left a comment

Choose a reason for hiding this comment

darshan-elastic commented Jun 21, 2022 •

edited

Loading

cla-checker-service bot commented Jun 21, 2022 •

edited

Loading

elasticmachine commented Jun 21, 2022 •

edited

Loading

elasticmachine commented Jun 22, 2022 •

edited

Loading

flash1293 commented Jun 28, 2022 •

edited

Loading

darshan-elastic Jul 11, 2022 •

edited

Loading

darshan-elastic commented Jul 5, 2022 •

edited

Loading

andrewkroh commented Aug 2, 2022 •

edited

Loading