New action type - D3 security SOAR

[d3-naiji]

Summary

Introduces a new built-in action type and sends the ElasticSearch alerts to D3 Security SOAR for incident response, workflow automation and deep investigation. This integrated alert action allows you to create Events or trigger Playbook workflow actions automatically in the D3 Security SOAR platform.

The new action type and the action connector with D3 Security SOAR are as follows:

and action params form available in Create Alert form:

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

For maintainers

v8.3.3
Release_note:feature

[kibanamachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[cla-checker-service]

💚 CLA has been signed

[cnasikas]

Hi @d3-naiji! Thank you so much for your contribution. I want to ask you if you want to support the connector in Cases? I am asking because Cases use a different flow that alerts. You can create an incident, update the incident, add comments, etc. To support both cases, the code needs to also consider the case user flow.

[pmuellr]

Thanks for the PR, @d3-naiji! A couple things we'll need, beyond questions that Christos mentioned ^^^, if the desire is to ship this as-is vs as a case connector.

• the CLA must be signed; do that here: https://www.elastic.co/contributor-agreement - it's the same link from the build comment ^^^
• we'll need some "function tests"

The last connector we added was xMatters in PR #122357 ; the "function test" I mentioned above are in the x-pack/test/alerting_api_integration directory, and we'd want the same kind of tests that were added for xMatters, so hopefully you can follow that PR to figure out what to do. Don't hesitate to ask us more questions!

[d3-naiji]

cnasikas
Hi cnasikas, thanks for the quick review. I think we can remove the "Case" tag for now. And we do plan to support cases in the future.

[d3-naiji]

Thanks for the PR, @d3-naiji! A couple things we'll need, beyond questions that Christos mentioned ^^^, if the desire is to ship this as-is vs as a case connector.

• the CLA must be signed; do that here: https://www.elastic.co/contributor-agreement - it's the same link from the build comment ^^^
• we'll need some "function tests"

The last connector we added was xMatters in PR #122357 ; the "function test" I mentioned above are in the x-pack/test/alerting_api_integration directory, and we'd want the same kind of tests that were added for xMatters, so hopefully you can follow that PR to figure out what to do. Don't hesitate to ask us more questions!

Hi @pmuellr , thanks for the information. We have signed CLA already. And will add the function test in the next commit.
An update for Christos's question: We will not make a case connector for now and will remove the "Case" tag. Will ping you when it's complete.

[cnasikas]

Thanks for the PR, @d3-naiji! A couple things we'll need, beyond questions that Christos mentioned ^^^, if the desire is to ship this as-is vs as a case connector.

• the CLA must be signed; do that here: https://www.elastic.co/contributor-agreement - it's the same link from the build comment ^^^
• we'll need some "function tests"

The last connector we added was xMatters in PR #122357 ; the "function test" I mentioned above are in the x-pack/test/alerting_api_integration directory, and we'd want the same kind of tests that were added for xMatters, so hopefully you can follow that PR to figure out what to do. Don't hesitate to ask us more questions!

Hi @pmuellr , thanks for the information. We have signed CLA already. And will add the function test in the next commit. An update for Christos's question: We will not make a case connector for now and will remove the "Case" tag. Will ping you when it's complete.

Thank you @d3-naiji. I would like to let you know that we have created a framework to aid developers in creating connectors in case you are interested. You can find the documentation here https://github.com/elastic/kibana/tree/main/x-pack/plugins/actions/server/sub_action_framework. I would suggest using it if you want to be "case" compatible in the future.

[d3-naiji]

Hi @cnasikas and @pmuellr, I have added the "funtion test" in the x-pack/test/alerting_api_integration directory and followed the instructions from https://www.elastic.co/guide/en/kibana/master/development-tests.html#development-functional-tests
So, I tried to run command's bellow
node scripts/functional_tests_server, which was successfully. But, when I tried to run node scripts/functional_test_runner, it required a --config, please see the image bellow.

After I added the config path, it says Error: attempted to use the "es" service to fetch Elasticsearch version info but the request failed: ConnectionError: write EPROTO 4771253760:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

I have searched up, but still couldn't fix the problem.
Do you know how can I test the "funtion test" files or if you have any references or tips for that?
I look forward to hearing from you, thank you!

[d3-naiji]

Hi @cnasikas and @pmuellr, "funtional test" has been already added and locally tested successfully. Also, "Case" tag has been removed and README.md has been updated. Let me know if you have any questions.

I look forward to hearing from you, thank you!

[EricDavisX]

Hi, @d3-naiji I wanted to reach out, is there a point of contact we can talk to about setting up a real 'test instance' of the D3 Soar code base for on-going end to end validation? I can't commit to how fast we can work it, but I'm a good resource to kick that chat off. I can read up on the docs, too. TIA!

[d3-naiji]

Hi, @d3-naiji I wanted to reach out, is there a point of contact we can talk to about setting up a real 'test instance' of the D3 Soar code base for on-going end to end validation? I can't commit to how fast we can work it, but I'm a good resource to kick that chat off. I can read up on the docs, too. TIA!

Hi @EricDavisX , thanks for the update. I will prepare a document and a demo video about how to use the Kibane Connector to create events or trigger playbook flows in D3 SOAR. I will send you the documents and the 'test instance' in an email this week.

[d3-naiji]

Hi, @d3-naiji I wanted to reach out, is there a point of contact we can talk to about setting up a real 'test instance' of the D3 Soar code base for on-going end to end validation? I can't commit to how fast we can work it, but I'm a good resource to kick that chat off. I can read up on the docs, too. TIA!

@EricDavisX The document and D3 SOAR "test instance" info was sent to your email.

[EricDavisX]

Confirming as I understand it that @MindyRS 's team will take this on from testing side. I have forwarded the note to her I received. @MadameSheema and @asnehalb CC. Regards!

[stephmilovic]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[stephmilovic]

@elasticmachine merge upstream

[stephmilovic]

@d3-naiji could you please update the PR and resolve the conflicts? I will be reviewing this PR today/this week. Thank you

[semd]

Hi @d3-naiji, thanks for the PR.
After checking the code, it looks pretty similar to the Webhook connector, have you tried using it? If so, what are the limitations you hit?

FYI, the conflicts are caused by a migration of all connectors that happened last week. Now they live in the stack_connectors plugin.
You should be able to move:

• backend in x-pack/plugin/actions/server -> x-pack/plugins/stack_connectors/server
• UI in x-pack/plugins/triggers_actions_ui -> x-pack/plugins/stack_connectors/public

[semd]

buildkite test this

[d3-naiji]

Hi @semd , @EricDavisX , @stephmilovic , @pmuellr , @cnasikas.

Sorry for the delay, the merge conflict has been already resolved and we have added some new features, please check the PR request below:
#144955

Let me know if you have any questions.

Thank you!

[semd]

buildkite test this

[semd]

Hi @d3-naiji,
Sorry, we have been busy during the last couple of weeks due to the release schedule.
There have been new connectors merged, which has caused some more conflicts to appear, they should be easy to fix.

[semd]

New code need to be formatted using prettier

[semd]

please remove comment

[semd]

?

[semd]

Is PUT a valid method?

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: 3fdb847
• Interpreting CI Failures

Failed CI Steps

• x-pack/test/detection_engine_api_integration/security_and_spaces/group1/config.ts
• x-pack/test/detection_engine_api_integration/security_and_spaces/group1/config.ts
• FTR Configs #8
• FTR Configs #8
• FTR Configs #12
• FTR Configs #12
• FTR Configs #14
• FTR Configs #14
• FTR Configs #26
• FTR Configs #26
• FTR Configs #40
• FTR Configs #40
• FTR Configs #43
• Linting
• Checks
• Check Types

Test Failures

• [job] [logs] FTR Configs #43 / alerting api integration spaces only Alerting builtin alertTypes index_threshold rule runs correctly: avg all
• [job] [logs] FTR Configs #12 / apis management index lifecycle management nodes list should return the list of ES node for each custom attributes
• [job] [logs] FTR Configs #12 / apis management index lifecycle management nodes list should return the list of ES node for each custom attributes
• [job] [logs] FTR Configs #26 / cases security and spaces enabled: basic Common suggest_user_profiles finds the profile for the user without deletion privileges
• [job] [logs] FTR Configs #26 / cases security and spaces enabled: basic Common suggest_user_profiles finds the profile for the user without deletion privileges
• [job] [logs] x-pack/test/detection_engine_api_integration/security_and_spaces/group1/config.ts / detection engine api security and spaces enabled - Group 1 Tests involving runtime fields of source indexes and the signals index "before all" hook in "Tests involving runtime fields of source indexes and the signals index"
• [job] [logs] x-pack/test/detection_engine_api_integration/security_and_spaces/group1/config.ts / detection engine api security and spaces enabled - Group 1 Tests involving runtime fields of source indexes and the signals index "before all" hook in "Tests involving runtime fields of source indexes and the signals index"
• [job] [logs] FTR Configs #14 / security APIs - PKI PKI authentication should reject API requests that use untrusted certificate
• [job] [logs] FTR Configs #14 / security APIs - PKI PKI authentication should reject API requests that use untrusted certificate
• [job] [logs] FTR Configs #12 / security APIs - User Profiles Getting user profiles in bulk "before all" hook for "can get multiple profiles"
• [job] [logs] FTR Configs #12 / security APIs - User Profiles Getting user profiles in bulk "before all" hook for "can get multiple profiles"
• [job] [logs] FTR Configs #40 / security app useremail login as new user and verify email
• [job] [logs] FTR Configs #40 / security app useremail login as new user and verify email
• [job] [logs] FTR Configs #8 / visualize app visual builder Time Series basics Clicking on the chart should create a filter
• [job] [logs] FTR Configs #8 / visualize app visual builder Time Series basics Clicking on the chart should create a filter

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
triggersActionsUi	595	601	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
triggersActionsUi	1.0MB	1.0MB	+20.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
triggersActionsUi	91.7KB	92.5KB	+795.0B

Unknown metric groups

async chunk count

id	before	after	diff
triggersActionsUi	96	100	+4

ESLint disabled in files

id	before	after	diff
apm	14	13	-1
observability	6	5	-1
total			-2

ESLint disabled line counts

id	before	after	diff
actions	20	21	+1
apm	81	78	-3
enterpriseSearch	13	11	-2
observability	44	43	-1
synthetics	57	51	-6
triggersActionsUi	183	186	+3
ux	10	9	-1
total			-9

Total ESLint disabled count

id	before	after	diff
actions	26	27	+1
apm	95	91	-4
enterpriseSearch	13	11	-2
observability	50	48	-2
synthetics	63	57	-6
triggersActionsUi	186	189	+3
ux	13	12	-1
total			-11

History

• 💔 Build #82169 failed 3fdb847

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[semd]

@EricDavisX @d3-naiji
Would it be possible to have the test instance info sent to my Elastic address (sergi.massaneda@elastic.co) as well? I will need it in order to approve the PR.

[d3-naiji]

@EricDavisX @d3-naiji Would it be possible to have the test instance info sent to my Elastic address (sergi.massaneda@elastic.co) as well? I will need it in order to approve the PR.

Hi @semd , have you received the test instance? Do you want me to send it again?
And is there any update for the review? We submitted another pull request #144955 for merging the conflicts of the file structure change. Please let me know if you need another other information from us.

[d3-naiji]

Hi @EricDavisX @semd,
In case it was lost, I have re-sent the test instance information to your email. Please let me know if you have any trouble accessing it or if there are any further questions or comments that you may have.

[mikecote]

This PR appears to be stale, and it seems the functionality it aimed to introduce has been addressed in a new PR #158569 that was branched off of this one. I'll go ahead and close this PR for now. If there's any reason to keep it open or further discuss its relevance, please feel free to reopen it. Thanks again for your contributions!