Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM][Detection Engine][Lists] Adds the ability for exception lists to be multi-list queried. #71540

Merged
merged 3 commits into from
Jul 14, 2020

Conversation

FrankHassanabad
Copy link
Contributor

@FrankHassanabad FrankHassanabad commented Jul 13, 2020

Summary

  • Adds the ability for exception lists to be multi-list queried
  • Fixes a bunch of script issues where I did not update everywhere I needed to use ip_list and deletes an old list that now lives within the new/lists folder
  • Fixes a few io-ts issues with Encode Decode while I was in there.
  • Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls.
  • Fixes one weird circular dep issue while adding more types.

You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so:

GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2"

And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists.

If at least one list is found this will not return a 404 but it will only query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception.

Script testing

See these files for more information:

  • find_exception_list_items.sh
  • find_exception_list_items_by_filter.sh

But basically you can create two lists and an item for each of the lists:

./post_exception_list.sh ./exception_lists/new/exception_list.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json

./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json

And then you can query these two lists together:

./find_exception_list_items.sh simple_list,endpoint_list single,agnostic

Or for filtering you can query both and add a filter for each one:

./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic

Checklist

Delete any items that are not applicable to this PR.

@FrankHassanabad FrankHassanabad self-assigned this Jul 13, 2020
@FrankHassanabad FrankHassanabad marked this pull request as ready for review July 13, 2020 20:52
@FrankHassanabad FrankHassanabad requested review from a team as code owners July 13, 2020 20:52
@FrankHassanabad FrankHassanabad changed the title Adds the ability for exception_lists to be multi-list queried. [SIEM][Detection Engine][Lists] Adds the ability for exception_lists to be multi-list queried. Jul 13, 2020
@FrankHassanabad FrankHassanabad changed the title [SIEM][Detection Engine][Lists] Adds the ability for exception_lists to be multi-list queried. [SIEM][Detection Engine][Lists] Adds the ability for exception lists to be multi-list queried. Jul 13, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

Copy link
Contributor

@yctercero yctercero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks so much for this - super appreciated. Pulled down and tested with scripts, the tests look awesome too!

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

id value diff baseline
lists 88 +3 85

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@FrankHassanabad FrankHassanabad merged commit 473806c into elastic:master Jul 14, 2020
@FrankHassanabad FrankHassanabad deleted the add-array-id branch July 14, 2020 01:07
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this pull request Jul 14, 2020
…to be multi-list queried. (elastic#71540)

## Summary

* Adds the ability for exception lists to be multi-list queried 
* Fixes a bunch of script issues where I did not update everywhere I needed to use `ip_list` and deletes an old list that now lives within the new/lists folder
* Fixes a few io-ts issues with Encode Decode while I was in there.
* Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls.
* Fixes one weird circular dep issue while adding more types.

You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so:

```ts
GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2"
```

And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists.

If at least one list is found this will _not_ return a 404 but it will _only_ query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception.

**Script testing**

See these files for more information:
* find_exception_list_items.sh
* find_exception_list_items_by_filter.sh 

But basically you can create two lists and an item for each of the lists:

```ts
./post_exception_list.sh ./exception_lists/new/exception_list.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json

./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json
```

And then you can query these two lists together:
```ts
./find_exception_list_items.sh simple_list,endpoint_list single,agnostic
```

Or for filtering you can query both and add a filter for each one:
```ts
./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic
```

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this pull request Jul 14, 2020
…to be multi-list queried. (#71540) (#71580)

## Summary

* Adds the ability for exception lists to be multi-list queried 
* Fixes a bunch of script issues where I did not update everywhere I needed to use `ip_list` and deletes an old list that now lives within the new/lists folder
* Fixes a few io-ts issues with Encode Decode while I was in there.
* Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls.
* Fixes one weird circular dep issue while adding more types.

You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so:

```ts
GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2"
```

And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists.

If at least one list is found this will _not_ return a 404 but it will _only_ query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception.

**Script testing**

See these files for more information:
* find_exception_list_items.sh
* find_exception_list_items_by_filter.sh 

But basically you can create two lists and an item for each of the lists:

```ts
./post_exception_list.sh ./exception_lists/new/exception_list.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json

./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json
```

And then you can query these two lists together:
```ts
./find_exception_list_items.sh simple_list,endpoint_list single,agnostic
```

Or for filtering you can query both and add a filter for each one:
```ts
./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic
```

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jul 14, 2020
* master: (314 commits)
  [APM] Use status_code field to calculate error rate (elastic#71109)
  [Observability] Change appLink passing the date range (elastic#71259)
  [Security] Add Timeline improvements (elastic#71506)
  adjust vislib bar opacity (elastic#71421)
  Fix ScopedHistory mock and adapt usages (elastic#71404)
  [Security Solution] Add hook for reading/writing resolver query params (elastic#70809)
  [APM] Bug fixes from ML integration testing (elastic#71564)
  [Discover] Add caused_by.type and caused_by.reason to error toast modal (elastic#70404)
  [Security Solution] Add 3rd level breadcrumb to admin page (elastic#71275)
  [Security Solution][Exceptions] Exception modal bulk close alerts that match exception attributes (elastic#71321)
  Change signal.rule.risk score mapping from keyword to float (elastic#71126)
  Added help text where needed on connectors and alert actions UI (elastic#69601)
  [SIEM][Detections] Value Lists Management Modal (elastic#67068)
  [test] Skips test preventing promotion of ES snapshot elastic#71582
  [test] Skips test preventing promotion of ES snapshot elastic#71555
  [ILM] Fix alignment of the timing field (elastic#71273)
  [SIEM][Detection Engine][Lists] Adds the ability for exception lists to be multi-list queried. (elastic#71540)
  initial telemetry setup (elastic#69330)
  [Reporting] Formatting fixes for CSV export in Discover, CSV download from Dashboard panel (elastic#67027)
  Search across spaces (elastic#67644)
  ...
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jul 14, 2020
…t-apps-page-titles

* 'master' of github.com:elastic/kibana: (88 commits)
  [ML] Functional tests - disable DFA creation and cloning tests
  [APM] Use status_code field to calculate error rate (elastic#71109)
  [Observability] Change appLink passing the date range (elastic#71259)
  [Security] Add Timeline improvements (elastic#71506)
  adjust vislib bar opacity (elastic#71421)
  Fix ScopedHistory mock and adapt usages (elastic#71404)
  [Security Solution] Add hook for reading/writing resolver query params (elastic#70809)
  [APM] Bug fixes from ML integration testing (elastic#71564)
  [Discover] Add caused_by.type and caused_by.reason to error toast modal (elastic#70404)
  [Security Solution] Add 3rd level breadcrumb to admin page (elastic#71275)
  [Security Solution][Exceptions] Exception modal bulk close alerts that match exception attributes (elastic#71321)
  Change signal.rule.risk score mapping from keyword to float (elastic#71126)
  Added help text where needed on connectors and alert actions UI (elastic#69601)
  [SIEM][Detections] Value Lists Management Modal (elastic#67068)
  [test] Skips test preventing promotion of ES snapshot elastic#71582
  [test] Skips test preventing promotion of ES snapshot elastic#71555
  [ILM] Fix alignment of the timing field (elastic#71273)
  [SIEM][Detection Engine][Lists] Adds the ability for exception lists to be multi-list queried. (elastic#71540)
  initial telemetry setup (elastic#69330)
  [Reporting] Formatting fixes for CSV export in Discover, CSV download from Dashboard panel (elastic#67027)
  ...

# Conflicts:
#	x-pack/plugins/index_management/public/application/index.tsx
@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants