[Security Solution][Detections] Rule execution logging overhaul #121644
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
banderror
added
performance
refactoring
technical debt
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
3 times, most recently
from
771a9ee to
df40deb
Compare
xcrzx
reviewed
Dec 23, 2021
...rity_solution/server/lib/detection_engine/rule_execution_log/rule_execution_logger/logger.ts
Outdated
Show resolved
Hide resolved
...rity_solution/server/lib/detection_engine/rule_execution_log/rule_execution_logger/logger.ts
Outdated
Show resolved
Hide resolved
..._solution/server/lib/detection_engine/rule_execution_log/rule_execution_log_client/client.ts
Show resolved
Hide resolved
...rity_solution/server/lib/detection_engine/rule_execution_log/rule_execution_logger/logger.ts
Show resolved
Hide resolved
...n/server/lib/detection_engine/rule_execution_log/rule_execution_info/saved_objects_client.ts
Outdated
Show resolved
Hide resolved
...n/server/lib/detection_engine/rule_execution_log/rule_execution_info/saved_objects_client.ts
Outdated
Show resolved
Hide resolved
...rity_solution/server/lib/detection_engine/rule_execution_log/rule_execution_logger/logger.ts
Outdated
Show resolved
Hide resolved
x-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts
Show resolved
Hide resolved
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
6 times, most recently
from
3ef5434 to
95f66dc
Compare
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
from
994be34 to
927f810
Compare
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
7 times, most recently
from
f62a19e to
0d884be
Compare
1 task
spong
approved these changes
Jan 20, 2022
There was a problem hiding this comment.
Checked out, desk tested locally, and code reviewed concepts and arch changes. Verified upgrade that the old status SO's aren't migrated over (though one nit about potential leftover mappings we need to talk to core about), and tested rule failures and UI around the the new rule-execution-info SO.
You've packed quite a bit in here @banderror! Thank you for all the cleanup and added plumbing for the new Rule Execution Events table! Approving now as we've discussed follow-up changes offline in a team architecture review and this is a hefty PR to be left open for too long. Appreciate the TODO:'s and contextual comments as well @banderror -- LGTM! 👍 🚀
FrankHassanabad
pushed a commit
that referenced
this pull request
Jan 20, 2022
## Summary New ECS FieldMap was generated in #123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See #122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: #123357 #121644 #123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 20, 2022
…123429) ## Summary New ECS FieldMap was generated in elastic#123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See elastic#122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: elastic#123357 elastic#121644 elastic#123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit d6917fc)
|
@elasticmachine merge upstream |
kibanamachine
added a commit
that referenced
this pull request
Jan 20, 2022
…#123433) ## Summary New ECS FieldMap was generated in #123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See #122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: #123357 #121644 #123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit d6917fc) Co-authored-by: Garrett Spong <spong@users.noreply.github.com>
xcrzx
approved these changes
Jan 20, 2022
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
from
7a3d986 to
fb564af
Compare
dokmic
approved these changes
Jan 20, 2022
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
from
fb564af to
f2fe48b
Compare
banderror
force-pushed
the
new-data-model-for-rule-statuses-and-metrics
branch
from
f2fe48b to
f4795b5
Compare
💚 Build Succeeded
Metrics [docs]Module Count
Async chunks
Public APIs missing exports
Page load bundle
Saved Objects .kibana field count
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @banderror |
|
Thanks everyone for your feedback! I'm merging this PR and will start addressing the remaining comments here and items from the |
8 tasks
8 tasks
6 tasks
6 tasks
This was referenced Jan 20, 2022
spong
pushed a commit
that referenced
this pull request
Feb 2, 2022
…w-up (#124194) **Related to:** #121644 **Addresses:** #86202 ## Summary Done in this PR: - Removed the deprecated `warning` rule execution status ([comment](#121644 (comment))). - Added a new `running` status ([ticket](#86202)). - Simplified the internal implementation of the `rule_execution_log` folder. Hopefully naming of folders, files and interfaces is clearer now as well. ([comment](#121644 (comment)), [comment](#121644 (comment))) - Added APM measurements with `withSecuritySpan`. - Added rule id to the react-query key used for loading last rule failures ([comment](#124198 (comment))) - Addressed most of the `// TODO: https://github.com/elastic/kibana/pull/121644` comments In the next PR that could be merged after the FF I'd address the rest of the stuff: - Add comments to all the interfaces and methods in the `rule_execution_log` folder. Write a readme for it. - Address the remaining of the `// TODO: https://github.com/elastic/kibana/pull/121644` comments. All of them are related to tests. - Fix for the gap column ([comment](#121644 (comment)))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Epic: #118324
Tickets: #119603, #119597, #91265, #118511
Summary
The legacy rule execution logging implementation is replaced by a new one that introduces a new model for execution-related data, a new saved object and a new, cleaner interface and implementation.
IRuleStatusResponseAttributes,IRuleStatusSOAttributes)siem-detection-engine-rule-statussaved object type is deleted and marked as deleted insrc/corex-pack/plugins/security_solution/common/detection_engine/schemas/common/rule_monitoring.ts). This data model doesn't contain a mixture of successful and failed statuses, which should simplify client-side code (e.g. the code of Rule Management and Monitoring tables, as well as Rule Details page).siem-detection-engine-rule-execution-infosaved object is introduced (x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log/rule_execution_info/saved_object.ts).x-pack/plugins/security_solution/server/lib/detection_engine/rule_execution_log). The previous rule execution log client is split into two objects:IRuleExecutionLogClientfor using it from route handlers, andIRuleExecutionLoggerfor writing logs from rule executors.IRuleExecutionLoggerinstance is scoped to the currently executing rule and space id. There's no need to pass rule id, name, type etc to.logStatusChange()every time.IRuleExecutionLogger. A rule returned from an API endpoint now has optionalexecution_summaryfield of typeRuleExecutionSummary.RuleExecutionSummary:/internal/detection_engine/rules/{ruleId}/execution/events. It is used for rendering the Failure History tab (last 5 failures) and is intended to be used in the coming UI of Rule Execution Log on the Details page.react-queryfor fetching execution eventsx-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/use_rule_execution_events.tsxx-pack/plugins/security_solution/public/detections/components/rules/rule_execution_status.// TODO: https://github.com/elastic/kibana/pull/121644comments in the code which I'm planning to address and remove in a follow-up PR. Lots of clean up work is needed, but I'd like to unblock the work on Rule Execution Log UI.In the next episodes
// TODO: https://github.com/elastic/kibana/pull/121644comments in the codesiem-detection-engine-rule-execution-infois safe and future-proof. Sync with the Core team. If there are risks, we will need to choose between risks and performance (reading the SO before updating it). It would be easy to submit a fix if needed.withSecuritySpanin methods ofrule_execution_logcitizens.Checklist
Delete any items that are not applicable to this PR.
For maintainers