Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SIEM][CEF] Add support for Check Point devices #16907

Merged
merged 11 commits into from
Mar 18, 2020
Merged

[SIEM][CEF] Add support for Check Point devices #16907

merged 11 commits into from
Mar 18, 2020

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Mar 9, 2020 •
edited by zube bot
Loading

This PR adds a new ingest pipeline and fields to populate from Check Point CEF logs.

Additional changes to the ECS processor:

  • Make CEF key translation case-insensitive.
  • Add missing CEF extension c6a2/deviceCustomIPv6Address2.
  • Add deprecated flexNumber[12] extensions. Used by Check Point. Dropped from CEF after V23.
  • Change case of DeviceCustomNumber2 to align with the other fields.

Closes #16041

@adriansr adriansr requested a review from andrewkroh March 9, 2020 17:43
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@adriansr adriansr requested a review from leehinman March 9, 2020 17:44
adriansr
adriansr commented Mar 9, 2020
View reviewed changes
x-pack/filebeat/module/cef/log/_meta/fields.yml Outdated
- name: version
type: keyword

# TODO: Update to ECS 1.5 and remove.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove and rebase once #16900 is merged

andrewkroh
andrewkroh approved these changes Mar 10, 2020
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look great 👍

filebeat/docs/modules/cef.asciidoc
Check Point CEF extensions are mapped as follows:
[options="header"]
|=======================================================================================================================
| CEF Extension | CEF Label value | ECS Fields | Non-ECS Field |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table is awesome. Helps me a lot in digesting the pipeline.

leehinman
leehinman reviewed Mar 11, 2020
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very Nice.

Thank you for adding comments about the field names and Checkpoint use being "unusual". The sourceNtDomain one was throwing me :-)

Do you think we could add some ECS categorization fields?
event.kind: most things would be events. could alerts be events that have rules and high risk_score or severity?
event.category: I think if the event has IP addresses we can add network, I'm not sure if we have enough info for malware or intrustion_detection
event.type: could we map event.action to allowed/denied for the network events?

@adriansr adriansr force-pushed the feature_checkpoint_cef branch 2 times, most recently from 2eca571 to e80d90f Compare March 13, 2020 16:33
@adriansr
Copy link
Contributor Author

@leehinman , I've attempted some categorisation by looking at the fields present. However, it is difficult to do anything acceptable without sample logs (I couldn't find any).

For example for event.action (or outcome) I only have the value "Allowed" in the sample logs but I don't know what other possible values are there so I'm reluctant to map everything else to denied.

Have a look at the changes, I'm open to suggestions

@leehinman
Copy link
Contributor

@leehinman , I've attempted some categorisation by looking at the fields present. However, it is difficult to do anything acceptable without sample logs (I couldn't find any).

For example for event.action (or outcome) I only have the value "Allowed" in the sample logs but I don't know what other possible values are there so I'm reluctant to map everything else to denied.

Have a look at the changes, I'm open to suggestions

changes look great. hopefully we can get more examples in the future.

@adriansr
Make CEF key name mapping case-insensitive
5aed151 
There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to
ignore case when mapping keys to full names.
@adriansr
Add missing custom CEF extensions
9639d3a 
This adds:
 - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected.
 - `flexNumber[12](Label)`: These two alternative custom numbers were
   dropped after V23 of the spec, but still used by some vendors.

[Maybe unnecessary] changes:

 - Changed the case of `DeviceCustomNumber2` from uppercase as
   documented) to lowercase to align with the other fields.
@adriansr
CEF module: Support Check Point devices
b2210e7 
This adds a new ingest pipeline and fields to populate from Check Point
CEF logs.

Closes elastic#16041
@adriansr
Add docs
e3f9f86
@adriansr
Changelog entry
92a30c0
@adriansr
Add PR number
aae6071
@adriansr
Update decode_cef golden files
2db8b7a
@adriansr
Remove temporary ECS fields after rebase
ab3418c
@adriansr
observer.interface -> observer.ingress.interface
98f9f17
@adriansr
Try to populate some event categorisation fields
d17acce
@adriansr
Update golden files
81a7702
@adriansr adriansr merged commit f6fde2e into elastic:master Mar 18, 2020
@adriansr adriansr mentioned this pull request Mar 19, 2020
Merged
adriansr added a commit to adriansr/beats that referenced this pull request Mar 19, 2020
@adriansr
[SIEM][CEF] Add support for Check Point devices (elastic#16907)
c77551d 
* Make CEF key name mapping case-insensitive

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to
ignore case when mapping keys to full names.

* Add missing custom CEF extensions

This adds:
 - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected.
 - `flexNumber[12](Label)`: These two alternative custom numbers were
   dropped after V23 of the spec, but still used by some vendors.

[Maybe unnecessary] changes:

 - Changed the case of `DeviceCustomNumber2` from uppercase as
   documented) to lowercase to align with the other fields.

* CEF module: Support Check Point devices

This adds a new ingest pipeline and fields to populate from Check Point
CEF logs.

Closes elastic#16041

(cherry picked from commit f6fde2e)
adriansr added a commit that referenced this pull request Mar 20, 2020
@adriansr
Cherry-pick #16907 to 7.x: [SIEM][CEF] Add support for Check Point de…
a4b5c4a 
…vices (#17111)

* Make CEF key name mapping case-insensitive

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to
ignore case when mapping keys to full names.

* Add missing custom CEF extensions

This adds:
 - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected.
 - `flexNumber[12](Label)`: These two alternative custom numbers were
   dropped after V23 of the spec, but still used by some vendors.

[Maybe unnecessary] changes:

 - Changed the case of `DeviceCustomNumber2` from uppercase as
   documented) to lowercase to align with the other fields.

* CEF module: Support Check Point devices

This adds a new ingest pipeline and fields to populate from Check Point
CEF logs.

Closes #16041

(cherry picked from commit f6fde2e)
@adriansr adriansr mentioned this pull request Apr 13, 2020
Merged
6 tasks
adriansr added a commit that referenced this pull request Apr 14, 2020
@adriansr
CEF CheckPoint: adjust fields for forward compatibility (#17681)
4f6da4f 
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related #16907 #17682
@adriansr adriansr mentioned this pull request Apr 14, 2020
Merged
6 tasks
adriansr added a commit to adriansr/beats that referenced this pull request Apr 14, 2020
@adriansr
CEF CheckPoint: adjust fields for forward compatibility (elastic#17681)
b309968 
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit 4f6da4f)
adriansr added a commit to adriansr/beats that referenced this pull request Apr 14, 2020
@adriansr
CEF CheckPoint: adjust fields for forward compatibility (elastic#17681)
f3d9e30 
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit 4f6da4f)
@adriansr adriansr mentioned this pull request Apr 14, 2020
Merged
6 tasks
adriansr added a commit that referenced this pull request Apr 15, 2020
@adriansr
CEF CheckPoint: adjust fields for forward compatibility (#17681) (#17711
4de9e9c 
)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related #16907 #17682

(cherry picked from commit 4f6da4f)
adriansr added a commit that referenced this pull request Apr 15, 2020
@adriansr
CEF CheckPoint: adjust fields for forward compatibility (#17681) (#17712
8165f14 
)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related #16907 #17682

(cherry picked from commit 4f6da4f)
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@adriansr
CEF CheckPoint: adjust fields for forward compatibility (elastic#17681)…
9c9c7ee 
… (elastic#17712)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit ddb92ca)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leehinman leehinman leehinman left review comments

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees
No one assigned
Labels
enhancement module review v7.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Filebeat] Add module for Checkpoint Firewall
4 participants
@adriansr @elasticmachine @leehinman @andrewkroh