Checkpoint Syslog Filebeat module #17682

P1llus · 2020-04-13T08:56:35Z

What does this PR do?

This PR adds checkpoint filebeat module.

The difference between this module and the CEF checkpoint module is that this is utilizing the syslog output format instead of CEF.

This output format supports a much larger set of fields from Checkpoint.

This is a collaboration work with @adriansr .

Why is it important?

Adding more supported products to the filebeat portfolio.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Closes #16041

Nosetests run successfully:
INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=checkpoint nosetests -v -s tests/system/test_xpack_modules.py

elasticmachine · 2020-04-13T08:56:38Z

Pinging @elastic/siem (Team:SIEM)

… conditional statement in pipeline to make it safer

… module

This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related #16907 #17682

This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related elastic#16907 elastic#17682 (cherry picked from commit 4f6da4f)

) This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related #16907 #17682 (cherry picked from commit 4f6da4f)

leehinman

Looking really good. I've got a couple of suggestions for increased use of ECS.

The related.ip, related.hash & related.user fields should be populated.
event.category, any chance we can figure out when malware & intrusion_detection could be added?
event.type in-line comments on allowed & denied could we also use sequence number for start?

x-pack/filebeat/module/checkpoint/firewall/ingest/pipeline.json

P1llus · 2020-04-16T20:30:00Z

All changes from comments has been applied, new expected.json generated, and nosetest has been run again.

CHANGELOG.next.asciidoc

x-pack/filebeat/module/checkpoint/_meta/docs.asciidoc

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

…kpoint-module Adding changes from remote branch to Changelog

P1llus · 2020-04-16T21:44:34Z

Added updated documentation changes as per @andrewkroh comments. Documentation from CEF to checkpoint module will be done in a separate PR.

andrewkroh

LGTM. @leehinman do you want to take another look?

tonymeehan · 2020-04-17T14:11:48Z

x-pack/filebeat/module/checkpoint/firewall/test/checkpoint.log-expected.json

+    {
+        "@timestamp": "2020-03-29T13:19:20Z",
+        "checkpoint.sys_message": ":\"The eth0 interface is not protected by the anti-spoofing feature. Your network may be at risk",
+        "event.category": [


Seems like the format for checkpoint.sys_message is different than the rest during errors and alerts, will add a fix for this.

The issue is because that specific field has a weird value, this is the raw event coming directly from the device:
sys_message::"The eth0 interface is not protected by the anti-spoofing feature. Your network may be at risk"]

I was commenting that it's good to see event.category as an array!

…a to ensure both correct and incorrect format is working

P1llus · 2020-04-20T09:38:10Z

@tonymeehan

I have added a workaround for this issue, as its not in the parser itself but rather a bug on the Checkpoint side.

I have updated the dataset to have at least 1 properly formatted and 1 incorrectly formatted message, to ensure the parser works for both. Re-ran nosetests afterwards.

tonymeehan · 2020-04-20T15:16:22Z

@P1llus really awesome to see this PR!

This adds a CheckPoint Filebeat module. The difference between this module and the CEF checkpoint module is that this is utilizing the syslog output format instead of CEF. This syslog output format supports a much larger set of fields from Checkpoint. (cherry picked from commit afc3a49)

This adds a CheckPoint Filebeat module. The difference between this module and the CEF checkpoint module is that this is utilizing the syslog output format instead of CEF. This syslog output format supports a much larger set of fields from Checkpoint. (cherry picked from commit afc3a49) Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

hazcod · 2020-04-29T14:22:50Z

@P1llus / @tonymeehan : This seems to not include any ssl configuration for input?
Any way to overwrite it? I'm trying the following:

- module: checkpoint
  access:
    input:
    - type: syslog
    protocol.tcp:
      host: "0.0.0.0:8001"
    tls:
    .....

… (elastic#17712) This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related elastic#16907 elastic#17682 (cherry picked from commit ddb92ca)

finishing first MVP of the checkpoint filebeat module

b25a422

P1llus added enhancement in progress Pull request is currently in progress. Team:SIEM labels Apr 13, 2020

P1llus requested a review from adriansr April 13, 2020 08:56

P1llus changed the title ~~finishing first MVP of the checkpoint filebeat module~~ Checkpoint Syslog Filebeat module Apr 13, 2020

updated Changelog for checkpoint filebeat module

269f1bf

adriansr mentioned this pull request Apr 13, 2020

CEF CheckPoint: adjust fields for forward compatibility #17681

Merged

6 tasks

P1llus added 6 commits April 13, 2020 11:15

updating docs and adding default timestamp to syslog ts

544121b

updated docs, added beta tag, added error catching in pipeline

8527f8e

fix documentation error

e929ea5

added ECS event field to checkpoint documentation, and added an extra…

d2682fa

… conditional statement in pipeline to make it safer

updated docs with references to Checkpoint documentation for filebeat…

50ce552

… module

adding overwrite to fields, due to collision with CEF checkpoint module

5e540b5

adriansr mentioned this pull request Apr 14, 2020

Cherry-pick #17681 to 7.x: CEF CheckPoint: adjust fields for forward compatibility #17711

Merged

6 tasks

adriansr mentioned this pull request Apr 14, 2020

Cherry-pick #17681 to 7.7: CEF CheckPoint: adjust fields for forward compatibility #17712

Merged

6 tasks

P1llus added review and removed in progress Pull request is currently in progress. labels Apr 15, 2020

leehinman requested changes Apr 15, 2020

View reviewed changes

P1llus and others added 2 commits April 16, 2020 22:16

added changes requested in PR comments

1a2b0b6

Merge branch 'master' into checkpoint-module

da66067

P1llus requested a review from leehinman April 16, 2020 20:28

andrewkroh reviewed Apr 16, 2020

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

x-pack/filebeat/module/checkpoint/_meta/docs.asciidoc Outdated Show resolved Hide resolved

P1llus and others added 3 commits April 16, 2020 23:16

Update CHANGELOG.next.asciidoc

31d3982

Co-Authored-By: Andrew Kroh <andrew.kroh@elastic.co>

Updated Check Point documentation to point to the related CEF module

60c0b9d

Merge branch 'checkpoint-module' of github.com:P1llus/beats into chec…

2f29a42

…kpoint-module Adding changes from remote branch to Changelog

andrewkroh approved these changes Apr 16, 2020

View reviewed changes

tonymeehan reviewed Apr 17, 2020

View reviewed changes

bypasses a bug in Check Point on sys_message format, updated test dat…

93d4c33

…a to ensure both correct and incorrect format is working

andrewkroh approved these changes Apr 20, 2020

View reviewed changes

leehinman approved these changes Apr 20, 2020

View reviewed changes

andrewkroh merged commit afc3a49 into elastic:master Apr 20, 2020

P1llus deleted the checkpoint-module branch April 20, 2020 15:58

andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Apr 20, 2020

andrewkroh mentioned this pull request Apr 20, 2020

Cherry-pick #17682 to 7.x: Checkpoint Syslog Filebeat module #17838

Merged

6 tasks

andrewkroh added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 20, 2020

hazcod mentioned this pull request Apr 30, 2020

Filebeat: add TLS + pipeline options to checkpoint module #18116

Closed

6 tasks

This was referenced Jul 1, 2020

Filebeat: add TLS + pipeline options to checkpoint module P1llus/beats#1

Closed

filebeat: add SSL options to checkpoint module #19560

Merged

marc-gr mentioned this pull request Oct 19, 2020

Cherry-pick #19560 to 7.x: filebeat: add SSL options to checkpoint module #21988

Merged

6 tasks

bbs2web mentioned this pull request Jul 18, 2022

[Filebeat][Checkpoint module] data stream timestamp field [@timestamp] is missing #32380

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Checkpoint Syslog Filebeat module #17682

Checkpoint Syslog Filebeat module #17682

P1llus commented Apr 13, 2020 •

edited by zube bot

Loading

elasticmachine commented Apr 13, 2020

leehinman left a comment

P1llus commented Apr 16, 2020

P1llus commented Apr 16, 2020

andrewkroh left a comment

tonymeehan Apr 17, 2020

P1llus Apr 20, 2020 •

edited

Loading

tonymeehan Apr 20, 2020

P1llus commented Apr 20, 2020

tonymeehan commented Apr 20, 2020

hazcod commented Apr 29, 2020 •

edited

Loading

Checkpoint Syslog Filebeat module #17682

Checkpoint Syslog Filebeat module #17682

Conversation

P1llus commented Apr 13, 2020 • edited by zube bot Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Apr 13, 2020

leehinman left a comment

Choose a reason for hiding this comment

P1llus commented Apr 16, 2020

P1llus commented Apr 16, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

tonymeehan Apr 17, 2020

Choose a reason for hiding this comment

P1llus Apr 20, 2020 • edited Loading

Choose a reason for hiding this comment

tonymeehan Apr 20, 2020

Choose a reason for hiding this comment

P1llus commented Apr 20, 2020

tonymeehan commented Apr 20, 2020

hazcod commented Apr 29, 2020 • edited Loading

P1llus commented Apr 13, 2020 •

edited by zube bot

Loading

P1llus Apr 20, 2020 •

edited

Loading

hazcod commented Apr 29, 2020 •

edited

Loading