[Filebeat] Improve ASA/FTD Ingest Pipeline #23766

hitchatwork · 2021-01-29T19:30:04Z

What does this PR do?

Fixes #21658

For messages 716002:

Changed to GROK; allows for better parsing of event.reason
Added field for cisco.webvpn.group_name
Added field for event.reason per cisco docs for why session was terminated
Added field for cisco.termination_user for the AAA username terminating the connection

For messages 722051:

Add angle brackets to dissect to properly dissect the message, per cisco docs
Added field for cisco.webvpn.group.name

For messages 305011:

Change to GROK; allows for variance in message format with identity firewall

For messages 302020:

Added GROK pattern to allows for variance in message format with identity firewall

For messages 302014/302016/302021:

Added patterns and modified order of patterns of GROK to better match teardown messages
Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match.
Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future.
Added icmp_type and icmp_code parsing for 302021 messages
Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME.

And:

Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor.
Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.
Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

Why is it important?

Improved parsing of ASA/FTD messages mentioned above.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat
PYTEST_ADDOPTS="-k cisco" TESTING_FILEBEAT_FILESETS=asa,ftd mage -v pythonIntegTest

Related issues

Closes Filebeat Cisco module doesn't correctly parse 305011, 302015, 302013, or 722051 message types #21658

elasticmachine · 2021-01-29T19:34:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23766 updated
Start Time: 2021-03-24T15:18:18.927+0000
Duration: 125 min 49 sec
Commit: 0c52b2d

Test stats 🧪

Test	Results
Failed	0
Passed	13173
Skipped	2243
Total	15416

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13173
Skipped	2243
Total	15416

elasticmachine · 2021-01-29T19:35:13Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2021-01-29T20:37:26Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

P1llus

Adding some small comments

x-pack/filebeat/module/cisco/asa/test/additional_messages.log-expected.json

x-pack/filebeat/module/cisco/asa/test/asa-fix.log-expected.json

x-pack/filebeat/module/cisco/meraki/test/generated.log-expected.json

x-pack/filebeat/module/cisco/asa/test/additional_messages.log-expected.json

jsoriano · 2021-02-01T11:55:20Z

jenkins run the tests please

elasticmachine · 2021-02-12T19:55:54Z

Pinging @elastic/ingest-management (Team:Ingest Management)

hitchatwork · 2021-02-13T17:11:46Z

@jsoriano - could you kick jenkins off again? I made adjustments from @P1llus comments and also updated from master. Thanks!

andrewkroh · 2021-02-13T17:31:43Z

jenkins, run tests

hitchatwork · 2021-02-13T19:31:23Z

Thanks for kicking that off @andrewkroh - 4 of the tests failed (dictionary item adds due to ICMP processing). I decided to simplify the PR by removing any changes to ICMP at all (will create a seperate PR for that). Could you kick Jenkins off again?

andrewkroh · 2021-02-13T19:51:24Z

jenkins, run tests

hitchatwork · 2021-02-13T21:00:34Z

I'm not sure why this build failed. Looks like something maybe in the backend with the Windows 2019 environment setup for the build?

andrewkroh · 2021-02-13T23:28:56Z

jenkins, run tests

hitchatwork · 2021-02-14T01:56:08Z

Thanks a lot @andrewkroh - looks like it passed and can be reviewed by the relevant teams. Appreciate the assistance.

andrewkroh

Thanks for your work on this. It LGTM, but it needs an entry in the changelog to describe the change. Can you please put a note into CHANGELOG.next.asciidoc at the root of the repo.

hitchatwork · 2021-02-15T20:42:38Z

Thanks for your work on this. It LGTM, but it needs an entry in the changelog to describe the change. Can you please put a note into CHANGELOG.next.asciidoc at the root of the repo.

Added changelog entry.

andrewkroh · 2021-02-17T20:13:05Z

Looks like we have a conflict in fields.go due to the fields changes. This will need a rebase.

hitchatwork · 2021-02-22T04:41:47Z

Looks like we have a conflict in fields.go due to the fields changes. This will need a rebase.

@andrewkroh - I have rebased and cleaned up some of the commits. LMK if something else needs work. Thanks!

andrewkroh · 2021-02-22T16:00:37Z

jenkins, run tests

hitchatwork · 2021-03-04T19:38:13Z

@andrewkroh if you could take a look, seems it's stuck in build or something odd. Thanks!

andrewkroh · 2021-03-05T18:55:15Z

jenkins, run tests

hitchatwork · 2021-03-12T19:18:30Z

@andrewkroh looks like the build is failing from when you pushed in the golden files commit. Looks like expected output for cisco AMP module changed, and that's failing the build.

andrewkroh · 2021-03-16T19:10:20Z

jenkins, run tests

Fixes elastic#21658 For messages 716002: - Changed to GROK; allows for better parsing of event.reason - Added field for cisco.webvpn.group_name - Added field for event.reason per cisco docs for why session was terminated - Added field for cisco.termination_user for the AAA username terminating the connection For messages 722051: - Add angle brackets to dissect to properly dissect the message, per cisco docs - Added field for cisco.webvpn.group.name For messages 305011: - Change to GROK; allows for variance in message format with identity firewall For messages 302020: - Added GROK pattern to allows for variance in message format with identity firewall For messages 302014/302016/302021: - Added patterns and modified order of patterns of GROK to better match teardown messages - Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match. - Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future. - Added icmp_type and icmp_code parsing for 302021 messages - Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME. And: - Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor. - Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. - Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long.

Fixes elastic#21658 For messages 716002: - Changed to GROK; allows for better parsing of event.reason - Added field for cisco.webvpn.group_name - Added field for event.reason per cisco docs for why session was terminated - Added field for cisco.termination_user for the AAA username terminating the connection For messages 722051: - Add angle brackets to dissect to properly dissect the message, per cisco docs - Added field for cisco.webvpn.group.name For messages 305011: - Change to GROK; allows for variance in message format with identity firewall For messages 302020: - Added GROK pattern to allows for variance in message format with identity firewall For messages 302014/302016/302021: - Added patterns and modified order of patterns of GROK to better match teardown messages - Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match. - Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future. - Added icmp_type and icmp_code parsing for 302021 messages - Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME. And: - Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor. - Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. - Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. (cherry picked from commit b5e43fc)

Fixes #21658 For messages 716002: - Changed to GROK; allows for better parsing of event.reason - Added field for cisco.webvpn.group_name - Added field for event.reason per cisco docs for why session was terminated - Added field for cisco.termination_user for the AAA username terminating the connection For messages 722051: - Add angle brackets to dissect to properly dissect the message, per cisco docs - Added field for cisco.webvpn.group.name For messages 305011: - Change to GROK; allows for variance in message format with identity firewall For messages 302020: - Added GROK pattern to allows for variance in message format with identity firewall For messages 302014/302016/302021: - Added patterns and modified order of patterns of GROK to better match teardown messages - Note that order of processing is important as the most specific messages are matched first, falling through to the appropriate match. - Added temp fields for teardown initiator and user; defined in cisco docs but currently no real place to put them, but could be in future. - Added icmp_type and icmp_code parsing for 302021 messages - Changed duration matching from TIME to NOTSPACE, as long-lived connections (over 24 hours) don't match TIME. And: - Added descriptions to many fields to make them easier to find in 7.9+ Kibana Ingest Node Pipeline editor. - Changed source.bytes field type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. - Changed destination.bytes type from integer to long, since long-lived flows can surpass the capacity of an integer; ECS reference field is defined as long. (cherry picked from commit b5e43fc) Co-authored-by: hitchatwork <56573831+hitchatwork@users.noreply.github.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 29, 2021

hitchatwork mentioned this pull request Jan 29, 2021

[Filebeat] Improve ASA/FTD pipeline #22001

Closed

6 tasks

ycombinator added the Team:SIEM label Jan 29, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 29, 2021

andresrc added Team:Security-External Integrations and removed Team:SIEM labels Jan 29, 2021

P1llus added enhancement Filebeat Filebeat labels Feb 1, 2021

P1llus requested a review from jsoriano February 1, 2021 06:50

P1llus reviewed Feb 1, 2021

View reviewed changes

jsoriano reviewed Feb 1, 2021

View reviewed changes

x-pack/filebeat/module/cisco/asa/test/additional_messages.log-expected.json Outdated Show resolved Hide resolved

hitchatwork requested review from a team as code owners February 12, 2021 19:55

botelastic bot added Team:Automation Label for the Observability productivity team Team:Ingest Management labels Feb 12, 2021

hitchatwork force-pushed the hitchatwork-ImproveASAPipeline-1 branch from 6bfca13 to ae59310 Compare February 13, 2021 00:27

andrewkroh requested changes Feb 14, 2021

View reviewed changes

hitchatwork requested a review from andrewkroh February 14, 2021 23:40

hitchatwork force-pushed the hitchatwork-ImproveASAPipeline-1 branch 2 times, most recently from 0a80440 to 5bbd0c5 Compare February 22, 2021 00:56

andrewkroh approved these changes Feb 22, 2021

View reviewed changes

andrewkroh mentioned this pull request Mar 16, 2021

[Filebeat] Cisco AMP module tests are failing #24573

Closed

andrewkroh force-pushed the hitchatwork-ImproveASAPipeline-1 branch 4 times, most recently from c090416 to 2d7997b Compare March 24, 2021 14:57

andrewkroh force-pushed the hitchatwork-ImproveASAPipeline-1 branch from 2d7997b to 0c52b2d Compare March 24, 2021 15:17

andrewkroh merged commit b5e43fc into elastic:master Mar 24, 2021

andrewkroh added needs_backport PR is waiting to be backported to other branches. and removed Team:Automation Label for the Observability productivity team Team:Ingest Management labels Mar 24, 2021

andrewkroh mentioned this pull request Mar 24, 2021

Cherry-pick #23766 to 7.x: [Filebeat] Improve ASA/FTD Ingest Pipeline #24748

Merged

4 tasks

andrewkroh added v7.13.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 24, 2021

hitchatwork deleted the hitchatwork-ImproveASAPipeline-1 branch March 29, 2021 22:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Improve ASA/FTD Ingest Pipeline #23766

[Filebeat] Improve ASA/FTD Ingest Pipeline #23766

hitchatwork commented Jan 29, 2021 •

edited by andrewkroh

Loading

elasticmachine commented Jan 29, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Jan 29, 2021

elasticmachine commented Jan 29, 2021

P1llus left a comment

jsoriano commented Feb 1, 2021

elasticmachine commented Feb 12, 2021

hitchatwork commented Feb 13, 2021

andrewkroh commented Feb 13, 2021

hitchatwork commented Feb 13, 2021

andrewkroh commented Feb 13, 2021

hitchatwork commented Feb 13, 2021

andrewkroh commented Feb 13, 2021

hitchatwork commented Feb 14, 2021

andrewkroh left a comment

hitchatwork commented Feb 15, 2021

andrewkroh commented Feb 17, 2021

hitchatwork commented Feb 22, 2021

andrewkroh commented Feb 22, 2021

hitchatwork commented Mar 4, 2021

andrewkroh commented Mar 5, 2021

hitchatwork commented Mar 12, 2021

andrewkroh commented Mar 16, 2021

[Filebeat] Improve ASA/FTD Ingest Pipeline #23766

[Filebeat] Improve ASA/FTD Ingest Pipeline #23766

Conversation

hitchatwork commented Jan 29, 2021 • edited by andrewkroh Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jan 29, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Jan 29, 2021

elasticmachine commented Jan 29, 2021

P1llus left a comment

Choose a reason for hiding this comment

jsoriano commented Feb 1, 2021

elasticmachine commented Feb 12, 2021

hitchatwork commented Feb 13, 2021

andrewkroh commented Feb 13, 2021

hitchatwork commented Feb 13, 2021

andrewkroh commented Feb 13, 2021

hitchatwork commented Feb 13, 2021

andrewkroh commented Feb 13, 2021

hitchatwork commented Feb 14, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

hitchatwork commented Feb 15, 2021

andrewkroh commented Feb 17, 2021

hitchatwork commented Feb 22, 2021

andrewkroh commented Feb 22, 2021

hitchatwork commented Mar 4, 2021

andrewkroh commented Mar 5, 2021

hitchatwork commented Mar 12, 2021

andrewkroh commented Mar 16, 2021

hitchatwork commented Jan 29, 2021 •

edited by andrewkroh

Loading

elasticmachine commented Jan 29, 2021 •

edited by jenkins-beats-ci bot

Loading