Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Filebeat Zeek Weird Ingest Pipeline #15906

Merged
merged 3 commits into from
Feb 3, 2020
Merged

Fix Filebeat Zeek Weird Ingest Pipeline #15906

merged 3 commits into from
Feb 3, 2020

Conversation

0huey
Copy link
Contributor

@0huey 0huey commented Jan 28, 2020

Some Zeek Weird logs do not contain IP addresses, causing the warning seen below:

Logstash Output
[2020-01-28T15:49:35,993][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-zeek-2020.01.28", :routing=>nil, :_type=>"_doc", :pipeline=>"filebeat-7.5.2-zeek-weird-pipeline"}, #<LogStash::Event:0x3f1f2270>], :response=>{"index"=>{"_index"=>"filebeat-zeek-2020.01.28", "_type"=>"_doc", "_id"=>"r3PX7G8BxFIJZtUR_Ruu", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [destination.ip] of type [ip] in document with id 'r3PX7G8BxFIJZtUR_Ruu'. Preview of field's value: ''", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'' is not an IP string literal."}}}}}

Sample from weird.log
{"ts":1580227259.342809,"name":"non_ip_packet_in_ethernet","notice":false,"peer":"ens3f1-4"}

@0huey 0huey requested a review from a team as a code owner January 28, 2020 16:13
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@kaiyan-sheng kaiyan-sheng added Filebeat Filebeat review Team:Services (Deprecated) Label for the former Integrations-Services team labels Jan 28, 2020
@kaiyan-sheng
Copy link
Contributor

Hi @Xander33, thanks for contributing! Could you also add a test file with Zeek Weird logs which doesn't contain IP address please? Similar to x-pack/filebeat/module/zeek/files/test/files-json.log and create an expected json file by running MODULES_PATH=pwd/module GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=zeek nosetests -v -d -s tests/system/test_xpack_modules.py.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@andresrc andresrc removed Team:Services (Deprecated) Label for the former Integrations-Services team [zube]: Inbox labels Jan 29, 2020
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please also add an entry to the CHANGELOG.next.asciidoc file in the Filebeat bugfix section. Thanks

x-pack/filebeat/module/zeek/weird/ingest/pipeline.json Outdated Show resolved Hide resolved
x-pack/filebeat/module/zeek/weird/ingest/pipeline.json Outdated Show resolved Hide resolved
@0huey
Copy link
Contributor Author

0huey commented Jan 31, 2020

@andrewkroh good point.

@kaiyan-sheng How does that look?

@andrewkroh
Copy link
Member

This needs a quick rebase to resolve the conflicts.

jenkins, test this

@kaiyan-sheng
Copy link
Contributor

jenkins, test this please

@kaiyan-sheng
Copy link
Contributor

Thanks @Xander33 for contributing! I will merge this PR and cherrypick to 7.x branch after CI passed.

@kaiyan-sheng kaiyan-sheng self-assigned this Feb 3, 2020
@kaiyan-sheng kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Feb 3, 2020
@kaiyan-sheng kaiyan-sheng merged commit 6b9c8cb into elastic:master Feb 3, 2020
@kaiyan-sheng kaiyan-sheng added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Feb 3, 2020
kaiyan-sheng added a commit that referenced this pull request Feb 4, 2020
…16037)

* Fix Filebeat Zeek Weird Ingest Pipeline (#15906)

* Update pipeline.json
* Fix zeek weird pipeline

(cherry picked from commit 6b9c8cb)

* update changelog

Co-authored-by: xander33 <34045167+xander33@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants