Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #16441 to 7.6: [Filebeat] Check expand_event_list_from_field before checking content-type #16627

Merged
merged 1 commit into from
Feb 26, 2020
Merged

Conversation

kaiyan-sheng
Copy link
Contributor

Cherry-pick of PR #16441 to 7.6 branch. Original message:

What does this PR do?

This PR is to fix s3 input reading json format logs. When expand_event_list_from_field config is given, such as cloudtrail fileset, s3 input should use json.NewDecoder first. If this expand_event_list_from_field is not given, then check content-type and use gzip.NewReader.

Why is it important?

This will fix s3 input when using cloudtrail fileset.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Related issues

Screenshots

When Metadata is set as below:
Screen Shot 2020-02-19 at 3 02 48 PM

I was able to read the cloudtrail json.gz file as shown below in Kibana:
Screen Shot 2020-02-19 at 4 19 52 PM

…-type (#16441)

* Check expand_event_list_from_field before checking content-type

(cherry picked from commit 4bb8fa3)
@kaiyan-sheng kaiyan-sheng merged commit f3f27c6 into elastic:7.6 Feb 26, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_16441_7.6 branch February 26, 2020 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants