Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add module for AWS S3 server access log #13170

Merged
merged 14 commits into from
Aug 16, 2019
Merged

[Filebeat] Add module for AWS S3 server access log #13170

merged 14 commits into from
Aug 16, 2019

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Aug 3, 2019

This module is to parse S3 server access logs that are retrieved by s3 input. Please see https://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html for more details on S3 server access log format.
Screen Shot 2019-08-09 at 12 31 05 PM

closes #12880

How to test this:

  1. Setup S3 bucket and SQS notification: https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html#step1-create-sqs-queue-for-notification
  2. Pull this PR(if it's not merged) and build a new filebeat binary under x-pack:
    kaiyansheng@KaiyanMacBookPro:~/go/src/github.com/elastic/beats/x-pack/filebeat (s3_access_log)$ mage update; mage build
  3. Enable aws module:
    kaiyansheng@KaiyanMacBookPro:~/go/src/github.com/elastic/beats/x-pack/filebeat (s3_access_log)$ ./filebeat modules enable aws
  4. Update aws.yml with SQS url and AWS credentials:
- module: aws
  s3access:
    enabled: true

    # AWS SQS queue url
    var.queue_url: https://sqs.ap-southeast-1.amazonaws.com/123/test-s3-notification

    # Profile name for aws credential
    var.credential_profile_name: test-mb
  1. Disable filebeat default log input in filebeat.yml by comment out the part in below:
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
  1. Start metricbeat with ./metricbeat -e

@kaiyan-sheng kaiyan-sheng self-assigned this Aug 3, 2019
@kaiyan-sheng kaiyan-sheng added Team:Integrations Label for the Integrations team [zube]: In Progress Filebeat Filebeat in progress Pull request is currently in progress. v7.4.0 labels Aug 3, 2019
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review August 7, 2019 19:14
@kaiyan-sheng kaiyan-sheng requested review from a team as code owners August 7, 2019 19:14
@kaiyan-sheng
Copy link
Contributor Author

jenkins, test this please

@kaiyan-sheng kaiyan-sheng added [zube]: In Review test-plan Add this PR to be manual test plan and removed [zube]: In Progress labels Aug 12, 2019
@kaiyan-sheng
Copy link
Contributor Author

jenkins, test this please

@kaiyan-sheng kaiyan-sheng merged commit 3f5e799 into elastic:master Aug 16, 2019
@kaiyan-sheng kaiyan-sheng deleted the s3_access_log branch August 16, 2019 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat in progress Pull request is currently in progress. Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan v7.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] Add filebeat module for server access logs to use s3 filebeat input
3 participants