Skip to content

smijolovic/fluent-bit-go-s3

 
 

Repository files navigation

fluent-bit s3 output plugin

Build Status Build status

Windows binaries are available in release pages.

This plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.

The configuration typically looks like:

fluent-bit --> AWS S3

Usage

$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf

Or,

$ docker build . -t fluent-bit/s3-plugin

and then, specify Url parameter as environment variables:

$ docker run -it -e="FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey" \
                 -e="FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret" \
                 -e="FLUENT_BIT_BUCKET_NAME=yourbucketname" \
                 -e="FLUENT_BIT_S3_PREFIX=yours3prefix" \
                 -e="FLUENT_BIT_REGION=awsregion" \
                 fluent-bit/s3-plugin

Using docker image from docker hub.

$ docker pull cosmo0920/fluent-bit-go-s3:latest

Other released images are available in DockerHub's fluent-bit-go-s3 image tags.

Prerequisites

  • Go 1.11+
  • gcc (for cgo)

Building

$ make

Configuration Options

Key Description Default value Note
Credential URI of AWS shared credential "" (See Credentials)
AccessKeyID Access key ID of AWS "" (See Credentials)
SecretAccessKey Secret access key ID of AWS "" (See Credentials)
Bucket Bucket name of S3 storage - Mandatory parameter
S3Prefix S3Prefix of S3 key - Mandatory parameter
Region Region of S3 - Mandatory parameter
Compress Choose Compress method "" gzip or plainText("")

Example:

Add this section to fluent-bit.conf:

[Output]
    Name s3
    Match *
    # Credential    /path/to/sharedcredentialfile
    AccessKeyID     yourawsaccesskeyid
    SecretAccessKey yourawssecretaccesskey
    Bucket          yourbucketname
    S3Prefix yours3prefixname
    Region us-east-1
    Compress gzip

fluent-bit-go-s3 supports the following credentials. Users must specify one of them:

Credentials

Specifying credentials is required.

This plugin supports the following credentials:

Shared Credentials

Create the following file which includes credentials:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

Then, specify the following parameter in fluent-bit configuration:

Credential    /path/to/sharedcredentialfile

Static Credentials

Specify the following parameters in fluent-bit configuration:

AccessKeyID     yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey

Environment Credentials

Specify AWS_ACCESS_KEY and AWS_SECRET_KEY as environment variables.

Useful links

About

fluent-bit output plugin for Amazon S3. https://aws.amazon.com/s3/

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 93.6%
  • Makefile 3.5%
  • Dockerfile 2.9%