Skip to content

out_s3: add support for server-side encryption headers #10901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

out_s3: add support for server-side encryption headers #10901

wants to merge 1 commit into from

Conversation

rishabh8481
Copy link

Add configuration options to enable server-side encryption when uploading files to S3, supporting both AWS managed and customer managed KMS keys.

This implementation follows AWS S3 documentation guidelines and adds support for the following encryption headers:

  • x-amz-server-side-encryption
    • Supported values: AES256, aws:kms, aws:kms:dsse
  • x-amz-server-side-encryption-aws-kms-key-id
    • Required when using customer managed KMS keys

The feature allows users to encrypt their log data at rest in S3 buckets using their preferred encryption method.

Reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change

Masking s3 bucket name and kms key arn in the logs

valid_no_sse.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    debug
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 15:39:58.229157000] [ info] [output:s3:s3.0] Running upload timer callback (cb_s3_upload)..
[2025/09/18 15:39:59.222972000] [debug] [task] created task=0x6000012bc000 id=0 OK
[2025/09/18 15:39:59.223026000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 15:39:59.223371000] [debug] [out flush] cb_destroy coro_id=7
[2025/09/18 15:39:59.223581000] [debug] [task] destroy task=0x6000012bc000 (task_id=0)
[2025/09/18 15:40:00.222335000] [debug] [task] created task=0x6000012bc000 id=0 OK
[2025/09/18 15:40:00.222466000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 15:40:00.222879000] [debug] [out flush] cb_destroy coro_id=8
[2025/09/18 15:40:00.222984000] [debug] [task] destroy task=0x6000012bc000 (task_id=0)
[2025/09/18 15:40:01.223896000] [debug] [task] created task=0x6000012bc000 id=0 OK
[2025/09/18 15:40:01.223948000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 15:40:01.224140000] [debug] [out flush] cb_destroy coro_id=9
[2025/09/18 15:40:01.224208000] [debug] [task] destroy task=0x6000012bc000 (task_id=0)
[2025/09/18 15:40:02.223357000] [debug] [task] created task=0x6000012b8000 id=0 OK
[2025/09/18 15:40:02.223406000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 15:40:02.223564000] [debug] [out flush] cb_destroy coro_id=10
[2025/09/18 15:40:02.223612000] [debug] [task] destroy task=0x6000012b8000 (task_id=0)
[2025/09/18 15:40:03.223035000] [debug] [task] created task=0x6000012b8000 id=0 OK
[2025/09/18 15:40:03.223121000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 15:40:03.223327000] [ info] [output:s3:s3.0] upload_timeout reached for test.local
[2025/09/18 15:40:03.223363000] [debug] [output:s3:s3.0] Running upload timer callback (upload_queue)..
[2025/09/18 15:40:03.375252000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is connected
[2025/09/18 15:40:03.375305000] [debug] [http_client] not using http_proxy for header
[2025/09/18 15:40:03.375347000] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/09/18 15:40:03.485907000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is now available
[2025/09/18 15:40:03.486295000] [debug] [output:s3:s3.0] PutObject http status=200
[2025/09/18 15:40:03.486316000] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/test.local/2025/09/18/22/39/51-objectnh6y87wB
[2025/09/18 15:40:03.491361000] [debug] [out flush] cb_destroy coro_id=11
[2025/09/18 15:40:03.491455000] [debug] [task] destroy task=0x6000012b8000 (task_id=0)
[2025/09/18 15:40:04.222109000] [debug] [task] created task=0x6000012b8000 id=0 OK
[2025/09/18 15:40:04.222181000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 15:40:04.224285000] [debug] [out flush] cb_destroy coro_id=12
[2025/09/18 15:40:04.224369000] [debug] [task] destroy task=0x6000012b8000 (task_id=0)

valid_sse_aes256.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption AES256
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:10:33.650413000] [ info] [output:s3:s3.0] Running upload timer callback (cb_s3_upload)..
[2025/09/18 16:10:34.644266000] [debug] [task] created task=0x600001f02940 id=0 OK
[2025/09/18 16:10:34.644300000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:10:34.644446000] [debug] [out flush] cb_destroy coro_id=7
[2025/09/18 16:10:34.644522000] [debug] [task] destroy task=0x600001f02940 (task_id=0)
[2025/09/18 16:10:35.644340000] [debug] [task] created task=0x600001f04000 id=0 OK
[2025/09/18 16:10:35.644470000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:10:35.645281000] [debug] [out flush] cb_destroy coro_id=8
[2025/09/18 16:10:35.645559000] [debug] [task] destroy task=0x600001f04000 (task_id=0)
[2025/09/18 16:10:36.644331000] [debug] [task] created task=0x600001f38000 id=0 OK
[2025/09/18 16:10:36.644445000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:10:36.644875000] [debug] [out flush] cb_destroy coro_id=9
[2025/09/18 16:10:36.645251000] [debug] [task] destroy task=0x600001f38000 (task_id=0)
[2025/09/18 16:10:37.644320000] [debug] [task] created task=0x600001f02940 id=0 OK
[2025/09/18 16:10:37.644373000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:10:37.644548000] [debug] [out flush] cb_destroy coro_id=10
[2025/09/18 16:10:37.644587000] [debug] [task] destroy task=0x600001f02940 (task_id=0)
[2025/09/18 16:10:38.644426000] [debug] [task] created task=0x600001f38000 id=0 OK
[2025/09/18 16:10:38.644535000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:10:38.644925000] [ info] [output:s3:s3.0] upload_timeout reached for test.local
[2025/09/18 16:10:38.644964000] [debug] [output:s3:s3.0] Running upload timer callback (upload_queue)..
[2025/09/18 16:10:38.802014000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is connected
[2025/09/18 16:10:38.802090000] [debug] [http_client] not using http_proxy for header
[2025/09/18 16:10:38.802287000] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/09/18 16:10:38.912426000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is now available
[2025/09/18 16:10:38.912491000] [debug] [output:s3:s3.0] PutObject http status=200
[2025/09/18 16:10:38.912504000] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/test.local/2025/09/18/23/10/26-object5miwK9gN
[2025/09/18 16:10:38.924801000] [debug] [out flush] cb_destroy coro_id=11

valid_sse_kms.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption aws:kms
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:12:04.671431000] [ info] [output:s3:s3.0] Running upload timer callback (cb_s3_upload)..
[2025/09/18 16:12:05.666649000] [debug] [task] created task=0x600001234000 id=0 OK
[2025/09/18 16:12:05.666755000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:12:05.667033000] [debug] [out flush] cb_destroy coro_id=7
[2025/09/18 16:12:05.667112000] [debug] [task] destroy task=0x600001234000 (task_id=0)
[2025/09/18 16:12:06.666036000] [debug] [task] created task=0x6000012300b0 id=0 OK
[2025/09/18 16:12:06.666808000] [debug] [out flush] cb_destroy coro_id=8
[2025/09/18 16:12:06.666292000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:12:06.666902000] [debug] [task] destroy task=0x6000012300b0 (task_id=0)
[2025/09/18 16:12:07.667042000] [debug] [task] created task=0x6000012300b0 id=0 OK
[2025/09/18 16:12:07.667087000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:12:07.667475000] [debug] [out flush] cb_destroy coro_id=9
[2025/09/18 16:12:07.667528000] [debug] [task] destroy task=0x6000012300b0 (task_id=0)
[2025/09/18 16:12:08.666539000] [debug] [task] created task=0x600001234000 id=0 OK
[2025/09/18 16:12:08.666604000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:12:08.667165000] [debug] [out flush] cb_destroy coro_id=10
[2025/09/18 16:12:08.667601000] [debug] [task] destroy task=0x600001234000 (task_id=0)
[2025/09/18 16:12:09.672375000] [debug] [task] created task=0x600001234000 id=0 OK
[2025/09/18 16:12:09.672487000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:12:09.672899000] [ info] [output:s3:s3.0] upload_timeout reached for test.local
[2025/09/18 16:12:09.672973000] [debug] [output:s3:s3.0] Running upload timer callback (upload_queue)..
[2025/09/18 16:12:09.835821000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is connected
[2025/09/18 16:12:09.835883000] [debug] [http_client] not using http_proxy for header
[2025/09/18 16:12:09.835909000] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/09/18 16:12:09.975700000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is now available
[2025/09/18 16:12:09.975780000] [debug] [output:s3:s3.0] PutObject http status=200
[2025/09/18 16:12:09.975797000] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/test.local/2025/09/18/23/11/57-objectYm7BlNeX
[2025/09/18 16:12:09.978989000] [debug] [out flush] cb_destroy coro_id=11

valid_sse_kms_dsse.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption aws:kms:dsse
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:15:24.436299000] [ info] [output:s3:s3.0] Running upload timer callback (cb_s3_upload)..
[2025/09/18 16:15:25.431270000] [debug] [task] created task=0x6000019875a0 id=0 OK
[2025/09/18 16:15:25.431464000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:15:25.431750000] [debug] [out flush] cb_destroy coro_id=7
[2025/09/18 16:15:25.431944000] [debug] [task] destroy task=0x6000019875a0 (task_id=0)
[2025/09/18 16:15:26.431728000] [debug] [task] created task=0x6000019875a0 id=0 OK
[2025/09/18 16:15:26.432098000] [debug] [out flush] cb_destroy coro_id=8
[2025/09/18 16:15:26.431814000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:15:26.432388000] [debug] [task] destroy task=0x6000019875a0 (task_id=0)
[2025/09/18 16:15:27.431259000] [debug] [task] created task=0x600001988000 id=0 OK
[2025/09/18 16:15:27.431290000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:15:27.431705000] [debug] [out flush] cb_destroy coro_id=9
[2025/09/18 16:15:27.432062000] [debug] [task] destroy task=0x600001988000 (task_id=0)
[2025/09/18 16:15:28.431164000] [debug] [task] created task=0x600001988000 id=0 OK
[2025/09/18 16:15:28.431322000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:15:28.431552000] [debug] [out flush] cb_destroy coro_id=10
[2025/09/18 16:15:28.431856000] [debug] [task] destroy task=0x600001988000 (task_id=0)
[2025/09/18 16:15:29.431113000] [debug] [task] created task=0x6000019875a0 id=0 OK
[2025/09/18 16:15:29.431155000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:15:29.431351000] [ info] [output:s3:s3.0] upload_timeout reached for test.local
[2025/09/18 16:15:29.431374000] [debug] [output:s3:s3.0] Running upload timer callback (upload_queue)..
[2025/09/18 16:15:29.573323000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is connected
[2025/09/18 16:15:29.573368000] [debug] [http_client] not using http_proxy for header
[2025/09/18 16:15:29.573391000] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/09/18 16:15:29.718938000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is now available
[2025/09/18 16:15:29.718971000] [debug] [output:s3:s3.0] PutObject http status=200
[2025/09/18 16:15:29.718983000] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/test.local/2025/09/18/23/15/17-objectgIZH5Gim
[2025/09/18 16:15:29.723072000] [debug] [out flush] cb_destroy coro_id=11

valid_sse_kms_with_key.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption aws:kms
    server-side-encryption-aws-kms-key-id <KMS_KEY_ARN>
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:16:35.537035000] [ info] [output:s3:s3.0] Running upload timer callback (cb_s3_upload)..
[2025/09/18 16:16:36.533014000] [debug] [task] created task=0x600000f1c000 id=0 OK
[2025/09/18 16:16:36.533110000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:16:36.533396000] [debug] [out flush] cb_destroy coro_id=7
[2025/09/18 16:16:36.533640000] [debug] [task] destroy task=0x600000f1c000 (task_id=0)
[2025/09/18 16:16:37.531636000] [debug] [task] created task=0x600000f0e5d0 id=0 OK
[2025/09/18 16:16:37.531687000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:16:37.531815000] [debug] [out flush] cb_destroy coro_id=8
[2025/09/18 16:16:37.531856000] [debug] [task] destroy task=0x600000f0e5d0 (task_id=0)
[2025/09/18 16:16:38.533061000] [debug] [task] created task=0x600000f18000 id=0 OK
[2025/09/18 16:16:38.533153000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:16:38.533489000] [debug] [out flush] cb_destroy coro_id=9
[2025/09/18 16:16:38.533702000] [debug] [task] destroy task=0x600000f18000 (task_id=0)
[2025/09/18 16:16:39.533004000] [debug] [task] created task=0x600000f1c000 id=0 OK
[2025/09/18 16:16:39.533117000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:16:39.533463000] [debug] [out flush] cb_destroy coro_id=10
[2025/09/18 16:16:39.533529000] [debug] [task] destroy task=0x600000f1c000 (task_id=0)
[2025/09/18 16:16:40.532926000] [debug] [task] created task=0x600000f1c000 id=0 OK
[2025/09/18 16:16:40.533016000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:16:40.533254000] [ info] [output:s3:s3.0] upload_timeout reached for test.local
[2025/09/18 16:16:40.533292000] [debug] [output:s3:s3.0] Running upload timer callback (upload_queue)..
[2025/09/18 16:16:40.677553000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is connected
[2025/09/18 16:16:40.677603000] [debug] [http_client] not using http_proxy for header
[2025/09/18 16:16:40.677632000] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/09/18 16:16:40.809545000] [debug] [upstream] KA connection #55 to s3.us-east-1.amazonaws.com:443 is now available
[2025/09/18 16:16:40.809637000] [debug] [output:s3:s3.0] PutObject http status=200

valid_sse_kms_dsse_with_key.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption aws:kms:dsse
    server-side-encryption-aws-kms-key-id <KMS_KEY_ARN>
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:18:42.634900000] [ info] [output:s3:s3.0] Running upload timer callback (cb_s3_upload)..
[2025/09/18 16:18:43.629923000] [debug] [task] created task=0x600003688000 id=0 OK
[2025/09/18 16:18:43.630044000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:18:43.630517000] [debug] [out flush] cb_destroy coro_id=7
[2025/09/18 16:18:43.630695000] [debug] [task] destroy task=0x600003688000 (task_id=0)
[2025/09/18 16:18:44.629875000] [debug] [task] created task=0x600003688000 id=0 OK
[2025/09/18 16:18:44.629931000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:18:44.630275000] [debug] [out flush] cb_destroy coro_id=8
[2025/09/18 16:18:44.630375000] [debug] [task] destroy task=0x600003688000 (task_id=0)
[2025/09/18 16:18:45.630433000] [debug] [task] created task=0x60000368c000 id=0 OK
[2025/09/18 16:18:45.630606000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:18:45.630890000] [debug] [out flush] cb_destroy coro_id=9
[2025/09/18 16:18:45.631078000] [debug] [task] destroy task=0x60000368c000 (task_id=0)
[2025/09/18 16:18:46.629946000] [debug] [task] created task=0x600003688000 id=0 OK
[2025/09/18 16:18:46.629995000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:18:46.630418000] [debug] [out flush] cb_destroy coro_id=10
[2025/09/18 16:18:46.630521000] [debug] [task] destroy task=0x600003688000 (task_id=0)
[2025/09/18 16:18:47.630087000] [debug] [task] created task=0x600003688000 id=0 OK
[2025/09/18 16:18:47.630202000] [debug] [output:s3:s3.0] task_id=0 assigned to thread #0
[2025/09/18 16:18:47.630713000] [ info] [output:s3:s3.0] upload_timeout reached for test.local
[2025/09/18 16:18:47.630776000] [debug] [output:s3:s3.0] Running upload timer callback (upload_queue)..
[2025/09/18 16:18:47.822718000] [debug] [upstream] KA connection #54 to s3.us-east-1.amazonaws.com:443 is connected
[2025/09/18 16:18:47.823122000] [debug] [http_client] not using http_proxy for header
[2025/09/18 16:18:47.823230000] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/09/18 16:18:47.962121000] [debug] [upstream] KA connection #54 to s3.us-east-1.amazonaws.com:443 is now available
[2025/09/18 16:18:47.962210000] [debug] [output:s3:s3.0] PutObject http status=200

invalid_sse_value.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption invalid-value
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:19:59.663900000] [debug] [tls] finished loading keychain certificates, total loaded: 153
[2025/09/18 16:19:59.666960000] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
[2025/09/18 16:19:59.666986000] [ info] [output:s3:s3.0] total_file_size is less than 10 MB, will use PutObject API
[2025/09/18 16:19:59.667235000] [error] [output:s3:s3.0] Invalid sse value 'invalid-value'. Valid values: AES256, aws:kms, aws:kms:dsse
[2025/09/18 16:19:59.667241000] [error] [output] failed to initialize 's3' plugin
[2025/09/18 16:19:59.667878000] [error] [engine] output initialization failed
[2025/09/18 16:19:59.667929000] [ info] [input] pausing dummy.0

kms_key_without_sse.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption-aws-kms-key-id <KMS_KEY_ARN>
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:20:55.137489000] [debug] [tls] finished loading keychain certificates, total loaded: 153
[2025/09/18 16:20:55.141456000] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
[2025/09/18 16:20:55.141484000] [ info] [output:s3:s3.0] total_file_size is less than 10 MB, will use PutObject API
[2025/09/18 16:20:55.141741000] [error] [output:s3:s3.0] server-side-encryption-aws-kms-key-id requires server-side-encryption to be 'aws:kms' or 'aws:kms:dsse'
[2025/09/18 16:20:55.141753000] [error] [output] failed to initialize 's3' plugin
[2025/09/18 16:20:55.142377000] [error] [engine] output initialization failed
[2025/09/18 16:20:55.142444000] [ info] [input] pausing dummy.0

kms_key_with_aes256.conf

[SERVICE]
    flush        1
    daemon       off
    log_level    info
    grace        1

[INPUT]
    name dummy
    tag  test.local
    rate 10

[OUTPUT]
    name s3
    match *
    bucket <BUCKET_NAME>
    region us-east-1
    server-side-encryption AES256
    server-side-encryption-aws-kms-key-id <KMS_KEY_ARN>
    total_file_size 1M
    upload_timeout 10s

[2025/09/18 16:21:18.393945000] [debug] [tls] finished loading keychain certificates, total loaded: 153
[2025/09/18 16:21:18.398740000] [ info] [output:s3:s3.0] Using upload size 1000000 bytes
[2025/09/18 16:21:18.398768000] [ info] [output:s3:s3.0] total_file_size is less than 10 MB, will use PutObject API
[2025/09/18 16:21:18.398962000] [error] [output:s3:s3.0] server-side-encryption-aws-kms-key-id requires server-side-encryption to be 'aws:kms' or 'aws:kms:dsse', got 'AES256'
[2025/09/18 16:21:18.398969000] [error] [output] failed to initialize 's3' plugin
[2025/09/18 16:21:18.399562000] [error] [engine] output initialization failed
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@rishabh8481
out_s3: add support for server-side encryption headers
6382ec1 
Add configuration options to enable server-side encryption when uploading
files to S3, supporting both AWS managed and customer managed KMS keys.

This implementation follows AWS S3 documentation guidelines and adds
support for the following encryption headers:

* x-amz-server-side-encryption
  - Supported values: AES256, aws:kms, aws:kms:dsse
* x-amz-server-side-encryption-aws-kms-key-id
  - Required when using customer managed KMS keys

The feature allows users to encrypt their log data at rest in S3 buckets
using their preferred encryption method.

Reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html

Signed-off-by: Rishabh Sharma <rishabh84.81@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PettitWesley PettitWesley Awaiting requested review from PettitWesley PettitWesley is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@rishabh8481