out_loki: add ability to add arbitrary http headers and adjust the URI path #7979

alarobric · 2023-09-26T21:47:07Z

Following pattern from prometheus_remote_write, this adds the option to add arbitrary http headers.

Tested against a Loki endpoint and logging reverse proxy.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1207

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

alarobric · 2023-09-26T22:22:38Z

Example configuration:

[SERVICE]
    flush        1
    daemon       Off
    log_level    debug

[INPUT]
    name cpu

[INPUT]
    Name dummy

[OUTPUT]
    name                       loki
    match                      *
    Host                       example.com
    uri                        /microservices/test/loki/api/v1/push
    port                       443
    labels                     job=edge-proxy-custom,host=${HOSTNAME}
    tls                        on
    tls.verify                 on
    bearer_token               sdlfhDSFKDdfdFDfd223FDd
    header                     X-testing abc
    header                     X-test2 def

Example output and valgrind

$ valgrind --leak-check=full ./bin/fluent-bit -c ../fluent-bit-minimal.conf 
==30953== Memcheck, a memory error detector
==30953== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30953== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==30953== Command: ./bin/fluent-bit -c ../fluent-bit.conf
==30953== 
Fluent Bit v2.1.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/26 22:15:03] [ info] Configuration:
[2023/09/26 22:15:03] [ info]  flush time     | 1.000000 seconds
[2023/09/26 22:15:03] [ info]  grace          | 5 seconds
[2023/09/26 22:15:03] [ info]  daemon         | 0
[2023/09/26 22:15:03] [ info] ___________
[2023/09/26 22:15:03] [ info]  inputs:
[2023/09/26 22:15:03] [ info]      cpu
[2023/09/26 22:15:03] [ info]      dummy
[2023/09/26 22:15:03] [ info] ___________
[2023/09/26 22:15:03] [ info]  filters:
[2023/09/26 22:15:03] [ info] ___________
[2023/09/26 22:15:03] [ info]  outputs:
[2023/09/26 22:15:03] [ info]      loki.0
[2023/09/26 22:15:03] [ info] ___________
[2023/09/26 22:15:03] [ info]  collectors:
[2023/09/26 22:15:03] [ info] [fluent bit] version=2.1.10, commit=adf20cc8b3, pid=30953
[2023/09/26 22:15:03] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/09/26 22:15:03] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/26 22:15:03] [ info] [cmetrics] version=0.6.3
[2023/09/26 22:15:03] [ info] [ctraces ] version=0.3.1
[2023/09/26 22:15:03] [ info] [input:cpu:cpu.0] initializing
[2023/09/26 22:15:03] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2023/09/26 22:15:03] [debug] [cpu:cpu.0] created event channels: read=24 write=25
[2023/09/26 22:15:03] [ info] [input:dummy:dummy.1] initializing
[2023/09/26 22:15:03] [ info] [input:dummy:dummy.1] storage_strategy='memory' (memory only)
[2023/09/26 22:15:03] [debug] [dummy:dummy.1] created event channels: read=26 write=27
[2023/09/26 22:15:03] [debug] [loki:loki.0] created event channels: read=28 write=29
[2023/09/26 22:15:03] [ info] [output:loki:loki.0] configured, hostname=example.com:443
[2023/09/26 22:15:03] [debug] [router] match rule cpu.0:loki.0
[2023/09/26 22:15:03] [debug] [router] match rule dummy.1:loki.0
[2023/09/26 22:15:03] [ info] [sp] stream processor started
[2023/09/26 22:15:04] [debug] [input chunk] update output instances with new chunk size diff=1121, records=1, input=cpu.0
[2023/09/26 22:15:04] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.1
[2023/09/26 22:15:05] [debug] [task] created task=0x5474b90 id=0 OK
[2023/09/26 22:15:05] [debug] [task] created task=0x547b150 id=1 OK
[2023/09/26 22:15:05] [debug] [input chunk] update output instances with new chunk size diff=1121, records=1, input=cpu.0
[2023/09/26 22:15:05] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.1
[2023/09/26 22:15:06] [debug] [task] created task=0x55be720 id=2 OK
[2023/09/26 22:15:06] [debug] [task] created task=0x55c4c60 id=3 OK
[2023/09/26 22:15:06] [debug] [input chunk] update output instances with new chunk size diff=1121, records=1, input=cpu.0
[2023/09/26 22:15:06] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.1
[2023/09/26 22:15:06] [debug] [upstream] KA connection #37 to example.com:443 is connected
[2023/09/26 22:15:06] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:06] [debug] [upstream] KA connection #38 to example.com:443 is connected
[2023/09/26 22:15:06] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:06] [debug] [upstream] KA connection #39 to example.com:443 is connected
[2023/09/26 22:15:06] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:06] [debug] [upstream] KA connection #40 to example.com:443 is connected
[2023/09/26 22:15:06] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:06] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:07] [debug] [upstream] KA connection #37 to example.com:443 is now available
[2023/09/26 22:15:07] [debug] [out flush] cb_destroy coro_id=0
[2023/09/26 22:15:07] [debug] [task] destroy task=0x5474b90 (task_id=0)
[2023/09/26 22:15:07] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:07] [debug] [upstream] KA connection #39 to example.com:443 is now available
[2023/09/26 22:15:07] [debug] [out flush] cb_destroy coro_id=2
[2023/09/26 22:15:07] [debug] [task] destroy task=0x55be720 (task_id=2)
[2023/09/26 22:15:07] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:07] [debug] [upstream] KA connection #40 to example.com:443 is now available
[2023/09/26 22:15:07] [debug] [out flush] cb_destroy coro_id=3
[2023/09/26 22:15:07] [debug] [task] destroy task=0x55c4c60 (task_id=3)
[2023/09/26 22:15:07] [debug] [task] created task=0x6727420 id=0 OK
[2023/09/26 22:15:07] [debug] [task] created task=0x672d960 id=2 OK
[2023/09/26 22:15:07] [debug] [input chunk] update output instances with new chunk size diff=1121, records=1, input=cpu.0
[2023/09/26 22:15:07] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.1
[2023/09/26 22:15:07] [debug] [upstream] KA connection #37 to example.com:443 has been assigned (recycled)
[2023/09/26 22:15:07] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:07] [debug] [upstream] KA connection #39 to example.com:443 has been assigned (recycled)
[2023/09/26 22:15:07] [debug] [http_client] not using http_proxy for header
^C[2023/09/26 22:15:07] [engine] caught signal (SIGINT)
[2023/09/26 22:15:07] [debug] [task] created task=0x67d94e0 id=3 OK
[2023/09/26 22:15:07] [debug] [task] created task=0x67dfa20 id=4 OK
[2023/09/26 22:15:07] [ warn] [engine] service will shutdown in max 5 seconds
[2023/09/26 22:15:07] [ info] [input] pausing cpu.0
[2023/09/26 22:15:07] [ info] [input] pausing dummy.1
[2023/09/26 22:15:07] [debug] [upstream] KA connection #40 to example.com:443 has been assigned (recycled)
[2023/09/26 22:15:07] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:07] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:07] [debug] [upstream] KA connection #39 to example.com:443 is now available
[2023/09/26 22:15:07] [debug] [out flush] cb_destroy coro_id=5
[2023/09/26 22:15:07] [debug] [task] destroy task=0x672d960 (task_id=2)
[2023/09/26 22:15:07] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:07] [debug] [upstream] KA connection #37 to example.com:443 is now available
[2023/09/26 22:15:07] [debug] [out flush] cb_destroy coro_id=4
[2023/09/26 22:15:07] [debug] [task] destroy task=0x6727420 (task_id=0)
[2023/09/26 22:15:07] [debug] [upstream] KA connection #34 to example.com:443 is connected
[2023/09/26 22:15:07] [debug] [http_client] not using http_proxy for header
[2023/09/26 22:15:07] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:07] [debug] [upstream] KA connection #40 to example.com:443 is now available
[2023/09/26 22:15:07] [debug] [out flush] cb_destroy coro_id=6
[2023/09/26 22:15:07] [debug] [task] destroy task=0x67d94e0 (task_id=3)
[2023/09/26 22:15:08] [ info] [task] cpu/cpu.0 has 0 pending task(s):
[2023/09/26 22:15:08] [ info] [task] dummy/dummy.1 has 2 pending task(s):
[2023/09/26 22:15:08] [ info] [task]   task_id=1 still running on route(s): loki/loki.0 
[2023/09/26 22:15:08] [ info] [task]   task_id=4 still running on route(s): loki/loki.0 
[2023/09/26 22:15:08] [ info] [input] pausing cpu.0
[2023/09/26 22:15:08] [ info] [input] pausing dummy.1
[2023/09/26 22:15:09] [ info] [input] pausing cpu.0
[2023/09/26 22:15:09] [ info] [input] pausing dummy.1
[2023/09/26 22:15:09] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:09] [debug] [upstream] KA connection #34 to example.com:443 is now available
[2023/09/26 22:15:09] [debug] [out flush] cb_destroy coro_id=7
[2023/09/26 22:15:09] [debug] [task] destroy task=0x67dfa20 (task_id=4)
[2023/09/26 22:15:10] [ info] [input] pausing cpu.0
[2023/09/26 22:15:10] [ info] [input] pausing dummy.1
[2023/09/26 22:15:10] [debug] [output:loki:loki.0] example.com:443, HTTP status=204
[2023/09/26 22:15:10] [debug] [upstream] KA connection #38 to example.com:443 is now available
[2023/09/26 22:15:10] [debug] [out flush] cb_destroy coro_id=1
[2023/09/26 22:15:10] [debug] [task] destroy task=0x547b150 (task_id=1)
[2023/09/26 22:15:11] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/26 22:15:11] [ info] [input] pausing cpu.0
[2023/09/26 22:15:11] [ info] [input] pausing dummy.1
==30953== 
==30953== HEAP SUMMARY:
==30953==     in use at exit: 0 bytes in 0 blocks
==30953==   total heap usage: 11,829 allocs, 11,829 frees, 4,859,246 bytes allocated
==30953== 
==30953== All heap blocks were freed -- no leaks are possible
==30953== 
==30953== For lists of detected and suppressed errors, rerun with: -s
==30953== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

edsiper · 2023-10-17T14:34:33Z

@alarobric thanks for this contribution. Note that the uri property was recently handled (sorry, I didn't see this PR on that moment), can you adjust this PR to the headers only part ?

…I path Following pattern from prometheus_remote_write, this adds the option to add arbitrary http headers. It also adds the ability to manually set the URI path. Previously it was hardcoded as `/loki/api/v1/push`. Existing behaviour is maintained and this is an optional override. Tested against a Loki endpoint and logging reverse proxy. Signed-off-by: Alan Richards <alarobric@gmail.com>

alarobric · 2023-10-17T22:59:51Z

@edsiper , Sure, I've rebased and updated the PR to remove the now-implemented custom URI. This now just adds the ability to set HTTP headers.

Do you want me to edit the docs PR as well? I don't see a change documenting the URL addition from #8040 yet.

edsiper · 2023-11-08T13:31:31Z

running CI

…I path (fluent#7979) Following the pattern from prometheus_remote_write, this adds the option to add arbitrary HTTP headers. Tested against a Loki endpoint and logging reverse proxy. Signed-off-by: Alan Richards <alarobric@gmail.com>

alarobric requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners September 26, 2023 21:47

github-actions bot added the docs-required label Sep 26, 2023

alarobric mentioned this pull request Sep 26, 2023

pipeline: outputs: loki: Add new header options fluent/fluent-bit-docs#1207

Merged

alarobric temporarily deployed to pr October 5, 2023 12:06 — with GitHub Actions Inactive

alarobric temporarily deployed to pr October 5, 2023 12:34 — with GitHub Actions Inactive

alarobric force-pushed the out_loki-optional-uri-and-headers branch from 23f5294 to eeb0778 Compare October 17, 2023 22:57

alarobric temporarily deployed to pr November 8, 2023 13:30 — with GitHub Actions Inactive

alarobric temporarily deployed to pr November 8, 2023 13:59 — with GitHub Actions Inactive

edsiper merged commit db8487d into fluent:master Nov 8, 2023
43 checks passed

BrewTestBot mentioned this pull request Nov 9, 2023

fluent-bit 2.2.0 Homebrew/homebrew-core#153758

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_loki: add ability to add arbitrary http headers and adjust the URI path #7979

out_loki: add ability to add arbitrary http headers and adjust the URI path #7979

alarobric commented Sep 26, 2023 •

edited

Loading

alarobric commented Sep 26, 2023

edsiper commented Oct 17, 2023

alarobric commented Oct 17, 2023 •

edited

Loading

edsiper commented Nov 8, 2023

out_loki: add ability to add arbitrary http headers and adjust the URI path #7979

out_loki: add ability to add arbitrary http headers and adjust the URI path #7979

Conversation

alarobric commented Sep 26, 2023 • edited Loading

alarobric commented Sep 26, 2023

edsiper commented Oct 17, 2023

alarobric commented Oct 17, 2023 • edited Loading

edsiper commented Nov 8, 2023

alarobric commented Sep 26, 2023 •

edited

Loading

alarobric commented Oct 17, 2023 •

edited

Loading