-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
out_loki: add support for structured metadata #9048
Conversation
Does this add any breaking changes to Loki 2.x users ? |
@edsiper possibly if they define the new So this would be a matter of the docs to caution users not to configure |
Signed-off-by: Jason Tackaberry <tack@urandom.ca>
Thanks @edsiper! I submitted fluent/fluent-bit-docs#1406 for the docs part. |
This PR adds support for structured metadata in the loki output. As of Loki 3.0, structured metadata has been made GA.
Implementation wise, labels and structured metadata are very similar: both are key/value pairs, and both support record accessors, consequently this PR does some light refactoring to avoid nontrivial code duplication between labels and structured metadata.
This PR only implements the analog of the
labels
config parameter, which is calledstructured_metadata
. Analogs forlabel_keys
andlabel_map_path
were not implemented to keep the change both simple and immediately useful, though they could be added later.Fixes #8080.
Testing
Because the code paths are slightly different when labels and/or structured metadata employ record accessors, I have two simple configurations: one that uses structured metadata with record accessors, and one that only uses static values.
Our Loki deployment uses mTLS for auth so I've kept that in the config to represent testing, although the Loki endpoint is obviously redacted.
All tests include a log containing combined output with Fluent Bit run under Valgrind.
Test 0: baseline without structured metadata
0-baseline-without-structured-metadata.log
Test 1: structured metadata with record accessors
1-with-record-accessor.log
Test 2: structured metadata (and labels) without record accessors
2-without-record-accessor.log
Documentation
Will submit a docs PR after this PR gets some initial feedback or indication it's likely to be accepted.
Backporting
N/A. It would be nice to make this in time for Fluent Bit 3.1 though. :)
Licensing
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.