Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_datadog: Add support for setting a static hostname #8988

Merged
merged 1 commit into from
Jun 25, 2024

Conversation

jszwedko
Copy link
Contributor

@jszwedko jszwedko commented Jun 20, 2024

This PR adds support for setting a static hostname in the Datadog output plugin. This field is analogous to the existing dd_service and dd_source configuration options that can be used to set a static value.

If unset, the default behavior is backwards compatible. This behavior is to not set an explicity hostname field, but if the record has a field that is detected as the hostname in Datadog (such as host or syslog.hostname), it will be picked up.

Closes: #8971


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

This PR adds support for setting a static hostname in the Datadog output
plugin. This field is analogous to the existing `dd_service` and
`dd_source` configuration options that can be used to set a static
value.

If unset, the default behavior is backwards compatible. This behavior is
to not set an explicity `hostname` field, but if the record has a field
that is detected as the hostname in Datadog (such as `host` or
`syslog.hostname`), it will be picked up.

Closes: fluent#8971

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
@jszwedko
Copy link
Contributor Author

Example configuration file:

pipeline:
  inputs:
  - name: dummy
    tag: dummy
    dummy: |
      {
        "message": "From Fluent Bit",
        "test": "default"
      }

  outputs:
  - name: datadog
    match: "*"
    apikey: ${DD_API_KEY}
    dd_hostname: foobar

You can run this and observe that events published have their hostname detected as foobar:
Screenshot 2024-06-20 at 5 57 55 PM

@jszwedko
Copy link
Contributor Author

Debug output:

[2024/06/20 21:58:35] [ info] Configuration:                                                                                                                                                                                                     [79/168]
[2024/06/20 21:58:35] [ info]  flush time     | 1.000000 seconds
[2024/06/20 21:58:35] [ info]  grace          | 5 seconds
[2024/06/20 21:58:35] [ info]  daemon         | 0
[2024/06/20 21:58:35] [ info] ___________
[2024/06/20 21:58:35] [ info]  inputs:
[2024/06/20 21:58:35] [ info]      dummy
[2024/06/20 21:58:35] [ info] ___________
[2024/06/20 21:58:35] [ info]  filters:
[2024/06/20 21:58:35] [ info] ___________
[2024/06/20 21:58:35] [ info]  outputs:
[2024/06/20 21:58:35] [ info]      datadog.0
[2024/06/20 21:58:35] [ info] ___________
[2024/06/20 21:58:35] [ info]  collectors:
[2024/06/20 21:58:35] [ info] [fluent bit] version=3.1.0, commit=2bef95819e, pid=77752
[2024/06/20 21:58:35] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/06/20 21:58:35] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/06/20 21:58:35] [ info] [cmetrics] version=0.9.1
[2024/06/20 21:58:35] [ info] [ctraces ] version=0.5.1
[2024/06/20 21:58:35] [ info] [input:dummy:dummy.0] initializing
[2024/06/20 21:58:35] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/06/20 21:58:35] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/06/20 21:58:35] [debug] [datadog:datadog.0] created event channels: read=23 write=24
[2024/06/20 21:58:35] [debug] [output:datadog:datadog.0] scheme: http://
[2024/06/20 21:58:35] [debug] [output:datadog:datadog.0] uri: /api/v2/logs
[2024/06/20 21:58:35] [debug] [output:datadog:datadog.0] host: http-intake.logs.datadoghq.com
[2024/06/20 21:58:35] [debug] [output:datadog:datadog.0] port: 80
[2024/06/20 21:58:35] [debug] [output:datadog:datadog.0] json_date_key: timestamp
[2024/06/20 21:58:35] [debug] [output:datadog:datadog.0] compress_gzip: 0
[2024/06/20 21:58:35] [ info] [sp] stream processor started
[2024/06/20 21:58:35] [trace] [input chunk] update output instances with new chunk size diff=59, records=1, input=dummy.0
[2024/06/20 21:58:36] [trace] [task 0x7fcf18018ec0] created (id=0)
[2024/06/20 21:58:36] [debug] [task] created task=0x7fcf18018ec0 id=0 OK
[2024/06/20 21:58:36] [trace] [input chunk] update output instances with new chunk size diff=59, records=1, input=dummy.0
[2024/06/20 21:58:36] [trace] [upstream] get new connection for http-intake.logs.datadoghq.com:80, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2024/06/20 21:58:36] [trace] [net] connection #31 in process to http-intake.logs.datadoghq.com:80
[2024/06/20 21:58:36] [trace] [engine] resuming coroutine=0x7fcf18018fe0
[2024/06/20 21:58:36] [trace] [io] connection OK
[2024/06/20 21:58:36] [debug] [upstream] KA connection #31 to http-intake.logs.datadoghq.com:80 is connected
[2024/06/20 21:58:36] [debug] [http_client] not using http_proxy for header
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [net_write] trying 274 bytes
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [fd 31] write_async(2)=274 (274/274)
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [net_write] ret=274 total=274/274
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [net_write] trying 77 bytes
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [fd 31] write_async(2)=77 (77/77)
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [net_write] ret=77 total=77/77
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [net_read] try up to 4095 bytes
[2024/06/20 21:58:36] [trace] [engine] resuming coroutine=0x7fcf18018fe0
[2024/06/20 21:58:36] [trace] [io coro=0x7fcf18018fe0] [net_read] ret=351
[2024/06/20 21:58:36] [debug] [output:datadog:datadog.0] http://http-intake.logs.datadoghq.com, port=80, HTTP status=202 payload={}
[2024/06/20 21:58:36] [debug] [upstream] KA connection #31 to http-intake.logs.datadoghq.com:80 is now available
[2024/06/20 21:58:36] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/06/20 21:58:36] [debug] [out flush] cb_destroy coro_id=0
[2024/06/20 21:58:36] [trace] [coro] destroy coroutine=0x7fcf18018fe0 data=0x7fcf18018ff8
[2024/06/20 21:58:36] [debug] [task] destroy task=0x7fcf18018ec0 (task_id=0)
[2024/06/20 21:58:37] [trace] [task 0x7fcf1801ac50] created (id=0)
[2024/06/20 21:58:37] [debug] [task] created task=0x7fcf1801ac50 id=0 OK
[2024/06/20 21:58:37] [trace] [input chunk] update output instances with new chunk size diff=59, records=1, input=dummy.0

@jszwedko
Copy link
Contributor Author

Valgrind output:

==78875==
==78875== HEAP SUMMARY:
==78875==     in use at exit: 0 bytes in 0 blocks
==78875==   total heap usage: 4,092 allocs, 4,092 frees, 3,144,616 bytes allocated
==78875==
==78875== All heap blocks were freed -- no leaks are possible
==78875==
==78875== For lists of detected and suppressed errors, rerun with: -s
==78875== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

@jszwedko
Copy link
Contributor Author

I tried to run ./packaging/local-build-all.sh but it fails with:

Unable to find fluent-bit/packaging/distros/{"target":"amazonlinux/Dockerfile

@sudomateo
Copy link
Contributor

Thank you so much for working on this! Please let me know if there's anything I can do to help this along.

@edsiper edsiper merged commit 1c58785 into fluent:master Jun 25, 2024
49 checks passed
@edsiper edsiper added this to the Fluent Bit v3.1.0 milestone Jun 25, 2024
jszwedko added a commit to jszwedko/fluent-bit-docs that referenced this pull request Jul 2, 2024
Added by fluent/fluent-bit#8988

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
@jszwedko jszwedko deleted the jszwedko/add-ddhostname-support branch July 2, 2024 14:35
@jszwedko
Copy link
Contributor Author

jszwedko commented Jul 2, 2024

PR for docs: fluent/fluent-bit-docs#1401

jszwedko added a commit to DataDog/documentation that referenced this pull request Jul 2, 2024
Added in fluent/fluent-bit#8988

Easiest to review with whitespace changes ignored

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
hestonhoffman added a commit to DataDog/documentation that referenced this pull request Jul 9, 2024
* Document new `dd_hostname` field for fluentbit integration

Added in fluent/fluent-bit#8988

Easiest to review with whitespace changes ignored

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>

* Update content/en/integrations/fluentbit.md

---------

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
Co-authored-by: Heston Hoffman <heston.hoffman@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

out_datadog: support configurable hostname
3 participants