Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/datadog] invalid api token causes failure when using logs agent exporter #33195

Closed
csquire opened this issue May 22, 2024 · 1 comment · Fixed by #33504
Closed

[exporter/datadog] invalid api token causes failure when using logs agent exporter #33195

csquire opened this issue May 22, 2024 · 1 comment · Fixed by #33504
Labels
bug Something isn't working data:logs Logs related issues exporter/datadog Datadog components needs triage New item requiring triage

Comments

@csquire
Copy link

csquire commented May 22, 2024

Component(s)

exporter/datadog

What happened?

Description

When the feature gate enabled for the Datadog logs agent exporter, an invalid Datadog API key will cause the collector to exit.

Requires this feature gate to be enabled:
#32327

This is the summary of what I believe is happening:

  1. The collector defaults the logging endpoint address to https://http-intake.logs.datadoghq.com/
  2. The Datadog Agent code used by the collector is performing a check to see if HTTP connectivity works before deciding on HTTP or TCP
  3. If HTTP works, TCP is not attempted
  4. If HTTP fails (including if the api key is invalid), the Datadog Agent code attempts TCP
    This is where the error occurs. When the API key is invalid, the HTTP connection validation fails and TCP is attempted. Parsing the default endpoint of https://http-intake.logs.datadoghq.com/ is attempted, but the code expects a <host>:<port> format and errors due to port not being a number.

Steps to Reproduce

Run collector with feature gate enabled:

RUN_ARGS=--feature-gates=exporter.datadogexporter.UseLogsAgentExporter make run

Expected Result

The collector continues to run, even if log forwarding is not possible

Actual Result

The collector crashes with the included log error

Collector version

v0.100.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog:
    start_at: beginning
    include:
      - ../../local/app.log

exporters:
  datadog:
    api:
      key: xyz
      
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [datadog]

Log output

Error: failed to build pipelines: failed to create "datadog" exporter for data type "logs": failed to create logs agent: Invalid endpoints: could not parse https://http-intake.logs.datadoghq.com: strconv.Atoi: parsing "//http-intake.logs.datadoghq.com": invalid syntax
2024/05/22 14:33:20 collector server run finished with error: failed to build pipelines: failed to create "datadog" exporter for data type "logs": failed to create logs agent: Invalid endpoints: could not parse https://http-intake.logs.datadoghq.com: strconv.Atoi: parsing "//http-intake.logs.datadoghq.com": invalid syntax

Additional context

No response

@csquire csquire added bug Something isn't working needs triage New item requiring triage labels May 22, 2024
@github-actions github-actions bot added the exporter/datadog Datadog components label May 22, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@mx-psi mx-psi added the data:logs Logs related issues label May 23, 2024
mx-psi pushed a commit that referenced this issue Jun 12, 2024
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

Add default config option to always use http for logs agent exporter.
Fixes bug where logs agent exporter was falling back to using tcp,
causing collector shut down.

Also opened a PR in datadog-agent for adding a warn log when the logs
agent connectivity check fails:
DataDog/datadog-agent#26626

**Link to tracking Issue:** <Issue number if applicable>
Fixes
#33195

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working data:logs Logs related issues exporter/datadog Datadog components needs triage New item requiring triage
Projects
None yet
2 participants