[exporter/datadog] Log events are lost in case of network issues without retry

[anmalysh-yb]

Component(s)

exporter/datadog

What happened?

Currently datadog exporter treats network error as permanent error and does not queue log event for retry, because of that.

Probably introduced with this diff: a21144b

Steps to Reproduce

Configure collector as shown in the ticket.
Start writing log.
Turn off network connection.
Turn network connection back on after some time.

Expected Result

All the logs are exported to datadog.

Actual Result

Logs written during network outage are lost.

Collector version

0.81.0

Environment information

Environment

OS: MacOS
Compiler(if manually compiled): N/A

OpenTelemetry Collector configuration

receivers:
   filelog:
    include: [ /Users/amalysh86/otelcol/simple.log*]
    start_at: beginning
    storage: file_storage/pts
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'
        severity:
          parse_from: attributes.sev

exporters:
  file:
    path: /Users/amalysh86/otelcol/out.json
  datadog:
    api:
      site: us3.datadoghq.com
      key: <API KEY HERE>
    retry_on_failure:
      enabled: true
      initial_interval: 1m
      max_interval: 1800m
    sending_queue:
      storage: file_storage/psq

extensions:
  file_storage/pts:
     directory: /Users/amalysh86/otelcol/pts
     timeout: 10s # in what time a file lock should be obtained

  file_storage/psq:
     directory: /Users/amalysh86/otelcol/psq
     timeout: 10s # in what time a file lock should be obtained
     compaction:
       directory: /Users/amalysh86/otelcol/psq
       on_start: true
       on_rebound: true
       rebound_needed_threshold_mib: 5
       rebound_trigger_threshold_mib: 3

service:
  extensions:
  - file_storage/pts
  - file_storage/psq
  pipelines:
    logs:
      receivers: [filelog]
      processors: []
      exporters: [datadog]

Log output

2023-06-30T19:05:28.650+0300	error	exporterhelper/queued_retry.go:391	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "logs", "name": "datadog", "error": "Permanent error: Post \"https://http-intake.logs.us3.datadoghq.com/api/v2/logs?ddtags=otel_source%3Adatadog_exporter\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 1}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr
• needs: Github issue template generation code needs this to generate the corresponding labels.

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @anmalysh-yb, it looks like this was intended from the change here: #16390. I'll defer to the code owners to confirm if this was intended or not.

Note: Looks like another user is running into the same issue, maybe this is an unintended consequence.

[mx-psi]

@dineshg13 can this be closed or is there work left to do?

[anmalysh-yb]

@mx-psi I think this is a real issue actually and should be fixed.
This issue basically prevents reliable enough logs delivery to DD.
In case network becomes unavailable - we basically start dropping all the log events instead of retrying them.

[siarhei-kharchanka-cko]

Hey @mx-psi @anmalysh-yb as users of Datadog Exporter, we are also facing log records lost because of blips in a network. I believe this is a regression issue and needs to be fixed. How can I help move this issue further?

[songy23]

👋 #27450 is expected to fix the issue.

[siarhei-kharchanka-cko]

Hey @songy23 thanks for the update. The referenced PR is quite a heavy refactoring. How feasible do you think we could fix this issue within the current approach of logs pushing without waiting for the refactoring will be released? I assume it might be a fairly quick/light fix (I am happy to create a PR). Or perhaps you could suggest a potential workaround to prevent potential data loss while waiting for the refactoring to be in the main?

[songy23]

@siarhei-kharchanka-cko It would be awesome if you can send a quick fix!

Context: this issue is only with HTTP log client that datadog log exporter is currently using. #27450 migrates the datadog log exporter from HTTP log client to logs agent. There is built-in retries in logs agent.

[siarhei-kharchanka-cko]

Thank you @songy23! I've linked PR