[exporter/datadog] make error retryable if logs sender received nil response

[siarhei-kharchanka-cko]

Description:

The Datadog exporter threats network/connectivity errors (HTTP client doesn't receive a response) as permanent errors, which can lead to log records loss. This change makes these errors retryable.

Link to tracking Issue: #24550

Testing:

Documentation:

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: siarhei-kharchanka-cko / name: Siarhei Kharchanka (2a83b6a)

[songy23]

At a high level I agree with this change, though AFAICT this can be a problem in 2 corner cases:

1. May lead to duplicated logs being sent if the initial request went through but response is lost
2. May increase the wait time if there's indeed a permanent error when exporter received nil response

WDYT? @dineshg13 @mx-psi

[siarhei-kharchanka-cko]

At a high level I agree with this change, though AFAICT this can be a problem in 2 corner cases:

1. May lead to duplicated logs being sent if the initial request went through but response is lost
2. May increase the wait time if there's indeed a permanent error when exporter received nil response

Just a quick note from my side. As we briefly discussed the migration to the logs agent should fix the issue as well - So I have taken a look on how it works in the logs agent. According to this source code the logs agent will retry a request on the error despite the fact empty/non-empty response. The only difference is that the logs agent source code also checks that the error is not a cancellation error triggered by the client (we can add this check as well). It looks like the described 2 cases can happen with the logs agent as well.

[songy23]

Thanks for checking, LGTM then