datadog-agent stop sending metrics

[k-cloudit]

I'm using mimir proxies main branch to send metrics from datadog-agent to mimir.
This this how datadog is configured :

additional_endpoints:
  "http://IP:8009/datadog":
  - grafana-labs

it works, mimir is receiving metrics but after few minutes I have this message in datadog-agent :

2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/check_run": "can't translate series: client_error: client error: 404\n", dropping it
2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/series": "can't translate series: client_error: client error: 404\n", dropping it

and this log message on mimir-proxies :

level=info ts=2023-09-07T11:34:51.057594497Z elapsed=2.481926ms traceID=71652a924ff0db99 sampled=false orgID=fake method=POST uri=/datadog/api/v1/series status=400
level=warn ts=2023-09-07T11:35:06.054259542Z traceID=5e64c8ef0da08f03 request_uri=/datadog/api/v1/check_run method=POST msg="can't translate series" response_code=400 err="client_error: client error: 404"

I attached a grafana dashboard show how the metrics are ingested, no mimir-proxies or datadog-agent have been restarted.

is there any solution to this behavior ?

[alograg]

WIP https://github.com/intmgroupe/mimir-proxies/tree/issue-74

[alograg]

@k-cloudit confirm that the problem no longer occurs after making the following updates:

• httpsnoop v1.0.3 -> httpsnoop v1.0.4
• mux v1.8.0 -> mux v1.8.1
• clientgolang v1.16.0 -> clientgolang v1.17.0
• common v0.44.0 -> common v0.45.0
• otel v1.14.0 -> otel v1.19.0
• opentracing v1.14.0 opentracing v1.19.0
• sdk v1.14.0 -> sdk v1.19.0
• trace v1.14.0 -> trace v1.19.0
• net v0.15.0 -> net v0.19.0
• golangprotobufextensions v1.0 -> golangprotobufextensions v2
• clientmodel v0.4.0 -> clientmodel v0.4.1-0.202307181
• procfs v0.10.1 -> procfs v0.11.1
• metric v0.37.0 -> metric v1.19.0
• crypto v0.13.0 -> crypto v0.16.0
• oauth2 v0.8.0 -> oauth2 v0.12.0
• sync v0.2.0 -> sync v0.3.0
• sys v0.12.0 -> sys v0.15.0
• text v0.13.0 -> text v0.14.0
• protobuf v1.30.0 -> protobuf v1.31.0