"Count" metrics being multiplied by 10

[tmargheim]

We're using import { StatsD } from 'hot-shots' to send count metrics to Datadog, from both an ECS service (API) and a Kinesis-triggered lambda. Both are built from the same NestJS project, using the same metrics package & general codebase.

Strangely, on Nov 28 2024, around 11:30am EST, we deployed an unrelated code change... and since then, the custom metrics from the lambda are being multiplied by 10. The similar metrics from the ECS service are unaffected.

I'm wondering, has anyone else seen this bug?

Note: I did find a similar "multiplied by 10" bug reported with a metrics agent package, and it had to do with "count" metrics actually being handled as "rate" metrics under the hood, and a mismatch between assumed intervals. But I'm stumped on why this would have started "spontaneously", and also only be affecting metrics from the lambda!

[bdeitte]

I haven't heard of this come up in this project before. My guess would be unrelated package updates coming in with the code change, but note at least from hot-shots this wouldn't affect anything, as no extra packages are used for anything other than the uds protocol.

On an unrelated note, this issue coming in made me do a double-take, as I work with @t-margheim