OOM caused by span metrics connector

[alburthoffman]

Component(s)

connector/spanmetrics

Describe the issue you're reporting

We tried to switch from span metrics processor to span metrics connector, and found OOM issue.

Below is the pod heap memory after using span metrics connector. The pod traffic is around 20K spans per second:

before this, span metrics processor is quite stable.

the profile shows the pmap takes lots of memory for span metrics connector.

which is in createAtrributes.

[github-actions]

Pinging code owners:

• connector/spanmetrics: @albertteoh @kovrus

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[alburthoffman]

@albertteoh @kovrus please help

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @albertteoh @kovrus

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[albertteoh]

@alburthoffman apologies for the slow response.

I suggest trying the following:

• Inspect the metrics sent to spanmetrics connector to see if there are any high cardinality metrics.
• Reduce the dimensions_cache_size
• It looks like the explicit histogram metrics are taking up most of the memory. Try reducing the number of dimensions buckets.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @albertteoh

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[vjsamuel]

@alburthoffman please try these and get back to @albertteoh

[alburthoffman]

@albertteoh the old span processor can handle the same config without OOM issue. So it should not be a config issue.

[albertteoh]

@albertteoh the old span processor can handle the same config without OOM issue. So it should not be a config issue.

Thanks for confirming, @alburthoffman. Is this something that's can be reproduced locally by any chance?

[aishyandapalli]

@albertteoh I have done a quick round of testing for this one. When we enable histogram metrics in spanmetrics connector, the memory usage is very high. I tried disabling the histogram metrics and memory usage was very low. Reducing the number of buckets didn't help much. Memory usage reduced a bit but didn't help much.

Can you please help identify the root cause?

[alburthoffman]

we did a new round of testing, and still see this issue

[portertech]

@alburthoffman does this issue persist with the latest release?

Related, I wonder if further reducing/aggregating on fewer resource attributes would help, see: #29711

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @portertech

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[aishyandapalli]

This issue is resolved. We have added a config to limit the number of exemplars to be added for sum metrics in spanmetrics connector. It's merged and deployed. @alburthoffman We can close this issue

[alburthoffman]

Thx @aishyandapalli