Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory leak when tracing is enabled #3432

Merged
merged 3 commits into from
Jun 18, 2024
Merged

Conversation

jamescrosswell
Copy link
Collaborator

@jamescrosswell jamescrosswell commented Jun 18, 2024

Resolves #3431

Memory Profile prior to this PR

image

Memory Profile after this PR

image

@jamescrosswell
Copy link
Collaborator Author

In a follow up PR we might want to replace ConcurrentBag<T> with a hand crafted ConcurrentBagLite<T> or ConcurrentCollection<T>... as I don't think ConcurrentBag<T> is the right collection to be used here... see this stackoverflow thread. I think the bag is appropriate when you plan to take out all the items you put into it... so kind of like an any-in-any-out data structure.

We're using ConcurrentBag<T> in quite a few other places as well... and I suspect we don't need to - a much simpler data structure could serve our purposes.

Copy link
Contributor

@bitsandfoxes bitsandfoxes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems like quite the gotcha! Thanks for fixing that!
I'm not sure I understand where that leak is coming from tho. The fix implies that the SDK doesn't get rid of the TransactionTracer?

Comment on lines +389 to +390
// Release tracked spans
ReleaseSpans();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why this is required? Is something holding the reference to the TransactionTracer? Does that reference need to be released too?

Copy link
Collaborator Author

@jamescrosswell jamescrosswell Jun 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I fully understand yet, no. The stack overflow thread linked above seems to imply ConcurrentBag<T> stores stuff in a thread local and leaves it there, unless you remove it. It seems improbable that we wouldn't have picked up on something so obvious years ago if that was the case though. That would mean massive memory leaks all over our SDK (we use ConcurrentBag<T> in a number of places).

I'll merge this PR, create a release, and then do a bit more digging on potentially replacing ConcurrentBag<T> with a light weight custom type that isn't so mysterious. That will require some benchmarking and testing though, as it would be replacing something quite foundational.

@jamescrosswell jamescrosswell merged commit 7c2a91f into main Jun 18, 2024
21 checks passed
@jamescrosswell jamescrosswell deleted the fix/memory_3431 branch June 18, 2024 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory issue/leak with Tracing enabled
2 participants