Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

notifications: Investigate possible memory leak with pending outbound substream tracking #295

Closed
lexnv opened this issue Dec 2, 2024 · 0 comments · Fixed by #296
Closed
Labels
bug Something isn't working

Comments

@lexnv
Copy link
Collaborator

lexnv commented Dec 2, 2024

The metrics implementation exposed an abnormal behaviour in our state tracking:

Screenshot 2024-12-02 at 11 17 17

It looks like the following data structure grows over time:

/// Pending outboudn substreams.
pending_outbound: HashMap<SubstreamId, PeerId>,

Elements are added to the hashmap on:

Elements are removed from the hashmap on:

Possible causes

It looks like we should remove the state on the connection closed as well:

OutboundState::OutboundInitiated { .. }

@lexnv lexnv added the bug Something isn't working label Dec 2, 2024
@lexnv lexnv closed this as completed in #296 Dec 3, 2024
lexnv added a commit that referenced this issue Dec 3, 2024
This PR fixes a subtle memory leak that can happen in the following
edge-case situation:
- connection is established and substream outbound is initiated with
remote peer
- the substream ID is tracked until the substream either completes
successfully or fails
- the connection is closed soon after, leading to no substream events
ever being generated

For this edge-cases, we need to remove the tracking of the substream ID
when the connection is reported as closed.

This has been detected after running a node for more than 2 days with
the following generic metrics PR:
- #294

Closes: #295

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant