Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The collector was OOMKilled when receiving massive events #147

Closed
dxsup opened this issue Mar 25, 2022 · 1 comment
Closed

The collector was OOMKilled when receiving massive events #147

dxsup opened this issue Mar 25, 2022 · 1 comment
Labels
area/collector Issues or PRs related to agent metric collector bug Something isn't working

Comments

@dxsup
Copy link
Member

dxsup commented Mar 25, 2022

As we all know, the collector consumes events from the probe via the UNIX socket domain using ZeroMQ. I have found that if the probe sends too many events and the collector is unable to keep up with the rate of incoming messages, the collector will use more and more memory until it is OOMKilled. And even if I stopped the test load on this node which means the collector was able to keep up with the events, the memory usage would not be deallocated. I know there is an option ZMQ_HWM which should have limited the memory allocated, but after I set this option, the problem was still there.

This seems like what ZeroMQ expects to do, so we should adapt the usage of ZeroMQ to prevent this happening. See more at zeromq/libzmq#4218

@dxsup dxsup added bug Something isn't working area/collector Issues or PRs related to agent metric collector labels Mar 25, 2022
@dxsup
Copy link
Member Author

dxsup commented Jun 27, 2022

This issue is invalid after merging #264 because ZeroMQ has been removed.

@dxsup dxsup closed this as completed Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/collector Issues or PRs related to agent metric collector bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant