-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leaks in nsmgr #675
Comments
I tried the instructions that @ljkiraly attached above but added scaling of NSEs too. From my test it seems that by creating NSEs NSMgr exhausts more memory than with scaling NSCs. |
Description updated with version information. Also might be important that I was tested on a kind cluster with 4 nodes. |
Another important detail, that I tested with nsmgr pod without exclude prefixes container and the same behavior can be seen, the memory increase still present. Edited the issue slogan. |
Hello! I think I managed to reproduce the leak. I tried several setups: NSEs with CIDR
|
Here I add three heap profiles I created during my tests. The first one is from an idle state, then one after scaling of NSCs and NSEs started and another one from a later phase of scaling. |
Hi @denis-tingaikin, I created an nsmgr image based on the following commits: ccf42a564dce826dd7e0b5647393c70037643447 - fix memory leaks
Asked to test it in a customer like environment with more then 80 endpoints and traffic running. Still there was a memory increase, especially in one of the nsmgr container:
Find the collected memprofiles attached. As you can see at the slice from May7 2:38am (CEST), the profiling tool shows that the memory used by nsmgr was 38428.29kB (~40MB). It is strange that kubelet's metrics server showing a higher RSS that time (around 90MB-100MB).
Hope that helps. |
Increased memory usage of nsmgr pods can be detected.
On some environment the memory usage reaches it's limit (400M) and leads to OOMKill.
Reproduction:
With the following example I was able to reproduce a considerable memory increase by scaling out and scaling in the NSCs.
https://github.com/ljkiraly/deployments-k8s/blob/1ac366dcb8cbffe52cf5428fe9bb6aa79a9511bd/examples/use-cases/2registry3nseXnsc/README.md
The test ran on kind cluster with 4 nodes and NSM v1.13.0 was deployed.
Sample output of
kubectl top
commandThe text was updated successfully, but these errors were encountered: