-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cAdvisor crashed due to OOM #2856
Comments
How many containers per node are you running? |
I guess that #2840 might be related. Looks like we may need to bisect 0.36 and 0.37. |
We have just one container per each node |
Why does it seem to be related? I'm running 0.39 |
I add these flags:
and as you see in the attached image, there is no more a sudden increase of the memory, but still the memory used is high and there are restarts due to OOM. The memory increase is linear on the nodes with less resources (which means less load), but it is critical on bigger nodes. The smallest one: |
We are experiencing a similar behaviour on our GKE cluster. Curiously it only happens on nodes that have Containerd as runtime (Container-Optimised OS with Containerd (cos_containerd)). Cadvisor image: latest EDIT: |
For me, the error was because of the flag |
We have deployed cadvisor v0.39.0 as daemonset in our Kubernetes cluster where nodes have version 1.14.10-gke.42.
Even if we have disabled many metrics, cadvisor instances continue experiencing OOM.
Here is our configuration:
image: gcr.io/cadvisor/cadvisor:v0.39.0
resources:
limits:
cpu: 2500m
memory: 700Mi
requests:
cpu: 100m
memory: 200Mi
As you can see in the attached image, the memory has suddenly a pick (without any particular reason) and then it crashes (since the memory limit is 700Mi).
The text was updated successfully, but these errors were encountered: