-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cadvisor daemonset with containerd goes into crashloopbackoff #2855
Comments
You might have encountered the same issue as I did. I faced the same behavior and in my case it was because cadvisor for containerd gets all the env vars of all the containers resulting in a ton of memory usage as they get exported as metric labels. You can add this flags to your cadvisor container in the daemonset, if it solves your issue then is probably the same as mine:
These flags are just a workaround of the issue. For docker, there is an explicit whitelisting for env vars which is not yet implemented for containerd (I made a PR for it to behave as it does with docker #2857). |
Thanks @skgsergio Let me check that |
Thanks @skgsergio Adding those parameters worked for me on the image gcr.io/cadvisor/cadvisor:v0.37.5 |
If it worked then 100% is the same issue where cadvisor is not filtering env vars for containerd containers. Regarding the image, is there any reason you want to use the |
@MonicaMagoniCom can you provide following information:
I agree that close to 700 MiB looks worrying. CC: @Creatone @dashpole - it seems to be related to the increase discussed in #2853. |
You find all the information here #2856 |
Looks like @MonicaMagoniCom is using GKE version 1.14.10-gke.42, and as far as I remember is not until 1.19 when GKE started shipping Kubernetes with containerd, so it might not be related to this specific issue. |
I have deployed the cadvisor daemonset in my kubernetes cluster. My runtime is containerd. I constantly see that my pods are going into crashloopbackoff. I tried with version v0.30.2, v0.37.5 and latest. kubectl logs doesnt show anything. The pods say "OOMKilled". please advise what is being missed.
NAME READY STATUS RESTARTS AGE
cadvisor-4wl5l 1/1 Running 0 96s
cadvisor-95gqg 1/1 Running 0 96s
cadvisor-pfwpk 1/1 Running 0 98s
cadvisor-xgrp8 1/1 Running 0 99s
cadvisor-4wl5l 0/1 OOMKilled 0 2m
cadvisor-4wl5l 1/1 Running 1 2m1s
cadvisor-xgrp8 0/1 OOMKilled 0 2m9s
cadvisor-xgrp8 1/1 Running 1 2m10s
cadvisor-95gqg 0/1 OOMKilled 0 2m16s
cadvisor-pfwpk 0/1 OOMKilled 0 2m18s
cadvisor-95gqg 1/1 Running 1 2m17s
cadvisor-pfwpk 1/1 Running 1 2m19
The text was updated successfully, but these errors were encountered: