Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error "Metrics not available for pod" while checking the metrics of Pods using "metrics-server" add-on #13969

Closed
meliwex opened this issue Apr 17, 2022 · 28 comments
Labels
kind/support Categorizes issue or PR as a support question. long-term-support Long-term support issues that can't be fixed in code triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@meliwex
Copy link

meliwex commented Apr 17, 2022

What Happened?

After enabling metrics-server add-on using minikube addons enable metrics-server command I tried to check the metrics of the Pods in kube-system namespace or just of the Pods that I created manually in default namespace.

This is what I got:

meliwex@vmubuntu:~$ kubectl top pods 
error: Metrics not available for pod default/dpl-test-6fb55dc999-lmxfm, age: 2m24.964714409s
meliwex@vmubuntu:~$ kubectl top pods -n kube-system
error: Metrics not available for pod kube-system/coredns-64897985d-9rksw, age: 25m27.304057516s

The metrics-server is running:

metrics-server-6b76bd68b6-6tk7p    1/1     Running

My environment:
Output of minikube version:
minikube version: v1.25.2

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:19:12Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}

I am using VirtualBox Virtual Machine with OS Ubuntu Server 20.04 LTS.

Notes:
I got also the same error in AWS EC2 instance with the same OS.
Using kind I didn't have any problem with metrics-server, everything worked well. I guess the problem is related to minikube.

Attach the log file

log.txt

Output of: kubectl logs metrics-server-6b76bd68b6-6tk7p -n kube-system

I0417 07:03:39.667629       1 serving.go:325] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0417 07:03:42.246802       1 secure_serving.go:197] Serving securely on [::]:4443
I0417 07:03:42.246848       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I0417 07:03:42.246870       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0417 07:03:42.247030       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0417 07:03:42.247090       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0417 07:03:42.247154       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0417 07:03:42.247203       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0417 07:03:42.247251       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0417 07:03:42.247300       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0417 07:03:42.347306       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0417 07:03:42.347382       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0417 07:03:42.348176       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 

Operating System

Ubuntu

Driver

Docker

@spowelljr
Copy link
Member

Hi @meliwex, thanks for reporting your issue with minikube!

This is a known issue which has since been fixed but hasn't been released yet.

Try deleting your instances minikube delete and then add --extra-config=kubelet.housekeeping-interval=10s to your start args and that should enable metrics reporting again.

@spowelljr spowelljr added kind/support Categorizes issue or PR as a support question. triage/needs-information Indicates an issue needs more information in order to work on it. labels Apr 18, 2022
@oprudkyi
Copy link
Contributor

Try deleting your instances minikube delete
it seems just stop/start is enough

@avizen-j
Copy link

Thanks @spowelljr , minikube start --extra-config=kubelet.housekeeping-interval=10s resolved the issue for me.

@meliwex
Copy link
Author

meliwex commented Apr 23, 2022

Hi @oprudkyi, for me simply deleting the cluster and starting the cluster again doesn't solve the issue.

meliwex@vmubuntu:~$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/meliwex/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
meliwex@vmubuntu:~$ minikube start --driver=docker
😄  minikube v1.25.2 on Ubuntu 20.04 (vbox/amd64)
✨  Using the docker driver based on user configuration

🧯  The requested memory allocation of 1983MiB does not leave room for system overhead (total system memory: 1983MiB). You may face stability issues.
💡  Suggestion: Start minikube with less memory allocated: 'minikube start --memory=1983mb'

👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=1983MB) ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
❗  Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 19.005758219s
💡  Restarting the docker service may improve performance.
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
meliwex@vmubuntu:~$ minikube addons enable metrics-server
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
🌟  The 'metrics-server' addon is enabled
meliwex@vmubuntu:~$ kubectl top pods -n kube-system
error: Metrics not available for pod kube-system/coredns-64897985d-fn458, age: 3m50.90840427s
meliwex@vmubuntu:~$ 

@oprudkyi
Copy link
Contributor

@meliwex sorry, I meant after stopping to start it like

minikube start --extra-config=kubelet.housekeeping-interval=10s ...

also, please note it increase cpu load and result in some unstable work, on i5 some heavy pods started to restart and freeze etc

@meliwex
Copy link
Author

meliwex commented Apr 23, 2022

@spowelljr, @oprudkyi

This some kind solved the issue but not completely.
I just created deployment in the default namespace and after successfully checking the metrics of the pods from the default namespace I cannot get the metrics again after some time. I don't have such a problem with kube-system namespace.

The output I truncated a little bit for simplicity:

meliwex@vmubuntu:~$ kubectl create deployment dpl-test --image=nginx
meliwex@vmubuntu:~$ kubectl get pods -n default
NAME                        READY   STATUS    RESTARTS   AGE
dpl-test-86c4679758-qrmwg   1/1     Running   0          112s         
meliwex@vmubuntu:~$ kubectl top pods -n default
NAME                        CPU(cores)   MEMORY(bytes)   
dpl-test-86c4679758-qrmwg   6m           3Mi             
meliwex@vmubuntu:~$ kubectl top pods -n default
error: Metrics not available for pod default/dpl-test-86c4679758-qrmwg, age: 2m26.722936865s
meliwex@vmubuntu:~$ kubectl top pods -n kube-system
NAME                               CPU(cores)   MEMORY(bytes)   
coredns-64897985d-q2fzg            4m           23Mi            
etcd-minikube                      33m          40Mi            
kube-apiserver-minikube            95m          277Mi           
kube-controller-manager-minikube   29m          42Mi            
kube-proxy-8rs94                   1m           26Mi            
kube-scheduler-minikube            5m           30Mi            
metrics-server-6b76bd68b6-5jwkf    6m           12Mi            
storage-provisioner                3m           15Mi  

Is there any solution?

@karinagrullon
Copy link

Hi @meliwex, thanks for reporting your issue with minikube!

This is a known issue which has since been fixed but hasn't been released yet.

Try deleting your instances minikube delete and then add --extra-config=kubelet.housekeeping-interval=10s to your start args and that should enable metrics reporting again.

This worked perfectly. Thank you!!

@meliwex
Copy link
Author

meliwex commented May 1, 2022

Hi @karinagrullon and @avizen-j, for your own created Pods, does it display the metrics?

@avizen-j
Copy link

avizen-j commented May 1, 2022

Yes, now, I am able to see both CPU and memory usage metrics in my own created namespace. If I remember correctly before that I had something like "unknown"
image

@karinagrullon
Copy link

Hi @meliwex, when I ran this command:

kubectl -n kube-system top pods

I got an error, but no anymore.

Also, after applying my hpa.yaml file I got TARGET <unknown>, but it has been fixed as well.

I disabled minikube metrics-server and downloaded it + installed it from GitHub instead.

@meliwex
Copy link
Author

meliwex commented May 3, 2022

@avizen-j, @karinagrullon
After executing minikube start --extra-config=kubelet.housekeeping-interval=10s, the metrics of the Pods in kube-system namespace started to be shown(anytime), but whenever I want to check the metrics of my own created Pods, the metrics are unstable like:

meliwex@hetzner:~$ kubectl top pods 
NAME    CPU(cores)   MEMORY(bytes)   
nginx   0m           6Mi             
meliwex@hetzner:~$ kubectl top pods
error: Metrics not available for pod default/nginx, age: 2m57.605095433s
meliwex@hetzner:~$ kubectl top pods
error: Metrics not available for pod default/nginx, age: 3m0.289038232s
meliwex@hetzner:~$

Same error I get when I use minikube metrics-server add-on and just official manifest kubernetes-sigs/metrics-server

Do you know what can cause this?

@karinagrullon
Copy link

@meliwex
That's the same error I used to get before starting minikube start --extra-config=kubelet.housekeeping-interval=10s, but I am not getting it anymore after applying it, even with my own pods its working fine for me.

Do you disable metrics-server add-on when applying the official manifest?

Also, do you do these fixes on the official Kubernetes manifest?

This is how the official metrics-server.yaml file looks for me (Deployment.spec):

spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-insecure-tls
        - --kubelet-use-node-status-port
        - --metric-resolution=15s

@meliwex
Copy link
Author

meliwex commented May 4, 2022

@karinagrullon

Do you disable metrics-server add-on when applying the official manifest? Yes
Also, do you do these fixes on the official Kubernetes manifest? Before I didn't do it, but after applying it I get the same error

I noticed a new thing that the metrics are shown for some of the Pods but not for all. I guess it depends on the image.

meliwex@hetzner:~$ kubectl get pods
No resources found in default namespace.
meliwex@hetzner:~$ kubectl create deployment nginx-dpl --image=nginx:latest
deployment.apps/nginx-dpl created
meliwex@hetzner:~$ kubectl create deployment wordpress-dpl --image=wordpress:latest
deployment.apps/wordpress-dpl created
meliwex@hetzner:~$ kubectl create deployment httpd-dpl --image=httpd:latest
deployment.apps/httpd-dpl created
meliwex@hetzner:~$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
httpd-dpl-5b7f5d99bd-qjvls       1/1     Running   0          2m17s
nginx-dpl-f4bff5b4-4g88b         1/1     Running   0          3m56s
wordpress-dpl-869dd745bc-hl8wf   1/1     Running   0          3m30s
meliwex@hetzner:~$ kubectl top pods 
NAME                             CPU(cores)   MEMORY(bytes)   
httpd-dpl-5b7f5d99bd-qjvls       1m           7Mi             
wordpress-dpl-869dd745bc-hl8wf   1m           22Mi            
meliwex@hetzner:~$ 

@karinagrullon
Copy link

@meliwex

I noticed a new thing that the metrics are shown for some of the Pods but not for all. I guess it depends on the image.

Oh wow, that's interesting. Yes, It seems to be related to the image in this case. Maybe try to play around about the way the image you are trying to extract metrics from is built.

Wish I knew what is really going on here...

@spowelljr spowelljr added the long-term-support Long-term support issues that can't be fixed in code label May 4, 2022
@meliwex
Copy link
Author

meliwex commented May 8, 2022

@karinagrullon
I guess it is related to minikube, because I don't have such a problem with kind

@karinagrullon
Copy link

@meliwex
That makes a lot of sense, considering spowelljr suggested fix.
It is nice you've found something that works for you!

@Al4DIN
Copy link

Al4DIN commented May 10, 2022

I had the same issue but It was solved with --extra-config=kubelet.housekeeping-interval=10s , but I just want to understand the utility and the definition of this argument.

@ghost
Copy link

ghost commented May 11, 2022

for my case i don't need to delete the minikube, just doing stop minikube and then start with minikube start --extra-config=kubelet.housekeeping-interval=10s work normal for me, after a while i can see both from the command line and the dashboard

@meliwex
Copy link
Author

meliwex commented May 11, 2022

Does anyone have a problem with nginx image?

minikube start --extra-config=kubelet.housekeeping-interval=10s doesn't work when it comes to nginx image, but for other images, the metrics are shown correctly and they are stable. The problem is that the metrics are shown for a very short duration but then they aren't available.

meliwex@hetzner:~$ kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created
meliwex@hetzner:~$ kubectl top pods
NAME                     CPU(cores)   MEMORY(bytes)   
nginx-7c658794b9-6sn99   6m           6Mi                            
meliwex@hetzner:~$ kubectl top pods
error: metrics not available yet
meliwex@hetzner:~$ kubectl top pods
error: metrics not available yet
meliwex@hetzner:~$ kubectl top pods
error: Metrics not available for pod default/nginx-7c658794b9-6sn99, age: 3m0.974166634s
meliwex@hetzner:~$ 

I use official metrics-server manifest with these arguments:

      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls

I also did some research and tried to add other arguments but they don't help.
Does anyone face this issue or maybe have a solution?

@meliwex
Copy link
Author

meliwex commented May 12, 2022

Does anyone have a problem with nginx image?

minikube start --extra-config=kubelet.housekeeping-interval=10s doesn't work when it comes to nginx image, but for other images, the metrics are shown correctly and they are stable. The problem is that the metrics are shown for a very short duration but then they aren't available.

meliwex@hetzner:~$ kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created
meliwex@hetzner:~$ kubectl top pods
NAME                     CPU(cores)   MEMORY(bytes)   
nginx-7c658794b9-6sn99   6m           6Mi                            
meliwex@hetzner:~$ kubectl top pods
error: metrics not available yet
meliwex@hetzner:~$ kubectl top pods
error: metrics not available yet
meliwex@hetzner:~$ kubectl top pods
error: Metrics not available for pod default/nginx-7c658794b9-6sn99, age: 3m0.974166634s
meliwex@hetzner:~$ 

I use official metrics-server manifest with these arguments:

      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls

I also did some research and tried to add other arguments but they don't help. Does anyone face this issue or maybe have a solution?

After some investigation, I solved this, but the solution is not perfect. I started using none driver to start a cluster and that solved the problem of the metrics for nginx image:

I truncated little bit the output for simplicity:

meliwex@hetzner:~$ minikube start --extra-config=kubelet.housekeeping-interval=10s --driver=none
😄  minikube v1.25.2 on Ubuntu 20.04 (amd64)
✨  Using the none driver based on user configuration

👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=2, Memory=1935MB, Disk=38146MB) ...
ℹ️  OS release is Ubuntu 20.04.4 LTS

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!/

🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
meliwex@hetzner:~$ minikube addons enable metrics-server
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
🌟  The 'metrics-server' addon is enabled
meliwex@hetzner:~$ kubectl create deployment nginx --image=nginx:latest
deployment.apps/nginx created
meliwex@hetzner:~$ kubectl expose deployment nginx --type=NodePort --port=80 --target-port=80
meliwex@hetzner:~$ kubectl top pods
NAME                     CPU(cores)   MEMORY(bytes)   
nginx-7c658794b9-k2gzl   0m           6Mi             
meliwex@hetzner:~$ kubectl top pods
NAME                     CPU(cores)   MEMORY(bytes)   
nginx-7c658794b9-k2gzl   3m           6Mi                      
meliwex@hetzner:~$ kubectl top pods
NAME                     CPU(cores)   MEMORY(bytes)   
nginx-7c658794b9-k2gzl   28m          6Mi             
meliwex@hetzner:~$ 

Notes:
--extra-config=kubelet.housekeeping-interval=10s option is required for this to work
This solution works with metrics-server add-on and with an official manifest

I don't want to close the issue, because maybe someone can find the solution for docker driver.

@Al4DIN
Copy link

Al4DIN commented May 12, 2022

@meliwex can you show your pod logs or just do kubectl describe pods/nginx-7c658794b9-6sn99

@meliwex
Copy link
Author

meliwex commented May 12, 2022

@Al4DIN
This is the output of kubectl describe pods/nginx-7c658794b9-rz2dv, but note that I already deleted the old cluster and created a new one with the same configurations, that's why the pod's name is different:

Name:         nginx-7c658794b9-rz2dv
Namespace:    default
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Thu, 12 May 2022 08:50:25 +0000
Labels:       app=nginx
              pod-template-hash=7c658794b9
Annotations:  <none>
Status:       Running
IP:           172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  ReplicaSet/nginx-7c658794b9
Containers:
  nginx:
    Container ID:   docker://d44131c615142f2162731e610dd5bc08864d3fb1b1cc3c818e6d652d17bbfc83
    Image:          nginx:latest
    Image ID:       docker-pullable://nginx@sha256:19da26bd6ef0468ac8ef5c03f01ce1569a4dbfb82d4d7b7ffbd7aed16ad3eb46
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 12 May 2022 08:50:33 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2czcv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-2czcv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m38s  default-scheduler  Successfully assigned default/nginx-7c658794b9-rz2dv to minikube
  Normal  Pulling    5m37s  kubelet            Pulling image "nginx:latest"
  Normal  Pulled     5m30s  kubelet            Successfully pulled image "nginx:latest" in 7.083140083s
  Normal  Created    5m30s  kubelet            Created container nginx
  Normal  Started    5m30s  kubelet            Started container nginx

@Al4DIN
Copy link

Al4DIN commented May 13, 2022

@meliwex It looks like everything is working, do you have the same issue with other pods (like you can just create another ones with other images) ?

@meliwex
Copy link
Author

meliwex commented May 13, 2022

@Al4DIN
With other Pods(with other images) I don't have such a problem
I already said about that before here - #13969 (comment)

@spowelljr
Copy link
Member

Just as a heads up, we released 1.26.0-beta.0 on Friday and that will return minikube to running --extra-config=kubelet.housekeeping-interval=10s by default on start so you don't have to manually do it.

In the case you do want to disable the metrics for improved CPU usage you can pass the --disable-metrics command on start.

@klaases
Copy link
Contributor

klaases commented Jun 29, 2022

Hi @meliwex, have you had a chance to review the latest release per the above comment, and if so, did this help resolve the issue?

@klaases
Copy link
Contributor

klaases commented Jul 13, 2022

Hi @meliwex – is this issue still occurring? Are additional details available? If so, please feel free to re-open the issue by commenting with /reopen. This issue will be closed as additional information was unavailable and some time has passed.

Additional information that may be helpful:

  • Whether the issue occurs with the latest minikube release

  • The exact minikube start command line used

  • Attach the full output of minikube logs, run minikube logs --file=logs.txt to create a log file

Thank you for sharing your experience!

@klaases klaases closed this as completed Jul 13, 2022
@meliwex
Copy link
Author

meliwex commented Oct 5, 2022

Hello @klaases, sorry for being late
Today I tested metrics-server add-on in three different versions of minikube, specifically:

  • v1.27.0
  • v1.26.0-beta.1
  • v1.26.0-beta.0

I tested metrics-server by simply creating three Pods using the following commands:

kubectl run nginx --image=nginx:latest
kubectl run httpd --image=httpd:latest
kubectl run getting-started --image=docker/getting-started:latest

These are the results:
v1.27.0:

ubuntu@server:~$ kubectl top pods 
NAME              CPU(cores)   MEMORY(bytes)   
getting-started   0m           2Mi             
httpd             1m           6Mi             
nginx             0m           2Mi             

v1.26.0-beta.1 and v1.26.0-beta.0 (they produce similar output):

ubuntu@server:~$ kubectl top pods 
NAME    CPU(cores)   MEMORY(bytes)   
httpd   1m           6Mi            

As you can see there is a problem with v1.26.0-beta.1 and v1.26.0-beta.0 versions because only metrics of httpd pod were displayed


The exact minikube start command used:
minikube start --driver=docker

My environment: AWS EC2 with OS Ubuntu 20.04
Logs: logs.txt

Note: I don't want to re-open this issue because everything works correctly with the current stable release(v1.27.0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question. long-term-support Long-term support issues that can't be fixed in code triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Development

No branches or pull requests

7 participants