Getting panic for metrics cadvisor command

[Dentrax]

Summary

Provide brief overview and context for the discovered bug.

Steps to Reproduce

1. curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.11/kubeletctl_darwin_amd64 && chmod a+x ./kubeletctl_darwin_amd64 && mv ./kubeletctl_darwin_amd64 /usr/local/bin/kubeletctl
2. kubeletctl -s 10.11.12.13 metrics cadvisor
3. See error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1763c4e]

goroutine 1 [running]:
kubeletctl/cmd.PrintPrettyHttpResponse(0x0, {0xc0000f17d0?, 0xc00021fd48?})
	/home/cyber/kubeletctl/cmd/print.go:56 +0x2e
kubeletctl/cmd/metrics.glob..func1(0x1f0d520?, {0x18c600e?, 0x4?, 0x4?})
	/home/cyber/kubeletctl/cmd/metrics/cadvisor.go:45 +0x4f
github.com/spf13/cobra.(*Command).execute(0x1f0d520, {0xc000324f00, 0x4, 0x4})
	/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:842 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x1f0cd40)
	/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:943 +0x37d
github.com/spf13/cobra.(*Command).Execute(...)
	/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:883
kubeletctl/cmd.Execute()
	/home/cyber/kubeletctl/cmd/root.go:83 +0x25
main.main()
	/home/cyber/kubeletctl/main.go:20 +0x17

Expected Results

It should work?

Actual Results

Throws panic.

Reproducible

• Always
• Sometimes
• Non-Reproducible

Version/Tag number

1.11

K8s: v1.23.7

Environment setup

macOS

Additional Information

Add any other context about the problem here.

[g3rzi]

It should work, but even if not, it shouldn't throw panic.
I didn't test this binary in macOs, but I can check tomorrow, maybe I will be able to do it.

To troubleshoot I have a number of questions.
If you run the below command:

curl -k https://10.11.12.13:10250/metrics/cadvisor  

Do you receive any results?

Another question,
If you run:

kubeletctl -s 10.11.12.13 healthz

Does it work?

[Dentrax]

I think you can't auth to kubelet from the local:

curl -s -k https://10.11.12.13:10250/metrics/cadvisor
Unauthorized

You should pass the necessary certs: (on remote host)

curl -s -k https://localhost:10250/metrics/cadvisor --cert /etc/kubernetes/ssl/apiserver-kubelet-client.crt --key /etc/kubernetes/ssl/apiserver-kubelet-client.key

healthz is working as expected:

kubeletctl -s 10.11.12.13 healthz
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
ok

[g3rzi]

Thank you for providing that information.
The healthz API worked for you so there is no problem with the compilation of the binary.
But there is some problem with the /metrics/cadvisor API.

In your example, you wrote the command like that:

kubeletctl -s 10.11.12.13 metrics cadvisor

You didn't specify any certificates so the result should also be Unauthorized when using kubeletctl.

We will test this API again and update it here.

[g3rzi]

Small update.
When I tested it I canceled the AuthN and AuthZ just for tests:

sed -i '/^authentication:$/,/^  webhook:$/ s/^\(\s*enabled:\s*\)false/\1true/' /var/lib/kubelet/config.yaml
sed -i '/^authorization:$/,/^  webhook:$/ s/^\(\s*mode:\s*\)Webhook/\1AlwaysAllow/' /var/lib/kubelet/config.yaml

To restore:

sed -i '/^authentication:$/,/^  webhook:$/ s/^\(\s*enabled:\s*\)true/\1false/' /var/lib/kubelet/config.yaml
sed -i '/^authorization:$/,/^  webhook:$/ s/^\(\s*mode:\s*\)AlwaysAllow/\1Webhook/' /var/lib/kubelet/config.yaml

In my case, on Ubuntu 22.04 LTS the command worked:

./build/kubeletctl_linux_amd64 -i metrics cadvisor

I received all the information.

The question is if the problem is because of using it on macOs or your metrics\cadvisor data cause it.
We will keep investigating.
I will check if I can run it on macOs and test it.

[g3rzi]

@Dentrax
I was able to reproduce it in MacOs Intel.

What CPU you have? Intel\M1\M2 ?
You can run the commands:

sysctl -n machdep.cpu.brand_string
uname -a

I want to know if it happens on M1 or M2.
My next plan is to debug it on macOs.

[Dentrax]

Hey, sorry missed your comment. @g3rzi

Apple M1 Max
Darwin REDACTED 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000 arm64

[g3rzi]

Can you try to build the code with darwin/arm64 (following @Rajchowdhury420 PR #38) and then test it?

[RajChowdhury240]

only with darwin/arm64 works fine.

[g3rzi]

It should be OK now. If after building darwin/arm64 it doesn't work, let us know.

[Dentrax]

Thanks @g3rzi, are you going to cut a new version soon? It seems almost a year passed since last tag.

[g3rzi]

Yes, published now version 1.12:
https://github.com/cyberark/kubeletctl/releases/tag/v1.12