Kyverno-pre is crashing with error couldn't get resource list for custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1

[abhishekghiya]

Kubernetes Environemnt : D2iq Konvoy

Kubernetes version : 1.18

Kyverno version: 1.3.0-rc8

After installing the Kyverno in D2iq Konvoy Kubernetes cluster, Getting below error in kyverno-pre container logs and container went into crash back off loop state.

I0118 09:22:28.083001 1 main.go:140] "msg"="Using in-cluster configuration"
E0118 09:22:28.145511 1 memcache.go:206] couldn't get resource list for custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1
E0118 09:22:28.146222 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.146264 1 client.go:286] dclient "msg"="schema not found" "kind"="ValidatingWebhookConfiguration"
E0118 09:22:28.149584 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.149622 1 client.go:286] dclient "msg"="schema not found" "kind"="ValidatingWebhookConfiguration"
E0118 09:22:28.151146 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.151175 1 client.go:286] dclient "msg"="schema not found" "kind"="MutatingWebhookConfiguration"
E0118 09:22:28.151197 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.151249 1 client.go:286] dclient "msg"="schema not found" "kind"="MutatingWebhookConfiguration"
E0118 09:22:28.152362 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.152388 1 client.go:286] dclient "msg"="schema not found" "kind"="ValidatingWebhookConfiguration"
E0118 09:22:28.152976 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.153003 1 client.go:286] dclient "msg"="schema not found" "kind"="ValidatingWebhookConfiguration"
E0118 09:22:28.153435 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.153466 1 client.go:286] dclient "msg"="schema not found" "kind"="MutatingWebhookConfiguration"
E0118 09:22:28.154244 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.154265 1 client.go:286] dclient "msg"="schema not found" "kind"="MutatingWebhookConfiguration"
E0118 09:22:28.154872 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.154892 1 client.go:286] dclient "msg"="schema not found" "kind"="Namespace"
E0118 09:22:28.155725 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.155753 1 client.go:286] dclient "msg"="schema not found" "kind"="ClusterPolicyReport"
E0118 09:22:28.158599 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.158630 1 client.go:286] dclient "msg"="schema not found" "kind"="ReportChangeRequest"
E0118 09:22:28.159276 1 client.go:337] dclient "msg"="failed to get registered preferred resources" "error"="unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: Got empty response for: custom.metrics.k8s.io/v1beta1"
I0118 09:22:28.159297 1 client.go:286] dclient "msg"="schema not found" "kind"="ClusterReportChangeRequest"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x115b2ba]

goroutine 38 [running]:
main.removeReportChangeRequest(0xc000089220, 0x13bfe9c, 0x13, 0xc00039a201, 0xc0000adef0)
/home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:312 +0x13a
main.executeRequest(0xc000089220, 0x13bfe9c, 0x13, 0x13b4908, 0x0, 0x0, 0x0)
/home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:128 +0x179
main.process.func1(0xc00039a300, 0xc00039a2a0, 0xc000089220, 0xc00039a240, 0x158b6a0, 0xc000384900, 0xc0001145a0)
/home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:188 +0xfb
created by main.process
/home/runner/work/kyverno/kyverno/cmd/initContainer/main.go:184 +0xd7

[realshuting]

@abhishekghiya Thanks for reporting the issue!

Can you upgrade to v1.3.0 and try again? It's better to uninstall Kyverno and re-install it.

We've seen the metrics server sometimes brings an issue to Kyverno, here's a reference that may help on debugging the problem.

[abhishekghiya]

Thanks @realshuting for quick reply

Looks like Prometheus Adaptor addon was down (pod was in pending state) in the kubernetes cluster that was causing the "custom.metrics.k8s.io/v1beta1" to return empty response. After fixing the addon kyverno came up successfully.

Our issue is resolved now but just wanted to check if kyverno has any dependeny on the availablilty of metrics api resources "custom.metrics.k8s.io/v1beta1" to come up ?

Thanks

[realshuting]

No there’s no dependency on this api group. Kyverno checks if all CRDs are installed correctly during start time, while it gets the error response from API server, so it exits and keeps restarting. We’ll send a patch to ignore such an error along with the fix for #1490.

…

On Fri, Jan 22, 2021 at 10:16 PM abhishekghiya ***@***.***> wrote: Thanks @realshuting <https://github.com/realshuting> for quick reply Looks like Prometheus Adaptor addon was down (pod was in pending state) in the kubernetes cluster that was causing the "custom.metrics.k8s.io/v1beta1" to return empty response. After fixing the addon kyverno came up successfully. Just wanted to check if kyverno has any dependeny on the availablilty of apis "custom.metrics.k8s.io/v1beta1" to come up ? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1481 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGEJFLWBTGTFOFP7FMYQWLLS3JSVNANCNFSM4WHASRPA> .

[abhishekghiya]

Thanks @realshuting for all your help and clarification