keda operator restarted at the time of start.(error retrieving resource lock keda/operator.keda.sh)

[vkamlesh]

Keda operator failed to elect leader after keda-operator pod restart. These restarts are not frequent but it's happening in a few days(6 days) time intervals.

KEDA Version: 2.6.1
Git Commit: efca71d

Kubernetes version: v1.20.9
Kubernetes Cluster : AKS

bash-3.2$ k get po -n keda
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-649f4ddbbd-v4pjp   1/1     Running   0          12d
keda-operator-68ddbdcc8f-6h767            1/1     Running   3          12d
bash-3.2$ 


bash-3.2$ kubectl get --raw "/apis/coordination.k8s.io/v1/namespaces/keda/leases/operator.keda.sh"
{"kind":"Lease","apiVersion":"coordination.k8s.io/v1","metadata":{"name":"operator.keda.sh","namespace":"keda","uid":"edb18fd7-b95e-463f-81cf-6a1010073409","resourceVersion":"135421212","creationTimestamp":"2022-02-23T14:33:32Z","managedFields":[{"manager":"keda","operation":"Update","apiVersion":"coordination.k8s.io/v1","time":"2022-02-23T14:33:32Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:acquireTime":{},"f:holderIdentity":{},"f:leaseDurationSeconds":{},"f:leaseTransitions":{},"f:renewTime":{}}}}]},"spec":{"holderIdentity":"keda-operator-68ddbdcc8f-6h767_53e840b6-5466-484f-a6f6-16978b7ee12c","leaseDurationSeconds":15,"acquireTime":"2022-03-07T12:07:54.000000Z","renewTime":"2022-03-07T17:36:42.260717Z","leaseTransitions":142}}




bash-3.2$ k logs keda-operator-68ddbdcc8f-6h767 -n keda -f -p


1.6466548397264059e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "observationsprocessor-func", "namespace": "platform-api"}
E0307 12:07:33.812275       1 leaderelection.go:330] error retrieving resource lock keda/operator.keda.sh: Get "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/keda/leases/operator.keda.sh": context deadline exceeded
I0307 12:07:33.812329       1 leaderelection.go:283] failed to renew lease keda/operator.keda.sh: timed out waiting for the condition
1.6466548538123553e+09	ERROR	setup	problem running manager	{"error": "leader election lost"}`

[crisp2u]

We've seeing this too, there were issues with cluster api connection and keda remains in non-ready state and does not scale anymore after restart. We are seeing restarts on other operators in the cluster (i.e. cluster autoscaler in eks) but they are able to recover after restart. @vkamlesh Can you create an issue please ?