Adapter authorization issue

[rbrigden]

When I check the logs with kubectl -n custom-metrics logs -l=app=k8s-cloudwatch-adapter, the following error message is firing off repeatedly

E0804 16:13:31.877587       1 reflector.go:153] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:custom-metrics:k8s-cloudwatch-adapter" cannot list resource "configmaps" in API group "" in the namespace "kube-system"

[lanecm]

@rbrigden -- I had same problem, but fixed by adding configmaps to k8s-cloudwatch-adapter-resource-reader cluster role:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-cloudwatch-adapter-resource-reader
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
      - pods
      - services
      - configmaps
    verbs:
      - get
      - list

Obviously, this still needs to be fixed in the manifest.

[chankh]

the usage of configmap had been removed, which version are you using?

[lanecm]

@chankh -- I'm using:

image: chankh/k8s-cloudwatch-adapter:v0.9.0

which matches: https://github.com/awslabs/k8s-cloudwatch-adapter/blob/master/deploy/adapter.yaml

[chankh]

@lanecm do you mind checking if your k8s-cloudwatch-adapter deployment has reference to any configmaps? The reference to configmap was removed since 93feeca

[lanecm]

@chankh -- Oh, good catch... Yes, I added a kustomize patch to set AWS_REGION via a configmap we have in the cluster, e.g.,

- op: add
  path: /spec/template/spec/containers/0/env
  value:
    - name: AWS_REGION
      valueFrom:
        configMapKeyRef:
          key: AWS_REGION
          name: env-region-config

Is AWS_REGION required?

[chankh]

By default the adapter will try to get the region from EC2 metadata API, unless you are getting metrics from a different region, you should not need to specify AWS_REGION

[lanecm]

@chankh -- I still get the following error:

F0813 11:32:12.206953       1 adapter.go:104] unable to run CloudWatch metrics adapter: unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:custom-metrics:k8s-cloudwatch-adapter" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

[chankh]

@lanecm seems like the adapter is trying to get configmap extension-apiserver-authentication, where was this defined?

[lanecm]

@chankh -- Searching for extension-apiserver-authentication returns two items in our repo:

1. In k8s-cloudwatch-adapter:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: k8s-cloudwatch-adapter-auth-reader
  namespace: custom-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: k8s-cloudwatch-adapter
    namespace: custom-metrics

2. In metrics-server:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  labels:
    app: metrics-server
    chart: metrics-server-2.8.8
    heritage: Tiller
    release: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

Might be a problem with the identical named rolebindings?

[chankh]

@lanecm I've check the configs from other adapters as well, they all seems to be sharing the extension-apiserver-authentication-reader. Also checked in my cluster, the role extension-apiserver-authentication-reader has the policy rules to get/list/watch the extension-apiserver-authentication configmap.

Name:         extension-apiserver-authentication-reader
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources   Non-Resource URLs  Resource Names                        Verbs
  ---------   -----------------  --------------                        -----
  configmaps  []                 [extension-apiserver-authentication]  [get list watch]

looks like the permissions should be granted there

[patelronak]

@chankh I am also getting same error with eks 1.14

 E0828 20:08:20.342746       1 reflector.go:153] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:custom-metrics:k8s-cloudwatch-adapter" cannot list resource "configmaps" in API group "" in the namespace "kube-system"

If I try to add configmaps permission to resources as @lanecm suggested to k8s-cloudwatch-adapter-resource-reader I get following error

E0828 22:22:19.502214       1 reflector.go:309] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to watch *v1.ConfigMap: unknown (get configmaps)

Any ideas?

[acgs771126]

@patelronak I fixed it error message after grant extension-apiserver-authentication-reader permissions.
command: kubectl edit roles.rbac.authorization.k8s.io -n kube-system extension-apiserver-authentication-reader

rules:
- apiGroups:
  - ""
  resourceNames:
  - extension-apiserver-authentication
  resources:
  - configmaps
  verbs:
    - get
+  - list
+  - watch

[patelronak]

@acgs771126 thank you very much for that insight. I was able to get over that error but pods are still not scaling and I do not see any errors in any of the logs. I am seeing unknown for targets when I look at hpa (sqs-consumer-scaler)

NAME                  REFERENCE                 TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
sqs-consumer-scaler   Deployment/sqs-consumer   <unknown>/30 (avg)   1         10        0          2d19h

kubectl -n custom-metrics logs k8s-cloudwatch-adapter-59bf47d556-ntsj5
is showing that sqs_helloworld_length values are computed correctly

I0831 16:36:46.328608       1 handler.go:69] externalMetricInfo: &{{ } {sqs-helloworld-length  default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/sqs-helloworld-length 87f56031-e893-11ea-8a2f-0e0c91594855 183392849 1 2020-08-27 18:31:32 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","metadata":{"annotations":{},"name":"sqs-helloworld-length","namespace":"default"},"spec":{"name":"sqs-helloworld-length","queries":[{"id":"sqs_helloworld_length","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"helloworld"}],"metricName":"ApproximateNumberOfMessagesVisible","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"resource":{"resource":"deployment"}}}
] [] []  []} {sqs-helloworld-length [{  sqs_helloworld_length  {{[{QueueName helloworld}] ApproximateNumberOfMessagesVisible AWS/SQS} 60 Average Count} true}]}}
I0831 16:36:46.328717       1 handler.go:114] adding to cache item 'sqs-helloworld-length' in namespace 'default'
I0831 16:36:46.328726       1 controller.go:122] successfully processed item '{default/sqs-helloworld-length ExternalMetric}'
I0831 16:36:46.328735       1 controller.go:79] processing next item
I0831 16:36:46.328744       1 controller.go:86] processing itemI0831 16:36:46.328608       1 handler.go:69] externalMetricInfo: &{{ } {sqs-helloworld-length  default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/sqs-helloworld-length 87f56031-e893-11ea-8a2f-0e0c91594855 183392849 1 2020-08-27 18:31:32 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","metadata":{"annotations":{},"name":"sqs-helloworld-length","namespace":"default"},"spec":{"name":"sqs-helloworld-length","queries":[{"id":"sqs_helloworld_length","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"helloworld"}],"metricName":"ApproximateNumberOfMessagesVisible","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"resource":{"resource":"deployment"}}}
] [] []  []} {sqs-helloworld-length [{  sqs_helloworld_length  {{[{QueueName helloworld}] ApproximateNumberOfMessagesVisible AWS/SQS} 60 Average Count} true}]}}
I0831 16:36:46.328717       1 handler.go:114] adding to cache item 'sqs-helloworld-length' in namespace 'default'
I0831 16:36:46.328726       1 controller.go:122] successfully processed item '{default/sqs-helloworld-length ExternalMetric}'
I0831 16:36:46.328735       1 controller.go:79] processing next item
I0831 16:36:46.328744       1 controller.go:86] processing item

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
sqs-consumer   1/1     1            1           14m

The hpa config looks correct to me

apiVersion: v1
items:
- apiVersion: autoscaling/v1
  kind: HorizontalPodAutoscaler
  metadata:
    annotations:
      autoscaling.alpha.kubernetes.io/metrics: '[{"type":"External","external":{"metricName":"sqs-helloworld-length","targetAverageValue":"30"}}]'
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sqs-consumer-scaler","namespace":"default"},"spec":{"maxReplicas":10,"metrics":[{"external":{"metricName":"sqs-helloworld-length","targetAverageValue":30},"type":"External"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1beta1","kind":"Deployment","name":"sqs-consumer"}}}
    creationTimestamp: "2020-08-28T21:23:41Z"
    name: sqs-consumer-scaler
    namespace: default
    resourceVersion: "183746041"
    selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/sqs-consumer-scaler
    uid: beef9d00-e974-11ea-8a2f-0e0c91594855
  spec:
    maxReplicas: 10
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1beta1
      kind: Deployment
      name: sqs-consumer
  status:
    currentReplicas: 0
    desiredReplicas: 0
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[patelronak]

I was able to resolve the above error by using helm deployment instead of the adapter.yaml file deployment for the adapter.

[acgs771126]

what is your external metric setting?
and also you can kubectl describe hpa to checking why hpa get unknown status.