[metricbeat] Missing clusterRole permissions for leader election in k8s 1.19+

[gwvandesteeg]

Chart version:

7.15+

With the leader election enabled in the kubernetes autodiscovery and running on a cluster 1.19+ the leader election fails due to a lack of permissions in the clusterRole being used.

The below

helm-charts/metricbeat/values.yaml

Lines 240 to 261 in e077086

	clusterRoleRules:
	- apiGroups: [""]
	resources:
	- nodes
	- namespaces
	- events
	- pods
	verbs: ["get", "list", "watch"]
	- apiGroups: ["extensions"]
	resources:
	- replicasets
	verbs: ["get", "list", "watch"]
	- apiGroups: ["apps"]
	resources:
	- statefulsets
	- deployments
	- replicasets
	verbs: ["get", "list", "watch"]
	- apiGroups: [""]
	resources:
	- nodes/stats
	verbs: ["get"]

needs to be expanded with:

  - apiGroups: ["coordination.k8s.io"]
    resources:
      - leases
    verbs: ["create", "get", "list", "update"]

Steps to reproduce:

1. Deploy on a 1.19 or later k8s cluster
2. Configure metricbeat with the following excerpt (add output destination as needed)

      metricbeatConfig:
        metricbeat.yml: |
          # enable the http endpoint for health checks
          http:
            enabled: true
            host: localhost
            port: 5066
   
          # autodiscover kubernetes nodes and collect the metrics from the workers
          metricbeat.autodiscover:
            providers:
              # this uses leader election to have one run as the master and have
              # the leader scrape the kube-state-metrics endpoint and the
              # kubernetes API endpoint
              - type: kubernetes
                scope: cluster
                node: ${NODE_NAME}
                unique: true
                identifier: leader-election-metricbeat
                templates:
                  - config:
                      # kubernetes state metrics
                      - module: kubernetes
                        hosts: ["kube-state-metrics.metrics:8080"]
                        period: 10s
                        add_metadata: true
                        metricsets:
                          - state_node
                          - state_deployment
                          - state_daemonset
                          - state_replicaset
                          - state_pod
                          - state_container
                          - state_job
                          - state_cronjob
                          - state_resourcequota
                          - state_statefulset
                          - state_service
                      # API server metrics
                      - module: kubernetes
                        metricsets:
                          - apiserver
                        hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                        ssl.certificate_authorities:
                          - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                        period: 30s
                      # kubernetes events
                      - module: kubernetes
                        metricsets:
                          - event

Without this the pod logs will be filled with:

error retrieving resource lock metrics/metricbeat-cluster-leader: leases.coordination.k8s.io "metricbeat-cluster-leader" is forbidden: User "system:serviceaccount:metrics:metricbeat-metricbeat" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "metrics"

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.