Automatic RBAC support for kubeletstats receiver and k8sobserver extension

[grandwizard28]

Component(s)

collector

Describe the issue you're reporting

There seems to be missing rbac permissions for the collector when trying to use receivers which query the K8s API.
There have been 2 instances which I have noticed:

1. The k8sobserver extension does not get the required rbac permissions. If I add the k8sattributes processor, then the permissions seem to be added.
2. The kubeletstats receiver is throwing a 403 forbidden when trying to query node metrics:

kubelet request GET https://xxxxxxx:10250/stats/summary failed - \"403 Forbidden\", response: \"Forbidden (user=system:serviceaccount:observability:<name-of-collector>, verb=get, resource=nodes, subresource=stats)\"", "scraper": "kubeletstats"

Expectation

The operator should automatically add permissions for the extensions/receivers to work. If not, what is the recommended method to add additional permissions to the collector?

Setup

Running a bunch of collectors as dameonset managed by the operator

[jaronoff97]

We don't yet support automatic RBAC for the kubelet stats receiver so for now you'll need to follow the steps in the kubelet stats documentation. That being said, this is definitely something we could/should do, i'll add it to our backlog :)

[grandwizard28]

I mentioned the k8sobserver extension also in this ticket. How would we go about that?

Do you want me to create a separate ticket for the extension?

[jaronoff97]

Maybe just change this issue's title to include it, thank you.

[rogercoll]

This would be really helpful and it will align with the kubelet preset of the collector's Helm Chart: https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector#configuration-for-retrieving-kubelet-metrics

I would suggest taking into account the optional node metrics of the kubeletstats receiver, if they are enabled by the user (e.g. kubeletstats::metrics::k8s.pod.memory.node.utilization::enabled: true), additional node permissions are required:

  rules:
    - apiGroups:
        - ""
      resources:
        - nodes/proxy
      verbs:
        - get
    - apiGroups:
        - ""
      resources:
        - nodes
      verbs:
        - get
        - watch
        - list

[iblancasa]

I'll work on this as soon as #3206 is merged :)