Read custom resource metrics configuration from a Custom Resource Definition

[chrischdi]

What would you like to be added:

As of today, kube-state-metrics allows to define a configuration file which is read on startup to define metrics for custom resources via configuration.

Instead of using a configuration file to define custom metrics, kube-state-metrics could define its own Custom Resource Definition, which defines the metric configuration for a single Custom Resource instead.

Why is this needed:

Custom Resource Definitions can be deployed dynamically. The same could apply to the configuration of metrics for the same.

There are also had been several comments and discussions which now results in this issue (see additional context below).

Describe the solution you'd like

• Define a custom resource definition for kube-state-metrics-customresourcemetrics (maybe a replacement/successor for the customresource configuration), it should allow the same featureset as the already existing configuration.
• Adjust kube-state-metrics to dynamically read/add/remove customresourcemetrics by watching the kube-state-metrics-customresourcemetrics custom resources

UX:

• Deploy kube-state-metrics (including a CustomResourceDefinition for customresourcemetrics)
• Deploy my Custom Resource Definition foo
• Deploy my "kube-state-metrics customresourcemetrics" CR which defines the metrics for my Custom Resource Definition foo

Additional context

• We would like to use kube-state-metrics with crossplane where we create dozens of CRD's

  It would be great if kube-state-metrics could provide CustomResourceStateMetrics CRD. Whenever object of that kind will be created it will generate new crd-cofig.yaml and reload metrics.

  With such CRD we can define for our every crossplane component CustomResourceStateMetrics configuration saying how to produce metrics
  [source]

• Different issue which mentions this idea: Prevent definition of same gvk in custom resource configuration #1810 (comment)

• There also had been some slack discussions which mentioned this idea.

The pattern would be similar to what the ServiceMonitor CRD is for the Prometheus operator.

[logicalhan]

/assign @mrueg
/triage accepted

[nathanperkins]

When I originally saw the custom resource metrics configuration YAML, I thought it was a CRD 😄.

Glad to see this proposal; it would be a great solution for our current problem. We maintain multiple components using CRDs across multiple platforms, which each deploy their own KSM instance. We would like to define the metrics gathering configurations alongside the component CRDs so that they can be included in all platforms that use the component and maintained easily.

In the current situation, we would have to come up with some mechanism to merge the configs from each component into a single config, either at build time or using a sidecar in the KSM pod.

This would be so much better! Thanks for the suggestion!

[cwrau]

Heyho, is implementing a CRD a done decision?

Nothing against that, I was just thinking that maybe loading configmaps like grafana does might be easier? 🤔

[nathanperkins]

From the user side, CRDs are much easier to use than configmaps for structured APIs and the standard thing to do in Kubernetes.

• API structure is clearly defined in a way that can be documented and referenced in CLI (kubectl explain).
• The GVK makes it easy to organize and see these resources in isolation, as opposed to alongside all unrelated configmaps (kubectl get customresourcemetrics vs kubectl get configmap -l <some-label-for-customresourcemetrics>
• The API server can validate the structure without a runtime webhook.
• The controller can report status.
• Automation and IDEs can provide linting / autocomplete.
• Etc.

[guettli]

I see many people are interested to configure metrics for CRDs via a CRD.

I think it will take several months until kube-state-metrics will be able to handle this.

Maybe there is already an alternative solution (without kube-state-metrics)?

If you know an alternative CRD based solution, please leave a comment here.

[chrischdi]

For folks which are interested: I've got a working poc implementation which dynamically reads configurations from configmaps which uses the kiwigrid/k8s-sidecar.

It also uses ClusterRole aggregation.

So for new metrics I would deploy a

• configmap which contains the config (requires the correct label to be set to make the sidecar read the ConfigMap)
• ClusterRole which contains the necessary rbac (requires the correct label to be set to make ClusterRole aggregation work)

https://github.com/chrischdi/cluster-api/tree/experiment-metrics-gen-flags/hack/observability/kube-state-metrics

[chrischdi]

Cleaned up to a new branch in CAPI: kubernetes-sigs/cluster-api#9390

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[dgrisonnet]

This will likely be covered as part of kubernetes/enhancements#4785.