Adhere to OTel-Prometheus standard for labels

[rexagod]

What this PR does / why we need it: Adhere to OTel-Prometheus standard for generated labels in CRS.

How does this change affect the cardinality of KSM: No change.

---

Additional info.: OSM has a hardcoded approach for doing this right now: https://github.com/openshift/openshift-state-metrics/blob/master/pkg/collectors/utils.go#L29

---

@@ -1,9 +1,9 @@
 # HELP kube_customresource_openshift_route_labels Kubernetes labels converted to Prometheus labels.
 # TYPE kube_customresource_openshift_route_labels info
-kube_customresource_openshift_route_labels{app.kubernetes.io/component="query-layer",app.kubernetes.io/instance="thanos-querier",app.kubernetes.io/name="thanos-query",app.kubernetes.io/part-of="openshift-monitoring",app.kubernetes.io/version="0.28.1",customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-monitoring",route="thanos-querier"} 1
+kube_customresource_openshift_route_labels{app_kubernetes_io_component="query-layer",app_kubernetes_io_instance="thanos-querier",app_kubernetes_io_name="thanos-query",app_kubernetes_io_part_of="openshift-monitoring",app_kubernetes_io_version="0.28.1",customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-monitoring",route="thanos-querier"} 1
 kube_customresource_openshift_route_labels{app="console",customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-console",route="console"} 1
 kube_customresource_openshift_route_labels{app="oauth-openshift",customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-authentication",route="oauth-openshift"} 1
-kube_customresource_openshift_route_labels{customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",ingress.openshift.io/canary="canary_controller",namespace="openshift-ingress-canary",route="canary"} 1
+kube_customresource_openshift_route_labels{customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",ingress_openshift_io_canary="canary_controller",namespace="openshift-ingress-canary",route="canary"} 1
 kube_customresource_openshift_route_labels{customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-console",route="downloads"} 1
 kube_customresource_openshift_route_labels{customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-monitoring",route="alertmanager-main"} 1
 kube_customresource_openshift_route_labels{customresource_group="route.openshift.io",customresource_kind="Route",customresource_version="v1",namespace="openshift-monitoring",route="prometheus-k8s"} 1

[mrueg]

As this is a breaking change to experimental _labels _annotations, we should either feature flag it or announce it visibly in a future changelog.

[logicalhan]

/triage accepted
/assign @dgrisonnet

[dgrisonnet]

It would be great if we could reuse mapToPrometheusLabels here so that we don't duplicate the logic and make sure that both part of the code use the same normalization.

[rexagod]

sanitizeLabelName felt a bit more "fitting" to me for this usecase, so I ended up using that here. LMK if you think otherwise and I'll change this accordingly.

[dgrisonnet]

As this is a breaking change to experimental _labels _annotations, we should either feature flag it or announce it visibly in a future changelog.

Should we perhaps wait for the next major version? Just to make sure we don't break anyone on a minor.

[mrueg]

As this is a breaking change to experimental _labels _annotations, we should either feature flag it or announce it visibly in a future changelog.

Should we perhaps wait for the next major version? Just to make sure we don't break anyone on a minor.

Since the _labels and _annotations metrics are experimental and not stable, I think it's fine to break them (we should just announce it visibly in the changelog).

[dgrisonnet]

Sounds good 👍

[mrueg]

/lgtm

thanks for your work on this @rexagod !

/hold

for others to review as well

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrueg, rexagod

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mrueg,rexagod]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dgrisonnet]

/unhold