Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(customresourcestate): add example to expose same metrics with different label #2580

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

docs(customresourcestate): add example to expose same metrics with different label #2580

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a4d64fb
docs(customresourestate): add example to expose same metrics with dif…
clavinjune Dec 23, 2024
1412eaa
docs(customresourestate): add kind on the example
clavinjune Dec 23, 2024
89f8110
docs(customresourestate): add example to re-expose all VPA metrics
clavinjune Jan 7, 2025
ae38bc4
docs(customresourestate): fix All VerticalPodAutoscaler Metrics section
clavinjune Jan 7, 2025
387bd8b
docs(customresourestate): fix All VerticalPodAutoscaler Metrics section
clavinjune Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
280 changes: 280 additions & 0 deletions docs/metrics/extend/customresourcestate-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ spec:
# in YAML files, | allows a multi-line string to be passed as a flag value
# see https://yaml-multiline.info
- |
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
Expand Down Expand Up @@ -65,6 +66,7 @@ spec:
# in YAML files, | allows a multi-line string to be passed as a flag value
# see https://yaml-multiline.info
- |
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
Expand Down Expand Up @@ -254,6 +256,73 @@ kube_customresource_ref_info{customresource_group="myteam.io", customresource_ki
kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_with_extensions"} 1
```

#### Same Metrics with Different Labels

```yaml
recommendation:
containerRecommendations:
- containerName: consumer
lowerBound:
cpu: 100m
memory: 262144k
```

For example in VPA we have above attributes and we want to have a same metrics for both CPU and Memory, you can use below config:

```
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: autoscaling.k8s.io
kind: "VerticalPodAutoscaler"
version: "v1"
labelsFromPath:
verticalpodautoscaler: [metadata, name]
namespace: [metadata, namespace]
target_api_version: [apiVersion]
target_kind: [spec, targetRef, kind]
target_name: [spec, targetRef, name]
metrics:
# for memory
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound"
help: "Minimum memory resources the container can use before the VerticalPodAutoscaler updater evicts it."
commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [lowerBound, memory]
# for CPU
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound"
help: "Minimum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [lowerBound, cpu]
```

Produces the following metrics:

```prometheus
# HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum memory resources the container can use before the VerticalPodAutoscaler updater evicts it.
# TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge
kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="memory",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="byte",verticalpodautoscaler="vpa-example"} 123456
# HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it.
# TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge
kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="cpu",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="core",verticalpodautoscaler="vpa-example"} 0.1
```

#### VerticalPodAutoscaler

In v2.9.0 the `vericalpodautoscalers` resource was removed from the list of default resources. In order to generate metrics for `verticalpodautoscalers`, you can use the following Custom Resource State config:
Expand Down Expand Up @@ -292,6 +361,217 @@ spec:

The above configuration was tested on [this](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/examples/hamster.yaml) VPA configuration, with an added annotation (`foo: 123`).

#### All VerticalPodAutoscaler Metrics

As an addition for the above configuration, here's the complete `CustomResourceStateMetrics` spec to re-enable all of the VPA metrics which are removed from the list of the default resources:

<details>

<summary>VPA CustomResourceStateMetrics</summary>

```yaml
kind: CustomResourceStateMetrics
spec:
resources:
- groupVersionKind:
group: autoscaling.k8s.io
kind: "VerticalPodAutoscaler"
version: "v1"
labelsFromPath:
namespace: [metadata, namespace]
target_api_version: [spec, targetRef, apiVersion]
target_kind: [spec, targetRef, kind]
target_name: [spec, targetRef, name]
verticalpodautoscaler: [metadata, name]
metricNamePrefix: "kube"
metrics:
# kube_verticalpodautoscaler_annotations
- name: "verticalpodautoscaler_annotations"
help: "Kubernetes annotations converted to Prometheus labels."
each:
type: Info
info:
labelsFromPath:
# annotation_*: [metadata, annotations]
name: [metadata, name]
# kube_verticalpodautoscaler_labels
- name: "verticalpodautoscaler_labels"
help: "Kubernetes labels converted to Prometheus labels."
each:
type: Info
info:
labelsFromPath:
# label_*: [metadata, labels]
name: [metadata, name]
# kube_verticalpodautoscaler_spec_updatepolicy_updatemode
- name: "verticalpodautoscaler_spec_updatepolicy_updatemode"
help: "Update mode of the VerticalPodAutoscaler."
each:
type: StateSet
stateSet:
labelName: "update_mode"
path: [spec, updatePolicy, updateMode]
list: ["Auto", "Initial", "Off", "Recreate"]
# Memory kube_verticalpodautoscaler_spec_resourcepolicy_container_policies_minallowed
- name: "verticalpodautoscaler_spec_resourcepolicy_container_policies_minallowed"
help: "Minimum memory resources the VerticalPodAutoscaler can set for containers matching the name."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to make metric name shorter if it's possible

For example, vpa_policies_minallowed

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to make it the same as the previous metrics which are removed, so I made it the same. If we changed the metrics name here, it defeats the purpose to re-expose the removed metrics. The plan is to have no parity between the metrics removed with this guide. wdyt @CatherineF-dev?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inferring the naming based on the Prometheus conventions, and as such, for instance, and taking into consideration Catherine's comment, this may be changed to:

vpa_spec_resourcepolicy_container_policies_minallowed_bytes

However, since the point is to polyfill these metrics, it makes sense to not break the previous names.

commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [spec, resourcePolicy, containerPolicies]
labelsFromPath:
container: [containerName]
valueFrom: [minAllowed, memory]
# CPU kube_verticalpodautoscaler_spec_resourcepolicy_container_policies_minallowed
- name: "verticalpodautoscaler_spec_resourcepolicy_container_policies_minallowed"
help: "Minimum cpu resources the VerticalPodAutoscaler can set for containers matching the name."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [spec, resourcePolicy, containerPolicies]
labelsFromPath:
container: [containerName]
valueFrom: [minAllowed, cpu]
# Memory kube_verticalpodautoscaler_spec_resourcepolicy_container_policies_maxallowed
- name: "verticalpodautoscaler_spec_resourcepolicy_container_policies_maxallowed"
help: "Maximum memory resources the VerticalPodAutoscaler can set for containers matching the name."
commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [spec, resourcePolicy, containerPolicies]
labelsFromPath:
container: [containerName]
valueFrom: [maxAllowed, memory]
# CPU kube_verticalpodautoscaler_spec_resourcepolicy_container_policies_maxallowed
- name: "verticalpodautoscaler_spec_resourcepolicy_container_policies_maxallowed"
help: "Maximum cpu resources the VerticalPodAutoscaler can set for containers matching the name."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [spec, resourcePolicy, containerPolicies]
labelsFromPath:
container: [containerName]
valueFrom: [maxAllowed, cpu]
# Memory kube_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound"
help: "Minimum memory resources the container can use before the VerticalPodAutoscaler updater evicts it."
commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [lowerBound, memory]
# CPU kube_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound"
help: "Minimum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [lowerBound, cpu]
# Memory kube_verticalpodautoscaler_status_recommendation_containerrecommendations_upperbound
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_upperbound"
help: "Maximum memory resources the container can use before the VerticalPodAutoscaler updater evicts it."
commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [upperBound, memory]
# CPU kube_verticalpodautoscaler_status_recommendation_containerrecommendations_upperbound
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_upperbound"
help: "Maximum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [upperBound, cpu]
# Memory kube_verticalpodautoscaler_status_recommendation_containerrecommendations_target
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_target"
help: "Target memory resources the VerticalPodAutoscaler recommends for the container."
commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [target, memory]
# CPU kube_verticalpodautoscaler_status_recommendation_containerrecommendations_target
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_target"
help: "Target cpu resources the VerticalPodAutoscaler recommends for the container."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [target, cpu]
# Memory kube_verticalpodautoscaler_status_recommendation_containerrecommendations_uncappedtarget
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_uncappedtarget"
help: "Target memory resources the VerticalPodAutoscaler recommends for the container ignoring bounds."
commonLabels:
unit: "byte"
resource: "memory"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [uncappedTarget, memory]
# CPU kube_verticalpodautoscaler_status_recommendation_containerrecommendations_uncappedtarget
- name: "verticalpodautoscaler_status_recommendation_containerrecommendations_uncappedtarget"
help: "Target memory resources the VerticalPodAutoscaler recommends for the container ignoring bounds."
commonLabels:
unit: "core"
resource: "cpu"
each:
type: Gauge
gauge:
path: [status, recommendation, containerRecommendations]
labelsFromPath:
container: [containerName]
valueFrom: [uncappedTarget, cpu]
```

</details>

### Metric types

The configuration supports three kind of metrics from the [OpenMetrics specification](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md).
Expand Down
Loading