Using kube-state-metrics (custom resource state metrics) seems to upset metricbeat

[MKruger777]

Hi there,

I am running metricbeat:8.6.1 and prometheus:2.43.1 on AKS 1.25.6
Both of these are scraping metrics from kube-state-metrics:2.8.2

Because of an edge case we were forced to make use of the Custom Resource State Metrics functionality within kube-state-metrics as described here:
https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md

A new metric was introduced that is of the type “Info”.
After this was deployed to kube-state-metrics Prometheus was able to scrape this and all worked perfect.

The only problem is that it seems to break meticbeat.
When looking in the logs, we see the following:
decoding of metric family failed: text format parsing error in line 13966: unknown metric type "info".

Is this a case of missing functionality or miss configuration?

Many thanks!

Kind regards,
Morné Kruger

Tasks

Preview Give feedback

No tasks being tracked yet.

[tetianakravchenko]

Hey @MKruger777

I've tested it and on stack and metricbeat version 8.6.2 I indeed can see this error:

{"log.level":"error","@timestamp":"2023-09-13T12:42:57.132Z","log.logger":"kubernetes.state_pod","log.origin":{"file.name":"kubernetes/state_metricset.go","file.line":97},"message":"decoding of metric family failed: text format parsing error in line 1693: unknown metric type \"info\"","service.name":"metricbeat","ecs.version":"1.6.0"}

In version 8.7 were introduced textparser in prometheus module helper library. I've tested 8.7.1 version and I am not getting the error above, all kubernetes metrics are available.

Can you upgrade metricbeat version to at least 8.7.1? You can use output.elasticsearch.allow_older_versions for testing before upgrading the stack.

Alternatively, I think the better option would be to run 2 kube-state-metrics. One for all standard metrics and another one for custom resource metrics only with --custom-resource-state-only=true flag.

[MKruger777]

Hey @tetianakravchenko , thank you so much for checking this out - much appreciated!
I will give the alternatives a whirl and get back to you.

[tetianakravchenko]

I will close this issue, since there were no updates. @MKruger777 feel free to reopen if any help is needed