Unable to scrape Kube-State-Metrics 2.8.0 when using Native Histograms Feature

[Joseph-Irving]

What did you do?

Upgraded Kube-State Metrics to 2.8.0, enabled native-histogram feature flag on Prometheus --enable-feature=native-histograms.

What did you expect to see?

Successful scraping of the kube-state-metrics endpoint

What did you see instead? Under which circumstances?

An error of proto: wrong wireType = 0 for field Metric when attempting to scrape the kube-state-metrics pod.
If native-histograms are disabled this error does not appear and it scrapes correctly as expected.

System information

No response

Prometheus version

2.42.0

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[Joseph-Irving]

I'm unsure whether this is a Kube-state-metrics issue or a Prometheus issue, but as it works when the experimental prometheus feature was disabled it made me think the issue lay within Prometheus. If that's not the case I can open an issue on the kube-state-metrics github instead. There was a change to the openmetrics format in this release kubernetes/kube-state-metrics#1974 which could be related? Seems others have seen this behaviour too prometheus-community/helm-charts#3050.

[Joseph-Irving]

It seems enabling the Native Histograms turns on the Protobuf negotiation for each scrape.
https://github.com/prometheus/prometheus/blob/v2.42.0/cmd/prometheus/main.go#L199

c.scrape.EnableProtobufNegotiation = true

Which changes the accept header to the header with protobuf https://github.com/prometheus/prometheus/blob/v2.42.0/scrape/scrape.go#L440

if sp.enableProtobufNegotiation {
    acceptHeader = scrapeAcceptHeaderWithProtobuf
}

so that would explain why it only seems to be effected when native histograms are enabled.

[Joseph-Irving]

FYI I've been looking into this on the KSM side and have opened an issue kubernetes/kube-state-metrics#2022

[Joseph-Irving]

fixed in kube-state-metrics kubernetes/kube-state-metrics#2024