[MetricBeat] Kubernetes module Go struct field .pods.volume.inodesFree of type int64

[dbpolito]

MetricBeat Version: 7.8.0
K8S Version: Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Errors:

2020-06-28T16:34:33.506Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-06-28T16:34:36.437Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-06-28T16:34:36.970Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-06-28T16:34:38.905Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[exekias]

Thank you for opening @dbpolito, this definitely looks like a bug, could you please detail how are you deploying Metricbeat?

[dbpolito]

Deploying with helm version 7.8.0 with the following values.

daemonset:
  hostNetworking: true

[TeroPihlaja]

We are seeing exactly similar messages:

11:41:08.750
2020-07-01T08:41:08.750Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
11:41:08.943
2020-07-01T08:41:08.943Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
11:41:09.353
2020-07-01T08:41:09.353Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
11:41:09.519
2020-07-01T08:41:09.519Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64

I deployed using the https://raw.githubusercontent.com/elastic/beats/7.8/deploy/kubernetes/metricbeat-kubernetes.yaml

[mehrnazix]

I have same issue. K8 version.Info{Major:"1", Minor:"19"}. I got this errors:

2020-09-30T08:19:13.831Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-09-30T08:19:23.764Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-09-30T08:19:23.822Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-09-30T08:19:23.822Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.node: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-09-30T08:19:23.827Z        INFO    module/wrapper.go:259   Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64

[matthew-mcdermott]

I am getting the same issue and cannot seem to do anything to work around it. I am using metricbeat 7.7.1, k8s 1.18.10, and ES 7.7.0.

2020-12-29T21:15:49.367Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-12-29T21:15:49.713Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-12-29T21:15:49.887Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2020-12-29T21:15:55.305Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64```

Edit: Quick update, I tried again using metricbeat 7.10.1, k8s 1.18.10, and ES 7.10.0 and the issue still exists exactly the same way. 

[davidspek]

I am also getting this issue on Kubernetes 1.20, metricbeat 7.10.1 deployed using ECK 1.3.1.

2021-01-01T18:30:59.489Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:30:59.991Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:30:59.988Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:04.288Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:09.487Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:09.688Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:09.888Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:13.887Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:19.088Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:19.288Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64
2021-01-01T18:31:19.888Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error in mapping: cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64

Upon closer inspection it seems 1 of my 3 nodes is not showing the above errors. This has persisted after many reconfigurations so I'm not sure what is different with the one node allowing to to grab the metrics properly.

[ChrsMark]

Hey! It is a bug indeed. It can happen regardless the version of k8s whenever we hit an integer overflow. We store these integers values as int64 while the API can serve uint64 (which means bigger unsigned values which we cannot parse). I will open a PR for this soon.

[fperalt3]

Hi. We are getting the same issue with metricbeat-oss:7,9,1 docker version with the field: .pods.volume.inodesFree

error in mapping: Cannot unmarshal json response: json: cannot unmarshal number 18446744073709551615 into Go struct field .pods.volume.inodesFree of type int64

[ChrsMark]

@fperalt3 this one will be fixed at 7.11 which is coming really soon. In the meantime you can check with 7.11 snapshot docker.elastic.co/beats/metricbeat:7.11.0-SNAPSHOT which includes the fix.

[fperalt3]

Thank you @ChrsMark for your response. But we should need to use the OSS version because we are integrating Beats with Elastic-OSS due to we cannot use Xpack (aws Elastic Service). Is there any metricbeat-oss version with this patch?

[ChrsMark]

@fperalt3 you can use the equivalent for oss which is docker.elastic.co/beats/metricbeat-oss:7.11.0-SNAPSHOT ;)

[fperalt3]

:) I didn't know that it was also available .
Thank you!