Collector fails with integer overflow

[mirceastoian]

Host operating system:

Linux rpi4srv2 6.1.0-rpi6-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.58-1+rpt2 (2023-10-27) aarch64 GNU/Linux

node_exporter version:

node_exporter, version 1.7.0 (branch: HEAD, revision: 7333465abf9efba81876303bb57e6fadb946041b)
  build user:       root@4214d220a8f2
  build date:       20231112-23:53:03
  go version:       go1.21.4
  platform:         linux/arm
  tags:             netgo osusergo static_build

node_exporter command line flags

/usr/local/bin/node_exporter --collector.systemd --collector.processes --collector.perf --web.listen-address=xxx:9100

node_exporter log output

2023-11-24T08:34:35.470077+02:00 rpi4srv2 node_exporter[194764]: ts=2023-11-24T06:34:35.469Z caller=collector.go:169 level=error msg="collector failed" name=processes duration_seconds=0.036614302 err="unable to retrieve number of allocated threads: error reading stat for pid 816: unsigned integer overflow on token 9950425088"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Probably after installing Kibana, the process belong to it.

Output of cat /proc/816/stat:

816 (node) R 1 816 816 0 -1 4194560 15687636 1849 654 44 256180 49710 2 8 20 0 11 0 1639 9950425088 131873 18446744073709551615 1 1 0 0 0 0 0 16781312 17923 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0

What did you expect to see?

No error in syslog.

What did you see instead?

see above

[SuperQ]

It looks like you have possibly installed the wrong platform binary. You have installed a 32-bit binary on a 64-bit system.

Your kernel says aarch64, but your binary is linux/arm not linux/arm64.

Example:

$ uname -a
Linux tock 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux
$ node_exporter --version
node_exporter, version 1.7.0 (branch: HEAD, revision: 7333465abf9efba81876303bb57e6fadb946041b)
  build user:       root@35918982f6d8
  build date:       20231112-23:54:05
  go version:       go1.21.4
  platform:         linux/arm64
  tags:             netgo osusergo static_build

[SuperQ]

However, it seems like we should be parsing with a 64-bit value anyway, since that's what the kernel is exposing.

I reviewed the code, the correct data types are used (VSize, RSSLimit). On a 32-bit system these would not produce an error.

[mirceastoian]

It looks like you have possibly installed the wrong platform binary. You have installed a 32-bit binary on a 64-bit system.

I will reinstall it and report back, however as you pointed out I feel it will not change anything.

[SuperQ]

I did a review of the ProcFS code. The values that produced an error would not have overflowed with a real 32-bit system. The values are related to memory size, which are parsed correctly as platform dependent values. I don't think any code changes are valid.

I highly recommend using https://github.com/prometheus-community/ansible as a way to install things, it ensures the correct platform binaries are deployed.

[mirceastoian]

Thanks for the tip, after reinstalling I confirm the error is gone.
I'm used to pick ARM builds for RPis and I didn't pay attention.