Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Index out of range error in parseMDStat #3198

Closed
ER-EPR opened this issue Dec 4, 2024 · 2 comments
Closed

Index out of range error in parseMDStat #3198

ER-EPR opened this issue Dec 4, 2024 · 2 comments

Comments

@ER-EPR
Copy link

ER-EPR commented Dec 4, 2024 •
edited
Loading

Host operating system: output of uname -a

Linux dl01harvester 5.14.21-150400.24.128-default #1 SMP PREEMPT_DYNAMIC Wed Aug 7 10:28:44 UTC 2024 (a6f23d4) x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.3.1 (branch: HEAD, revision: a2321e7)
build user: root@243aafa5525c
build date: 20211205-11:09:49
go version: go1.17.3
platform: linux/amd64

node_exporter command line flags

--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --web.listen-address=[$(HOST_IP)]:9796 --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/) --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

node_exporter log output

ts=2024-12-04T08:40:33.142Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"
2024-12-04T08:40:33.142899675Z ts=2024-12-04T08:40:33.142Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)"
ts=2024-12-04T08:40:33.143Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
ts=2024-12-04T08:40:33.143Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=arp
2024-12-04T08:40:33.167589481Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=bcache
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=bonding
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=btrfs
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=conntrack
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=cpu
2024-12-04T08:40:33.167619910Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=cpufreq
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=diskstats
2024-12-04T08:40:33.167639956Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=dmi
2024-12-04T08:40:33.167653853Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=edac
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=entropy
2024-12-04T08:40:33.167661318Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=fibrechannel
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=filefd
2024-12-04T08:40:33.167700466Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=filesystem
2024-12-04T08:40:33.167703446Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=hwmon
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=infiniband
2024-12-04T08:40:33.167708931Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=ipvs
2024-12-04T08:40:33.167711757Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=loadavg
2024-12-04T08:40:33.167714401Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=mdadm
2024-12-04T08:40:33.167726630Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=meminfo
2024-12-04T08:40:33.167729740Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=netclass
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=netdev
2024-12-04T08:40:33.167734993Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=netstat
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=nfs
2024-12-04T08:40:33.167750616Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=nfsd
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=nvme
2024-12-04T08:40:33.167762791Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=os
2024-12-04T08:40:33.167765824Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=powersupplyclass
2024-12-04T08:40:33.167773264Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=pressure
2024-12-04T08:40:33.167776028Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=rapl
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=schedstat
2024-12-04T08:40:33.167781018Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=sockstat
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=softnet
2024-12-04T08:40:33.167790872Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=stat
2024-12-04T08:40:33.167793592Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=tapestats
2024-12-04T08:40:33.167800476Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=textfile
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=thermal_zone
2024-12-04T08:40:33.167805716Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=time
2024-12-04T08:40:33.167810999Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=timex
2024-12-04T08:40:33.167813587Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=udp_queues
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=uname
2024-12-04T08:40:33.167821372Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=vmstat
2024-12-04T08:40:33.167828189Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=xfs
ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:115 level=info collector=zfs
2024-12-04T08:40:33.167961038Z ts=2024-12-04T08:40:33.167Z caller=node_exporter.go:199 level=info msg="Listening on" address=[0.0.0.0]:9796
2024-12-04T08:40:33.168159794Z ts=2024-12-04T08:40:33.168Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
panic: runtime error: index out of range [0] with length 0
2024-12-04T08:41:23.971372712Z 
2024-12-04T08:41:23.971377402Z goroutine 119 [running]:
2024-12-04T08:41:23.971380878Z github.com/prometheus/procfs.evalStatusLine({0xc00047809e, 0x4b}, {0xc0004780ea, 0x6})
2024-12-04T08:41:23.971385218Z 	/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/mdstat.go:170 +0x498
2024-12-04T08:41:23.971388501Z github.com/prometheus/procfs.parseMDStat({0xc000476c00, 0xc0003a8f90, 0xa})
2024-12-04T08:41:23.971401903Z 	/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/mdstat.go:105 +0x2e9
2024-12-04T08:41:23.971407103Z github.com/prometheus/procfs.FS.MDStat({{0x7ffe2d2c9fa7, 0xa}})
	/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/mdstat.go:71 +0xfb
2024-12-04T08:41:23.971413562Z github.com/prometheus/node_exporter/collector.(*mdadmCollector).Update(0xc00030b340, 0xc0000fb860)
2024-12-04T08:41:23.971417282Z 	/app/collector/mdadm_linux.go:111 +0xbb
2024-12-04T08:41:23.971420994Z github.com/prometheus/node_exporter/collector.execute({0xb1130b, 0x0}, {0xbfa980, 0xc00030b340}, 0x0, {0xbfa3c0, 0xc0002bc500})
2024-12-04T08:41:23.971435112Z 	/app/collector/collector.go:161 +0x9c
2024-12-04T08:41:23.971454659Z github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb1130b, 0x0}, {0xbfa980, 0xc00030b340})
2024-12-04T08:41:23.971485454Z 	/app/collector/collector.go:152 +0x3d
2024-12-04T08:41:23.971489837Z created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
2024-12-04T08:41:23.971492702Z 	/app/collector/collector.go:151 +0xd5

Are you running node_exporter in Docker?

yes in kubernetes, using rancher and harvester, as an harvester addon

What did you do that produced an error?

Start the [node_exporter] container.

What did you expect to see?

[node_exporter] running

What did you see instead?

[node_exporter] crashed

From the log I see it was processing /proc/mdstat
And by cat /proc/mdstat shows

Personalities : [raid6] [raid5] [raid4]
md125 : inactive sdc1[29](S) sdb1[27](S) sde1[28](S) sda1[26](S) sdd1[30](S)
      41937920 blocks super 1.2

md126 : active raid5 super 1.2 level 5, 64k chunk, algorithm 2 [4/0] [____]

md127 : inactive sdc2[28](S) sdb2[30](S) sda2[27](S) sdd2[29](S) sde2[26](S)
      10480640 blocks super 1.2

unused devices: <none>

I don't know exactly which part of this file cause the error.
@ER-EPR ER-EPR mentioned this issue Dec 4, 2024
Open
@w13915984028
Copy link

w13915984028 commented Dec 9, 2024 •
edited
Loading

The issue has been fixed via prometheus/procfs@b9b5ad9#diff-1661c40d89115402230dda8d0195c034b40fb46a3d60d05b9baa649b37277c8aL170

It is validated harvester/harvester#7095 (comment)

@ER-EPR Should we close this issue? thanks.

@ER-EPR
Copy link
Author

ER-EPR commented Dec 10, 2024

Sure.

@ER-EPR ER-EPR closed this as completed Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@w13915984028 @ER-EPR