BTRFS stats missing #2632
Comments
|
Ah, just found this comment: #2173 (comment) Is the issue that I am on a kernel earlier than 5.14? If this is the issue then I will open a pull to add this exception to the docs/README. Output of the sysfs directory: cc @leth |
|
The implementation gathers this data via syscalls rather than Can you try with debug logging enabled please! You should be able to grep it down to just things mentioning btrfs. It'd also be helpful to know what user node_exporter is running as, the ownership & permissions on the mount points, and if there's anything else relevant in-play like containers etc! |
|
Not running in a container at all, but it is running as a systemd service on Debian 11. Here is the service file: [Unit]
Description=Prometheus Node Exporter
Requires=prometheus-node-exporter.socket
[Service]
User=prom-node-exporter
Group=prom-node-exporter
EnvironmentFile=/etc/default/prometheus-node-exporter
ExecStart=/usr/local/bin/prometheus-node-exporter --web.systemd-socket $OPTIONS
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
It is running as a system user with its own group. Let me know if it needs to be in any other groups in order to access the stats: Relevant fstab line: It's just the main root mountpoint Probably the culprit line with debug logging: I think this maybe should show up as an error and not a debug log line? Or maybe it could error if you explcitly enable the BTRFS node_exporter module? It also still says collector succeeded. Surely I don't need to run the exporter as root? I would rather not do that, although I could jail it in pretty well with systemd unit restrictions if I needed to. Debug log (removed anything from the systemd collector): |
|
Hmm, at least running |
|
The implementation here makes the same syscalls as I was torn on what level to log at; the filesystem could be mounted in multple places, so it might not be a fatal error, and we fall back to the old metrics from I never got around to joining the btrfs mailing list to feed back to them about the data in /sys/fs. |
Are you able to use it without root? Or are you using a capability or additional group? What kernel version are you on (it may have been patched later)? |
|
The error you're seeing comes from here: node_exporter/collector/btrfs_linux.go Lines 128 to 137 in 1724b28 The error is returned from call here: OpenFile maps to the open syscall. I just had a small look at the man page, perhaps the O_PATH flag might reduce the permissions required? I'll see if I can try it out |
No luck unfortunately! The ioctl calls simply fail with "bad file descriptor". |
|
I see, so the issue is that @leth What happens if you remove the It's strange it has |
|
OK, yes I think that is probably it (I have not tried this myself). Unrelated project with the same issue: https://gitlab.gnome.org/GNOME/glib/-/issues/874#note_262355 (see last comment) So this means that:
|
|
Further info: https://man7.org/linux/man-pages/man2/openat.2.html So falling back to without |
|
Oh nice! Thanks, I'll try and give that a try over the next few days.
…On 11 Mar 2023, 21:56, at 21:56, Perry Naseck ***@***.***> wrote:
Further info:
```
O_NOATIME (since Linux 2.6.8)
Do not update the file last access time (st_atime in the
inode) when the file is
[read(2)](https://man7.org/linux/man-pages/man2/read.2.html).
This flag can be employed only if one of the following
conditions is true:
* The effective UID of the process matches the owner UID
of the file.
* The calling process has the CAP_FOWNER capability in
its user namespace and the owner UID of the file has a
mapping in the namespace.
```
https://man7.org/linux/man-pages/man2/openat.2.html
So falling back to without `O_NOATIME` seems to be the best option.
--
Reply to this email directly or view it on GitHub:
#2632 (comment)
You are receiving this because you were mentioned.
Message ID:
***@***.***>
|
|
It works, thank you so much for the help! I'll raise the PR on https://github.com/dennwc/btrfs to get this updated :) |
This was referenced Mar 12, 2023
|
@leth Amazing! We just need to update deps here and the issue is solved? |
|
Yeah, though you may have to wait for that PR to get merged and the next node_exporter release too (unless you're building from source), I don't really know how frequent they are! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Host operating system: output of
uname -anode_exporter version: output of
node_exporter --versionnode_exporter command line flags
node_exporter log output
Are you running node_exporter in Docker?
No
What did you do that produced an error?
No error but BTRFS stats are missing.
What did you expect to see?
Missing stats from #2193 like:
What did you see instead?
The following BTRFS metrics appear:
I also do see the BTRFS volume in
node_filesystemmetrics:Output of some BTRFS stats (not through
node_exporter):Note that
/dev/nvme0n1p2and/dev/nvme1n1p2are a BTRFS RAID1 arrayNOTE: Apologies for the initial confusing issue, accidentally hit enter too early!
The text was updated successfully, but these errors were encountered: