Scrape CPU latency stats from /proc/schedstat

[bitglue]

These are useful as a direct indication of CPU contention and task
scheduler latency.

Handy references:

• https://github.com/torvalds/linux/blob/master/Documentation/scheduler/sched-stats.txt
• https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha.tuning.taskscheduler.html

procfs is updated to pull in the enabling change:
prometheus/procfs#186

Signed-off-by: Phil Frost phil@postmates.com

[bitglue]

@SuperQ @discordianfish I reckon this is ready for review. This is my first node_exporter PR so please let me know if there are any things to fix.

[SuperQ]

Looks pretty good so far.

I know it's a bit of a pain, but we're trying to move proc file parsing to prometheus/procfs. We're trying to avoid adding new code that uses procFilePath() and adds parsing directly in the node_exporter.

[bitglue]

No problem, I'll move the parsing logic over there.

[bitglue]

Sorry, go changes its dependency management practices every time I use it, and I'm not sure how best to convince CI to build with prometheus/procfs#186 before those changes have been accepted and released. But the changes are pretty straightforward so it shouldn't be too hard to figure it out on the fly.

[SuperQ]

Yea, it's hard to do external tests on a remote branch without doing crazy stuff with go modules.

[bitglue]

@SuperQ procfs changes have been merged, and I've updated the PR. Would you please re-review?

[bitglue]

Not sure what's up with buildkite.

[SuperQ]

The other end-to-end fixture needs to be updated. (collector/fixtures/e2e-64k-page-output.txt)

# TYPE node_qdisc_requeues_total counter
--
  | node_qdisc_requeues_total{device="eth0",kind="pfifo_fast"} 2
  | node_qdisc_requeues_total{device="wlan0",kind="fq"} 1
  | +# HELP node_schedstat_running_seconds_total Number of seconds CPU spent running a process.
  | +# TYPE node_schedstat_running_seconds_total counter
  | +node_schedstat_running_seconds_total{cpu="0"} 2.045936778163039e+13
  | +node_schedstat_running_seconds_total{cpu="1"} 1.904686152592476e+13
  | +# HELP node_schedstat_timeslices_total Number of timeslices executed by CPU.
  | +# TYPE node_schedstat_timeslices_total counter
  | +node_schedstat_timeslices_total{cpu="0"} 4.767485306e+09
  | +node_schedstat_timeslices_total{cpu="1"} 5.145567945e+09
  | +# HELP node_schedstat_waiting_seconds_total Number of seconds spent by processing waiting for this CPU.
  | +# TYPE node_schedstat_waiting_seconds_total counter
  | +node_schedstat_waiting_seconds_total{cpu="0"} 3.43796328169361e+12
  | +node_schedstat_waiting_seconds_total{cpu="1"} 3.64107263788241e+12
  | # HELP node_scrape_collector_duration_seconds node_exporter: Duration of a collector scrape.
  | # TYPE node_scrape_collector_duration_seconds gauge
  | # HELP node_scrape_collector_success node_exporter: Whether a collector succeeded.
  | @@ -2472,6 +2484,7 @@
  | node_scrape_collector_success{collector="pressure"} 1
  | node_scrape_collector_success{collector="processes"} 1
  | node_scrape_collector_success{collector="qdisc"} 1
  | +node_scrape_collector_success{collector="schedstat"} 1
  | node_scrape_collector_success{collector="sockstat"} 1
  | node_scrape_collector_success{collector="stat"} 1
  | node_scrape_collector_success{collector="textfile"} 1

[bitglue]

@SuperQ fixed! 🎉

[SuperQ]

Nice!

[SuperQ]

LGTM

[SuperQ]

Two final things.

Please add the new collector to the README.

Also, add it to the changelog.

* [FEATURE] Add new schedstat collector #1389 

[bitglue]

@SuperQ done.

[bitglue]

@SuperQ bump, any more to do here?

[SuperQ]

LGTM

[bitglue]

@SuperQ there's a bug in this, which I've fixed at prometheus/procfs#191. Please let's postpone any release until that's incorporated.