[macOS] CPU utilization contains zeroes

[ArneTR]

Summary

• OS: macOS 13.6.3 (22G436)
• Architecture: ARM
• Psutil version: 5.9.8
• Python version: 3.10.13
• Type: core

Description

I searched for utilization and could not find anything that matches this, so I hope it is not a dup.

When using psutil.cpu_percent(0.1) on my macOS while running a constant stress in parallel (stress-ng -c 1) I get a lot of zeros in between.

The same behaviour is also when I increase the stress to more cores. Say 8

>>> while True: psutil.cpu_percent(0.1)
...
100.0
100.0
100.0
100.0
0.0
0.0
0.0
0.0
0.0

The same behaviour does not happen on linux machines.

[giampaolo]

What does htop or similar tools show? 0.1 is a small interval. Task manager tools usually use 1 sec instead. What happens if you use 1 sec?

[ArneTR]

htop works fine. You can just let it run with 100ms resolution and the many consecutive "0" values do not occur.

For clarification: These values do occuer always no matter what kind of load we are running. So even if all cores are fully stressed zeros creep in, which is probably due to the used syscall.

We have actually created working code for one our tools. Please find it here: https://github.com/green-coding-solutions/green-metrics-tool/blob/main/metric_providers/cpu/utilization/mach/system/source.c

We are using host_processor_info and not host_statistics.

See also discussion here: green-coding-solutions/green-metrics-tool#330

[ArneTR]

@ribalba FYI

[ribalba]

I wrote a blog article about the problem here:

https://www.green-coding.io/blog/cpu-utilization-mac/

[giampaolo]

Nice finding! It's kind of shocking that CPU metrics on macOS have been wrong for so long and nobody noticed.

[ArneTR]

To be fair, it is unknown exactly in which version of macOS this isse happens and if it also is on all chip.s

@ribalba and I both have M1 machines. Untested on Intel and M2.

We are happy to provide a PR and some tests for macOS that test this feature (Stressing a core and seeing if zeros still exist in output), but I see that this repo currently has no macOS Github Actions runner defined.

If we provide a PR, does this repo have the possiblity to have a macOS Runner running that can test the CPU Utilization?

[giampaolo]

We currently have CI runners for macOS x86-64 and arm (this one was added yesterday).

[giampaolo]

Looking back at this, I took your blog post as an example, and modified code to use host_processor_info (diff below). I then tried running cpu_percent in a loop like you shown in the first message, and I basically see the same behavior (recurrent zeroes). Apparently host_processor_info didn't make any difference (but my code may be wrong).

diff --git a/psutil/arch/osx/cpu.c b/psutil/arch/osx/cpu.c
index d2f8b185..bdeb4984 100644
--- a/psutil/arch/osx/cpu.c
+++ b/psutil/arch/osx/cpu.c
@@ -61,28 +61,34 @@ psutil_cpu_count_cores(PyObject *self, PyObject *args) {
 
 PyObject *
 psutil_cpu_times(PyObject *self, PyObject *args) {
-    mach_msg_type_number_t count = HOST_CPU_LOAD_INFO_COUNT;
     kern_return_t error;
-    host_cpu_load_info_data_t r_load;
+    processor_info_array_t cpuInfo = NULL;
+    mach_msg_type_number_t numCpuInfo;
+    natural_t numCPUsU = 0U;
+    float user, nice, system, idle = 0.0;
 
     mach_port_t host_port = mach_host_self();
-    error = host_statistics(host_port, HOST_CPU_LOAD_INFO,
-                            (host_info_t)&r_load, &count);
+    error = host_processor_info(
+        host_port, PROCESSOR_CPU_LOAD_INFO, &numCPUsU, &cpuInfo, &numCpuInfo
+    );
+
     if (error != KERN_SUCCESS) {
         return PyErr_Format(
             PyExc_RuntimeError,
-            "host_statistics(HOST_CPU_LOAD_INFO) syscall failed: %s",
+            "host_processor_info() syscall failed: %s",
             mach_error_string(error));
     }
+
+    for (unsigned i = 0; i < numCPUsU; ++i) {
+        user = cpuInfo[(CPU_STATE_MAX * i) + CPU_STATE_USER];
+        nice = cpuInfo[(CPU_STATE_MAX * i) + CPU_STATE_NICE];
+        system = cpuInfo[(CPU_STATE_MAX * i) + CPU_STATE_SYSTEM];
+        idle = cpuInfo[(CPU_STATE_MAX * i) + CPU_STATE_IDLE];
+    }
+
     mach_port_deallocate(mach_task_self(), host_port);
 
-    return Py_BuildValue(
-        "(dddd)",
-        (double)r_load.cpu_ticks[CPU_STATE_USER] / CLK_TCK,
-        (double)r_load.cpu_ticks[CPU_STATE_NICE] / CLK_TCK,
-        (double)r_load.cpu_ticks[CPU_STATE_SYSTEM] / CLK_TCK,
-        (double)r_load.cpu_ticks[CPU_STATE_IDLE] / CLK_TCK
-    );
+    return Py_BuildValue("(dddd)", user, nice, system, idle);
 }

[ArneTR]

I did not look into detail how you adapter your code, but if I check our codes and psutil I can still reproduce the problem.

See this screenshot of my terminal with running our current CPU utilization code (left) and psutil (right).
(Our code displays first a timestamp and the the utilization as ratio, which means *100. Please also dot not compare the exactness of the output, as I scrolled up my terminal buffer, so both scripts have not been started at the same time ... however same pattern persists. Left no zeros, right zeroes.)

Our code is from here: https://github.com/green-coding-solutions/green-metrics-tool/tree/main/metric_providers/cpu/utilization/mach/system

Just run make and then execute as in the screenshot (./metric-provider-binary -i 100)

Possible differences between what you see and what I see maybe:

• Which chip are you running on. M1 from 2021 ?
• Which OS are you running on: I have macOS 14.7.2 (23H311) (Sonoma)