Intel CPU Frequency Scaling Broken

[sylentprofet]

Qubes OS version:

R4.0

Affected component(s):

intel_pstate
acpi-cpufreq
xenpm

---

Steps to reproduce the behavior:

Tested on:

• Lenovo T480s
• Lenovo X1 Carbon Gen. 6
• Huawei Matebook X Pro.

All with Intel i7-8550U.

Latest BIOS revisions for the respective systems as of Dec. 2018

Kernel: 4.19.2-3.pvops.qubes.x86_64.

EFI install.

In dom0, sudo xenpm get-cpufreq-para

Expected behavior:

The processor is rated at 1.8 GHz (4.0 turbo), so we would expect to see appropriate scaling in that range, available frequencies from 1800000 - 4000000.

Further, we would expect to see scaling_driver = intel_pstate.

Actual behavior:

The CPU frequencies do not scale correctly. Why?

Frequencies are pinned at 2 GHz max, 400 MHz min, across all cores.

# xenpm cpu-freq-para
...
cpu id               : 0
affected_cpus        : 0
cpuinfo frequency    : max [2001000] min [400000] cur [2001000]
scaling_driver       : acpi-cpufreq
scaling_avail_gov    : userspace performance powersave ondemand
current_governor     : ondemand
  ondemand specific  :
    sampling_rate    : max [10000000] min [10000] cur [20000]
    up_threshold     : 80
scaling_avail_freq   : 2001000 2000000 1900000 1800000 1700000 1500000 1400000 1300000 1200000 1100000 1000000 800000 700000 600000 500000 *400000
scaling frequency    : max [2001000] min [400000] cur [400000]
turbo mode           : enabled
...

Confirmed with watch -n1 "cat /proc/cpuinfo | grep \"[c]pu MHz\""

xenpm set-scaling-maxfreq and -minfreq have no effect.

xenpm get-cpufreq-states shows 16 total/usable P-states.

Changing the governor to performance has no effect. Default is ondemand

dmidecode reports a max of 2 GHz on the Lenovos, and an apparently erroneous speed on the Huawei (~ 8 GHz).

# dmidecode | grep -i speed
	Speed: 2400 MT/s
	Configured Clock Speed: 2400 MT/s
	Speed: 2400 MT/s
	Configured Clock Speed: 2400 MT/s
	Speed: Unknown
	Speed: Unknown
	Speed: Unknown
	Max Speed: 2000 MHz
	Current Speed: 1800 MHz

The scaling_driver is legacy acpi-cpufreq. Interestingly, intel_pstate can be seen initializing during boot, but it does not take over handling anything. Attempting to blacklist acpi-cpufreq in modprobe.d has no effect.

# dmesg | grep pstate
[    5.067624] intel_pstate: Intel P-state driver initializing

/sys/devices/system/cpu/intel_pstate/ contains the expected attributes, but as mentioned in the "related issue" linked below, no_turbo, num_pstates, and turbo_pct error Resource temporarily unavailable.

/sys/devices/system/cpu/intel_pstate/status always returns off, and does not respond to echo "active" >. This behavior has been tested with various kernel command line parameters, including intel_pstate=force, intel_pstate=disabled, intel_pstate=no_hwp, intel_pstate=enable with no change in performance aside from ../cpu/intel_pstate/ attributes disappearing when no_hwp or disabled were in effect. Also tried processor.ignore_ppc=1.

Strangely, none of the appropriate attributes for cpufreq exist in /sys/devices/system/cpu/cpu*/.

# ls /sys/devices/system/cpu/cpu0/
acpi_cppc  driver         hotplug  power      topology
cache      firmware_node  node0    subsystem  uevent

lsmod | grep cpufreq shows no results, trying to modprobe acpi-cpufreq or cpufreq-xen returns errors. xen_acpi_processor is loaded.

# modprobe acpi-cpufreq
modprobe: ERROR: could not insert 'acpi_cpufreq': No such device
# modprobe cpufreq-xen
modprobe: FATAL: Module cpufreq-xen not found in directory /lib/modules/4.19.2-3.pvops.qubes.x86_64

cpupower frequency-info is completely unresponsive, with zero information available about the processor.

analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  CPUs which run at the same hardware frequency: Not Available
  CPUs which need to have their frequency coordinated by software: Not Available
  maximum transition latency:  Cannot determine or is not supported.
Not Available
  available cpufreq governors: Not Available
  Unable to determine current policy
  current CPU frequency: Unable to call hardware
  current CPU frequency:  Unable to call to kernel
  boost state support:
    Supported: yes
    Active: yes

Though it shouldn't have any effect, testing was attempted with smt=on and off, and Hyperthreading enabled/disabled in the BIOS appropriately.

Testing was also performed while toggling various BIOS settings.

• enable/disable Intel SpeedStep
• power settings at Maximum Performance vs. Balanced

It does not appear to be a thermal throttling issue, with idle ~ 37C and under load ~60C observed consistently.

tlp was tested with no effect on the frequency scaling, regardless of being enabled or disabled. tlp-stat yields minimal additional info, with what seems to be an outdated recommendation for the Lenovos to install tp-smapi kernel modules, that are in fact deprecated in favor of thinkpad_acpi, which appears to be active on the Thinkpads.

dmesg | grep thinkpad
[   19.589434] thinkpad_acpi: ThinkPad ACPI Extras v0.26
[   19.589439] thinkpad_acpi: http://ibm-acpi.sf.net/
[   19.589440] thinkpad_acpi: ThinkPad BIOS N22ET50W (1.27 ), EC unknown
[   19.589441] thinkpad_acpi: Lenovo ThinkPad T480s, model 20L7CTO1WW
[   19.591883] thinkpad_acpi: radio switch found; radios are enabled
[   19.591898] thinkpad_acpi: This ThinkPad has standard ACPI backlight brightness control, supported by the ACPI video driver
[   19.591899] thinkpad_acpi: Disabling thinkpad-acpi brightness events by default...
[   19.612278] thinkpad_acpi: rfkill switch tpacpi_wwan_sw: radio is unblocked
[   19.643468] thinkpad_acpi: Standard ACPI backlight interface available, not loading native one
[   19.674512] thinkpad_acpi: battery 1 registered (start 0, stop 100)
[   19.674576] input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/

thermald is not loaded.

General notes:

https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html

• This link suggests removing irqbalance but I'm skeptical.
  https://askubuntu.com/questions/1067866/ubuntu-18-04-steam-games-frame-rate-drop/1073353#1073353?newreg=c7c120f373da4effb7317104571cd573

• https://cateee.net/lkddb/web-lkddb/XEN_ACPI_PROCESSOR.html
  Regarding xen_acpi_processor: "It also registers itself as the SMM so that other drivers (such as ACPI cpufreq scaling driver) will not load."

How could lsmod report xen_acpi_processor as loaded but xenpm shows the scaling driver acpi-cpufreq ? This might make sense as to the missing /sys/devices/.../cpufreq entries.

• The following exchange is dubious at best, the final post gets down to the point of disabling intel microcode. They also suggest the use of msr-tools, but that really shouldn't be necessary.
  https://bbs.archlinux.org/viewtopic.php?id=231077

• This is good work, but in my opinion, running a script every few seconds in dom0 isn't a legitimate fix.
  https://github.com/erpalma/lenovo-throttling-fix

---

Related issues:

#4491
#450

[marmarek]

As you can see in #4491, dom0 kernel have no full access to power management, including cpufreq, so some data may be inaccurate. This is also why acpi-cpufreq driver does not load, or why intel-pstate doesn't provide full info.

The thing you should look at is xenpm, especially this part:

scaling_avail_freq : 2001000 2000000 1900000 1800000 1700000 1500000 1400000 1300000 1200000 1100000 1000000 800000 700000 600000 500000 *400000
scaling frequency : max [2001000] min [400000] cur [400000]

As you can see, right now it's at 400MHz.

[sylentprofet]

So what's the solution here? Do I just assume that somehow I'm magically getting full performance, despite what every sensor tells me?

I spent most of yesterday looking at xenpm, and I can see that it isn't scaling correctly.

Should I try cpufreq=dom0-kernel?

Your response doesn't provide much of an explanation. According to these links, pstate should work. Or I should be able to see some attributes for cpufreq.

https://xenserver.org/partners/developing-products-for-xenserver/19-dev-help/138-xs-dev-perf-turbo.html

https://wiki.xenproject.org/wiki/Xen_power_management

https://lists.xenproject.org/archives/html/xen-devel/2015-10/msg03048.html

[marmarek]

I know terminology can be quite confusing here - there is acpi-cpufreq in both dom0 Linux and Xen. And the same for intel_pstate - there are two of them: one in dom0, one in Xen. By default, only Xen ones have real impact, even though dom0 one may report some data. For managing power management in Xen, user xenpm tool and do not believe in anything that dom0 kernel reports (/sys etc)...

The documentation you've linked is quite old (for example mentions Xen 4.0 as the latest version, which was released in 2010...). But I'm not sure if anything more up to date exists.

As for the xenpm get-cpufreq-para output - there are two set of parameters:

• "cpuinfo frequency" - info about what processor can, not necessary what are the current parameters - basically what you can set with xenpm set-scaling-*
• "scaling frequency" - the current parameters, including current frequency

There is also scaling_governor and its parameters.

As for the frequency, xenpm get-cpufreq-states output is IMO less confusing. You may also want to look at xenpm get-cpuidle-states, including statistics there.

That said, it may be the case that frequency scaling doen't work properly on this machine(s). We've seen other power-management related issues with Intel Core 8th gen CPUs, specifically around system suspend. We have limited access to such machines (one of them), but there should be some progress soon.

[sylentprofet]

I understand that the documentation is quite old - there's not a lot out there regarding this issue. I literally went through ~ 100 links yesterday trying to distill out good information.

It may be the case that xenpm is reporting the correct frequencies in realtime (i.e. it is running at 400 MHz, 2GHz, etc.), but my point is that the pstate settings themselves are incorrect (i.e. it should be able to scale to 4 GHz and it does not).

As for xenpm get-cpufreq-states it reports much the same information. All cores report similar to below, I truncated the output for readability. I was reluctant to copy it because it was somewhat redundant, but for posterity:

...
cpu id               : 7
total P-states       : 16
usable P-states      : 13
current frequency    : 400 MHz
P0         [2001 MHz]: transition [                   0]
                       residency  [                  12 ms]
P1         [2000 MHz]: transition [                   0]
                       residency  [                   0 ms]
P2         [1900 MHz]: transition [                   0]
                       residency  [                   0 ms]
P3         [1800 MHz]: transition [                1113]
                       residency  [               21579 ms]
P4         [1700 MHz]: transition [                  61]
                       residency  [                 351 ms]
P5         [1500 MHz]: transition [                  37]
                       residency  [                 275 ms]
P6         [1400 MHz]: transition [                  29]
                       residency  [                 157 ms]
P7         [1300 MHz]: transition [                  43]
                       residency  [                 206 ms]
P8         [1200 MHz]: transition [                  51]
                       residency  [                 253 ms]
P9         [1100 MHz]: transition [                  44]
                       residency  [                 189 ms]
P10        [1000 MHz]: transition [                 114]
                       residency  [                 572 ms]
P11        [ 800 MHz]: transition [                  64]
                       residency  [                 328 ms]
P12        [ 700 MHz]: transition [                  75]
                       residency  [                 350 ms]
P13        [ 600 MHz]: transition [                  79]
                       residency  [                 254 ms]
P14        [ 500 MHz]: transition [                 100]
                       residency  [                 377 ms]
*P15       [ 400 MHz]: transition [                 881]
                       residency  [               17952 ms]

xenpm get-cpuidle-states:

cpu id               : 7
total C-states       : 8
idle time(ms)        : 682402
C0                   : transition [             1591933]
                       residency  [               66243 ms]
C1                   : transition [             1303701]
                       residency  [               23089 ms]
C2                   : transition [              131054]
                       residency  [               59419 ms]
C3                   : transition [               14775]
                       residency  [               16809 ms]
C4                   : transition [               33119]
                       residency  [               51541 ms]
C5                   : transition [               22137]
                       residency  [               53372 ms]
C6                   : transition [               61415]
                       residency  [              283879 ms]
C7                   : transition [               25732]
                       residency  [              180918 ms]

Mentioned in the OP, xenpm set-scaling-frequency does not allow us to reach the correct frequencies. The governor can be set, but not much else beyond that of value.

On a side note, I just booted a USB key of Xubuntu 18.04 and experienced flawless performance. The scaling governor was correctly reported as intel_pstate, and I saw frequencies from 400 MHz - 4 GHz. Interestingly, the number of pstates is reported at 37.

Has the scaling_driver been purposefully forced to acpi-cpufreq instead of intel_pstate in Xen?

[marmarek]

Has the scaling_driver been purposefully forced to acpi-cpufreq instead of intel_pstate in Xen?

That's interesting question. When I look at the Xen version we currently use, with a grep and sources in hand, I can't see intel_pstate driver included there at all. To be hones, I can't see it in xen-unstable either. Looks like the patches adding intel_pstate driver to Xen were never committed.

[sylentprofet]

For reference, a few additional links we discovered:

https://lists.gt.net/xen/devel/376881
https://github.com/mirage/xen/commits?author=wei-w-wang-intel
https://lists.xenproject.org/archives/html/xen-devel/2015-10/msg03047.html

These highlight the intel_pstate xen patch set from Wei Wang @ intel. It sounds like the correct implementation should be adding intel_pstate=enable in the kernel cmd line, once the patches are merged.

Thank you for helping track this down!

[ghost]

I hadn't really looked into this, I just assumed that it was working just fine. Thank you for taking the time to look into this and sharing your findings @sylentprofet

[sylentprofet]

I wonder if @wei-w-wang-intel would have any input here? If it's the same person from the commits..

Perhaps @wei-w-wang ?

[sylentprofet]

The plot thickens.. My colleague ran across the following Citrix documentation:

https://support.citrix.com/article/CTX200390?_ga=1.158728573.690652182.1439902385

Using the command from that, xenpm start 1|grep "Avg freq" appears to report correct frequencies. With this command, and only this command, are we seeing frequencies above the 2 Ghz range under load, in the 3 to 3.5 GHz range.

Strangely, if we simultaneously issue xenpm get-cpufreq-states , it continues to report incorrect information (2 Ghz max frequency) while xenpm start... reports frequencies above 3 GHz.

I have been unable to achieve max turbo state (4 GHz) with limited testing, but this is an interesting development nonetheless.

Any thoughts as to what might cause xenpm to behave in such a way? Which readout is correct? And what frequencies do the VMs actually have access to?

[marmarek]

Have you tried xenpm enable-turbo-mode (even though it's already reported as enabled)?

BTW I've asked about intel_pstate patches here: https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg01400.html

[dylangerdaly]

Matebook X Pro user here, it seems setting anything with xenpm has no affect.

[ddaly@dom0 ~]$ sudo xenpm disable-turbo-mode
[CPU1] failed to disable turbo mode (22 - Invalid argument)
[CPU3] failed to disable turbo mode (22 - Invalid argument)
[ddaly@dom0 ~]$ sudo xenpm set-scaling-minfreq 400000
[CPU1] failed to set scaling min freq (22 - Invalid argument)
[CPU3] failed to set scaling min freq (22 - Invalid argument)
[ddaly@dom0 ~]$ sudo xenpm set-scaling-maxfreq 400000
[CPU1] failed to set scaling max freq (22 - Invalid argument)
[CPU3] failed to set scaling max freq (22 - Invalid argument)
[ddaly@dom0 ~]$ sudo xenpm set-scaling-governor powersave
[CPU1] failed to set governor name (22 - Invalid argument)
[CPU3] failed to set governor name (22 - Invalid argument)
[ddaly@dom0 ~]$ # Forcing 400Mhz, on powersave
[ddaly@dom0 ~]$ xenpm start 1|grep Avg freq
  Avg freq2421210KHz
  Avg freq2421210KHz
  Avg freq2381190KHz
  Avg freq2381190KHz
[ddaly@dom0 ~]$ # Ignored.
[ddaly@dom0 ~]$ sudo xenpm get-cpufreq-para
cpu id               : 0
affected_cpus        : 0
cpuinfo frequency    : max [2001000] min [400000] cur [2001000]
scaling_driver       : acpi-cpufreq
scaling_avail_gov    : userspace performance powersave ondemand
current_governor     : powersave
scaling_avail_freq   : 2001000 2000000 1900000 1800000 1700000 1500000 1400000 1300000 1200000 1100000 1000000 800000 700000 600000 500000 *400000
scaling frequency    : max [400000] min [400000] cur [400000]
turbo mode           : disabled or n/a

[CPU1] failed to get cpufreq parameter
cpu id               : 2
affected_cpus        : 2
cpuinfo frequency    : max [2001000] min [400000] cur [2001000]
scaling_driver       : acpi-cpufreq
scaling_avail_gov    : userspace performance powersave ondemand
current_governor     : powersave
scaling_avail_freq   : 2001000 2000000 1900000 1800000 1700000 1500000 1400000 1300000 1200000 1100000 1000000 800000 700000 600000 500000 *400000
scaling frequency    : max [400000] min [400000] cur [400000]
turbo mode           : disabled or n/a

I've been able to achieve close to boost, but never anything close to 4Ghz (Boost)

sudo xenpm get-cpufreq-para dosen't know what it's doing.

[ghost]

Under high load on a Lenovo X1C6:

dom0 $ xenpm start 1 | grep "Avg freq"
Avg Freq 3844830 KHz
Avg Freq 3844830 KHz
Avg Freq 3844830 KHz
Avg Freq 3844830 KHz

these are the largest set of values that were returned

At idle, xenpm start 1 | grep "Avg freq" was not observed to be below 1995950 KHz

[evilaliv3]

Is there any update on this? On my Lenovo T480 it seems the battery is lasting very short, that make me think the scaling could be really buggy. thank you so much!

for any useful debugging info just ask

[dexinthecity]

I would like to get an update on this as well. It would be nice if we can are able to undervolt the cpu as well to prevent overheating.

[lunarthegrey]

According to the last response on the mailing list Wei looks reluctant to work on the patch any further. https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg01548.html

Not sure what we should do here. Keep requesting?

[tasket]

I don't know how specific any of this is to certain CPU generations, but on Intel pre-Skylake models scaling seems to work correctly except that xenpm readouts are a little odd.

On a nominal 2.6GHz/3.3GHz turbo CPU, for example, if turbo mode is disabled then the highest speed shown by xenpm is 2600000. But if turbo is enabled it shows up as 2601000. This is reflected in xenpv get-cpufreq-para output. I always assumed xenpm used that +1000 to denote turbo frequencies without showing the actual frequency, and that the CPU handled frequencies internally when turbo was engaged.

[evilaliv3]

Did anyone found a solution? thank you for sharing.

If there is anything you need to test get back to me. thank you!

[afilini]

Same issue here, if anyone need to test something I can try to help!

[Cairo555]

I'm experiencing the exact same issue on XenServer 7.6 running on an HP DL 380 Gen10.
When I boot the server using a live cd for CPU bench-marking. It shows the maximum allowed turbo speed of 3.7 Ghz on my Intel Xeon Gold 6132 Processors.

Same results with this command xenpm start 1|grep "Avg freq" as other people in this thread.
See max "boost" to 3.2 instead of 3.7 Ghz. I believe due to a lack of c-states.

I tested using 0 VM's on the host, tried to pin vCPU to a couple of pCPU so I would be assured that the other pCPU's would go to sleep (c states above 3) but nothing works.

I think it has to do with the driver.
If we can find a way of getting Xen to better understand the p-states we should be able to get to higher C-states. And therefore be able to boost.

Our Windows and Linux VM's only show the base frequency of 2.6 Ghz, It's just static.
Our use-case is the fact that we are running many single threaded applications on our hosts, which I therefore want to run on 3.7 Ghz whenever possible.

[brendanhoar]

Hmm, xenpm seems pull four values on my four core system...but the odd numbered values are the same as the preceding even numbered values on R4.01 current-testing. Without the grep, you also see that much of the details of the odd numbered cores are missing.

Perhaps it is assuming that hyperthreading is turned on (it is not) and is confused and really only reporting on half of the cores?

EDIT: ah, known to have bugs: #4456

Also...seven(???) (virtual?) cores listed in --get-topology:

[admin@dom0 ~]$ xenpm get-cpu-topology
CPU	core	socket	node
CPU0	 0	 0	 0
CPU1	 0	 0	 0
CPU2	 1	 0	 0
CPU3	 1	 0	 0
CPU4	 2	 0	 0
CPU5	 2	 0	 0
CPU6	 3	 0	 0
[admin@dom0 ~]$ 

B

[Cairo555]

I think I've got it figured out for my environment..

Xen reports CPUspeed in an odd way, the basefrequencie+1Mhz.
If I test the CPU on Xen, using xenpm start 1|grep "Avg freq" I can see it boost to 3.2 Ghz.
I believe it's not going to 3.7 Ghz, because the CPU is running AVX2 instructions as well.

The CPU speed that Intel uses to advertise with is only for NON (Intel) AVX code. More and more applications are running AVX2 instructions. And AVX2 has a much lower base core speed and much lower turbo speed.
See this Intel document for more details: https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html

Take a look at page 14 and 15 of the above document.
That will show you the graphs.

I have been told that AMD works in a different way, and those not downscale the cores. I'm afraid I can't test this, because I don't have the hardware.

Hope this helps.

[marmarek]

As for xenpm reporting and hyperthreading being disabled, this is most likely the case. In many cases Xen tools confuses "number of CPUs" with "maximum CPU ID".
There were multiple such issues, but not all are suitable for backporting to Xen 4.8.x.

[lunarthegrey]

I have been able to test this issue against a AMD Ryzen 7 Pro 3700U and it suffers from the same problem unfortunately. Doesn't seem to only be an Intel problem. I'm using R4.0 with kernel-latest (5.4.10-1) and linux-firmware (20200122-102.1).

xenpm get-cpufreq-para reports the minimum clock speed is 1400 MHz and the max is 2300 MHz which is the base clock. It should have a max turbo clock of 4000 MHz. https://www.amd.com/en/products/apu/amd-ryzen-7-pro-3700u

[k0l0ss]

Hello,

This is on my Lenovo Thinkpad x390 with an i7 8665u

/proc/cpuinfo reports 2100 static (which is base for this cpu)
dmidecode -t processor reports base of 2100 current 1900 static aswell

xenpm get-cpufreq-states reports wrong freq as stated by many of you before. seeing 2100 on load.

Tried with xenpm start 1 and it does show higher boost, up to 4.2ghz (still far from its 4.8 max) but I can live with it.

I am a bit concern about battery duration...
On idle cpu chills around 800mhz with 37-40º but still I am seeing lower battery duration than ubuntu (which i tested with live usb).

I know its not big trouble but is there any update/plan to find out about this?

I know it is off topic, but I can't find much around. Any of you have been able to undervolt? would be nice to know.
I tried with undervolt and I am getting ERROR:root:Failed to apply core: set -99.609375, read 0.0.
I've read somewhere it is an issue with new bioses.
Can anyone confirm?

Thank you for QubesOS! :D

[jandryuk]

Hi, @n1m1. Sure, the weekend would be fine. Thanks! You'll have to add cpufreq=xen:hwp to Xen's command line enable HWP for testing. We want to see if you can boot successfully. If so, please run xenpm get-cpufreq-para to ensure hwp is use. Thanks again!

[Rot127]

In case someone wants to enable this again and needs clear instructions:

Edit the grub config:

sudo nano /etc/default/grub 

Add cpufreq=xen:hwp,verbose to the parameter list of GRUB_CMDLINE_XEN_DEFAULT. Save and close.

Update grub config:

# For legacy BIOS
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# For UEFI BIOS
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grubx64.cfg

Reboot

[jandryuk]

Hi, @n1m1, If you could please try out the modified kernel and hypervisor, that would really help determine if the approach will work. Thanks!

[CrsiX]

I just stumbled upon this, because my CPU wouldn't go higher than 1.6 GHz even though 3.8 GHz (turbo) would be possible. I suspect that a recent Xen update (I can't really determine which, tough), might have broken something™, because it worked before.

In case someone wants to enable this again and needs clear instructions

So, I did this and set cpufreq=xen:performance,verbose instead based on docs. Now I have 2.8 GHz all-time. May be bad for battery, but I'm fine with that :)
Just for people who may stumble upon this as well.

[ra--]

Some brief feedback on this issue, which was the most pressing one with Qubes to me for some years now:
After updating the few packages in dom0 from the 4.1 current-testing repo (on a machine with Intel Core i7 Kaby Lake CPU) and adding cpufreq=xen:hwp,verbose as Xen parameter to Grub, I can see hwp being used for frequency scaling (checked with xenpm getcpufreq-para). Cannot say anything yet with regard to battery life and temperature, but it looks good so far. Thanks to everyone involved! 🙏

[n1m1]

Hi, @n1m1, If you could please try out the modified kernel and hypervisor, that would really help determine if the approach will work. Thanks!

Dear @jandryuk , I am terribly sorry for the unacceptable delay.

I tried to install the new kernel (+modules) you suggested, but for some reasons my laptop does not even boot when hwp is enabled. Moreover, I was not able to the xen hypervisors packages, as they create conflicts.

If there is something else you want me to test, over the next few days I should be kind of free.

Thanks and apologies again for the delay in my reply.

[jandryuk]

Hi, @n1m1

No worries. Thanks for trying.

Would you please tell me the Linux and Xen versions you successfully boot? uname -r should give the Linux version and xl info | grep xen_version for Xen.

Xen probably didn't install because of a version mismatch - I built 4.14 and 4.17 since I wasn't sure which you were using. For the next try, I think I'll just give you a hypervisor binary to drop in /boot. That'll avoid messing with your existing install and should be good enough for a test.

Thanks!

[n1m1]

Linux 6.2.10-1
Xen 4.14.5-20

Both from latest repo, both without hwp.

[jandryuk]

Hi, @n1m1 , please test the following:

xen-4.14.5-hwp.efi
https://drive.google.com/file/d/1XZDhed94mFYjAWkcuLzKaktfpNnk8RrU/view?usp=share_link

xen-4.14.5-hwp.gz
https://drive.google.com/file/d/1GX2U6bkffiH4WWJJg0EL6ZkCCKll9nbS/view?usp=share_link

kernel-latest-6.2.10-1.qubeshwp2.fc32.x86_64.rpm
https://drive.google.com/file/d/1njkNFJKNvX714-Fu5eeDdzNwra598Mo1/view?usp=share_link

kernel-latest-modules-6.2.10-1.qubeshwp2.fc32.x86_64.rpm
https://drive.google.com/file/d/1jOefkW8I3Nuc0BR17zBLTcgnFQIb_Vtg/view?usp=share_link

In dom0, copy xen-4.14.5-hwp.gz /boot and copy xen-4.14.5-hwp.efi to /boot/efi/EFI/qubes/.

Then sudo rpm -i kernel-latest-6.2.10-1.qubeshwp2.fc32.x86_64.rpm kernel-latest-modules-6.2.10-1.qubeshwp2.fc32.x86_64.rpm to install the kernel.

The kernel installation will run grub2-mkconfig, so you will get xen 4.14.5-hwp entries in grub. Modify an HWP one's command line to add loglvl=all cpufreq=xen:hwp,verbose and try booting. Fingers crossed it works. If it boots, please check xl dmesg to verify HWP was enabled.

Note, this is just the hypervisor binaries, so xenpm will not have HWP support in dom0. If it works, we can figured how to get full support integrated. Thanks for testing.

[jandryuk]

Code here:
https://github.com/jandryuk/qubes-linux-kernel/tree/hwp-test-2
and here:
https://github.com/jandryuk/qubes-vmm-xen/tree/xen-4.14-hwp-2

[n1m1]

Hi there,
thanks for this.

It worked. My machine booted with HWP enabled and working. I did not install the EFI binary as I am not using UEFI.

[jandryuk]

Great! Thanks, @n1m1! Also, I realized that `xenpm` does support HWP since it's built into qubes, but runtime-disabled.

…

On Sun, Apr 23, 2023, 6:37 AM n1m1 ***@***.***> wrote: Hi there, thanks for this. It worked. My machine booted with HWP enabled and working. I did not install the EFI binary as I am not using UEFI. — Reply to this email directly, view it on GitHub <#4604 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOKZP532MMGKXB2TQZP7HLXCUA6PANCNFSM4GKCH7NQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[arkenoi]

will this go to the default distro? it is weird to run the cpu constantly underclocked :/

[marmarek]

will this go to the default distro?

At some point, likely yes, but it needs to not regress on systems people already use. But note all the patches are included by default already, it just isn't enabled, but you can simply enable it on your system by adding cpufreq=xen:hwp to Xen options in grub config.

[UndeadDevel]

Maybe relevant: my NV41 ran with Turbo Boost not working before I started using cpufreq=xen:hwp, which has a pretty serious performance impact...I'm guessing most people will not be reading the forum or github as diligently as me, so maybe this should be put into a more prominent place? Perhaps even on the announcement page of NV41 certification unless the NovaCustom variant is not affected...

[PetrVladimirov]

Tested with with i5-6300U. cpufreq=xen:hwp works but with noticeable performance impact. Typing in all VMs including Dom0 becomes slower (still functional though). Probably @marmarek may suggest whether we can safely play with hwp variables

TLDR in terms of performance: xen:hwp < acpi-cpufreq (default) < xen:performance

Proposal 1
There are too many various combinations of computers & use-cases to cover them with one default one-size-fits-all setting.

What if we put this up to the user to decide which scaling_driver to use by providing them a drop-down setting in Qubes Global Setting? IMO this is quite important as can unleash performance, improve power management and user experience.

Proposal 2
It might also help, at least for this group, to use a standard unified set of command outputs and tests, so everyone compare "apples" to "apples".

xenpm get-cpufreq-para
cpu id               : 0
affected_cpus        : 0
cpuinfo frequency    : base [2500000] turbo [3000000]
scaling_driver       : hwp-cpufreq
scaling_avail_gov    : hwp-internal
current_governor     : hwp-internal
hwp variables        :
  hardware limits    : lowest [1] most_efficient [4]
                     : guaranteed [24] highest [30]
  configured limits  : min [1] max [255] energy_perf [128]
                     : activity_window [0 hardware selected]
                     : desired [0 hw autonomous]
turbo mode           : enabled

[CPU1] failed to get cpufreq parameter
cpu id               : 2
affected_cpus        : 2
cpuinfo frequency    : base [2500000] turbo [3000000]
scaling_driver       : hwp-cpufreq
scaling_avail_gov    : hwp-internal
current_governor     : hwp-internal
hwp variables        :
  hardware limits    : lowest [1] most_efficient [4]
                     : guaranteed [24] highest [30]
  configured limits  : min [1] max [255] energy_perf [128]
                     : activity_window [0 hardware selected]
                     : desired [0 hw autonomous]
turbo mode           : enabled

[CPU3] failed to get cpufreq parameter

Interestingly enough that using commands like xenpm start 1|grep "Avg freq" still show relatively decent performance (e.g. avg above 2500000 KHz)...

[jandryuk]

Hi. I wrote the Xen hwp driver to try and improve power savings on Intel laptops. It showed some benefit on lightly loaded systems. I didn't focus on top-line performance - I assumed under load, the CPU would max out. I did observed turbo boost hitting near maximum with xenpm start 1 before it throttled down as expected.

Are you measuring performance in some specific way, or is it just typing performance?

You can try setting options with xenpm set-cpufreq-hwp (It'll be xenpm set-cpufreq-cppc` in Xen 4.18).

If you want to favor performance, you can try something like:
xenpm set-cpufreq-hwp energy-perf:64
Or try other values:

energy-perf:N (0-255)
             energy/performance hint
             lower - favor performance
             higher - favor powersave
             128 - balance

[PetrVladimirov]

@jandryuk, thank you for developing this driver and your valuable suggestions!

Are you measuring performance in some specific way, or is it just typing performance?

Subjective evaluation: just typing performance. Would be glad to test via other means.

xenpm set-cpufreq-hwp energy-perf:64

I played a bit with this options, here are some outcomes (still rather subjective though):

• with 32 and 64 and smt=on the typing was improved to the level that hardly distinguishable from xen:performance
• Regardless of the options it seems that under non-interrupted workloads (e.g. booting VMs, processing something) it keeps the clock at a high frequency, thus showing decent performance
• At the same time when the system is idle it shows the drop to 800/900 MHz even with energy-perf:32 (xenpm start 1|grep "Avg freq") that is logical and obviously good for energy-consumption, but maybe prevent the system from awaking quickly (?) or handling well certain type of processing activities like typing (?).

If you can recommend some means to test the performance, I would be glad to do it. I can do some standard tests via sysbench in Dom0/AppVM, but I don't expect much different as this is usually a steady workload... We may need something special to test low-demanding CPU activities.. E.g. the frequency of "awakening/sleeping" cycles? Time to reach the full performance? Any thoughts?

[jandryuk]

@PetrVladimirov are you typing through a laptop keyboard, or through a USB keyboard with sys-usb? USB and sys-usb have more pieces to get the key strokes into their destination Qube. Is it consistently laggy, or does it lag initially and then seem to be okay while you continue to type? That would point to a delay in scaling up the processor frequency.

You could try playing with the activity window:

act-window:N{,m,u}s range 1us-1270s
    window for internal calculations.
    units default to \"us\" if unspecified.
    truncates un-representable values.
    0 lets the hardware decide.

My understanding is that it is a moving window to track load. If you make it smaller, it should scale up faster. It defaults to 0, letting the HW decide, but I don't know what is used in that case.

Maybe try something like:
xenpm set-cpufreq-hwp balance act-window:100ms

You can try other activity window values too.

I don't have any particular recommendations to test performance - I was curious how you were doing it. Subjective typing is fine.

Also, with smt=off, you only have 2 cores, correct? With a USB Keyboard, the keystrokes have to go sys-usb -> dom0 -> Qube. At least one of those VMs needs to get scheduled in by Xen before it can perform it can handle a keystroke. The Qube also needs to render the keystroke before dom0 can display, which would be be another potential context switch. But if smt=off with acpi-cpufreq works well enough, we'd want to match that.

[rwiesbach]

As suggested by apparatus on the forum I'd like to add my observation on Lenovo T490:

With Qubes 4.1 after 3h of Idle I have about 60% battery remaining, with Qubes 4.2 and Xen tweaked to disable hwp its down to about 40%. hwp needs to be set because of #8825

[jandryuk]

Hi @rwiesbach in #8825 (comment) @marmarek's suggestion disables hwp. Can you confirm that status of hwp in 4.2? xenpm get-cpufreq-para will mention hwp if it's enabled.

[rwiesbach]

@jandryuk Sorry, I refrased the sentence - it was misleading before. Q4.2 test was done with hwp disabkled.

[andrewdavidwong]

@r3t4k3r: Please note that this issue tracker (qubes-issues) is not intended to serve as a help desk or tech support center. Instead, we've set up other venues where you can ask for help and support, ask questions, and have discussions. (By contrast, the issue tracker is more of a technical tool intended to support our developers in their work.) Thank you for your understanding.