PREEMPT_RT kernel 6.6.35-rt34 latency under load vs stock kernel

[kdoren]

Here are latency measurements for PEEMPT_RT kernel 6.6.35-rt34-rpi-2712, made on Raspberry Pi 5.

Latency was measured over 100M events and over 5 hours.

sudo ./cyclictest -l100000000 -m -Sp91 -i200 -h400 -q >$OUTPUTFILE

Latency was measured while running a stress test, including USB SSD I/O:

stress-ng --hdd 4 --io 4 --vm 4 --cpu 4 --timeout 6h

Here is the PREEMPT_RT kernel, with max latency of 139 uSec:

Here is the stock PREEMPT (low-latency) kernel 6.6.31+rpt-rpi-2712, which has much worse latency.
Max latency is 1945 uSec. You can't see the graph falloff because I was only binning to 400 uSec:

[git-aniket]

Hello, thank you so much for this project.
I had a question regarding the response times of my pi5 after booting into it with this realtime kernel.

I see that after ssh ing into the pi from my pc, the responses from the pi towards wireless keyboard inputs is quite lagging. Is this normal behaviour in this realtime kernel? Please let me know. I was wondering if something went wrong on my end or not.

[git-aniket]

Ah. Okay thank you for the info. Will wait for the new builds :)

Kind regards
Aniket

[kdoren]

@git-aniket I just posted a release for kernel version 6.6.50 (they published the patch set for it today). I tested it on a Pi5, and WiFi works fine for me. Please try it and post your results.

Thanks,
Kevin

[git-aniket]

Thanks for the new builds. I tried the install instructions on the page
https://github.com/kdoren/linux/wiki/Installation-of-kernel-from-deb-package-%28Raspberry-Pi-OS%29

This time, the device gets stuck in a boot loop. I guess this is because I tried with the instructions on the page, which I think are out of date. The current default kernel on the Raspberry Pi OS is 6.6.47(from what I remember before the thing got stuck in the loop). Could you also please update the instructions on the page?

Kind regards
Aniket

[kdoren]

@git-aniket

Try editing and inspecting the file /boot/firmware/config.txt

When you use my instructions a second time to install another kernel, you'll have 2 sections at the bottom, one for each kernel. This allows you to select which kernel is used to boot.

Above each section, set to [all] (use for boot) or [none] (don't use).

Only one section should be set to [all]. I'm not sure what happens if 2 sections are set to [all], but that might be causing the problem.

Or you can just delete the section for rt kernel 6.6.35.

If all sections are set to [none], it should boot to stock kernel.

Also check that the second section looks correct, and is for kernel 6.6.50. I just updated the wiki, but the value for KERN in the wiki was meant to be replaced with the value from the release notes.

Also check that there is a directory under /boot/firmware for 6.6.50 kernel, and that it isn't empty.

Good luck,
Kevin

[git-aniket]

okay.. finally it worked.. Thanks a lot again for your help and prompt responses.

Kind regards
Aniket