Realtime audio scheduling: RTKIT vs. RLIMIT_RTPRIO. Choose RLIMIT_RTPRIO, NOW!

[rerdavies]

Note: the following discussion does not apply to PiPedal, which grants the pipedald daemon RTLIMITS permissions at install time. However, it most certainly applies to anyone who's trying to use the TooB plugins on other platforms.

There are two systems for granting permissions to applications that want to do realt-itme audio scheduling on Linux.

The first, and increasingly, the default system is the RTKIT module. RTKIT is a DBus service that allows any application to request real-time audio scheduling, subject to strict limitations imposed by RTKIT that are designed to prevent rogue applications from completely locking up a Linux system using malicious attacks. By default, RTKIT allows any application to use the real-time scheduler; but the default settings for RTKIT are not suitable for pro-audio use (more on this later). RTKIT has been controversial ever since 2009 when it was first introduced, specifically because it is needlessly and senselessly unsuitable for use by pro audio applications.

The second system is the via the RLIMIT subsystem. RLIMIT typically allows only users who belong to a group (usually the audio group, but sometimes the realtime group) to use real-time scheduling. The RLIMIT subsystem sets resource limits for users (maximum number of open files, number of simultaneous logins, etc.). But among the limits it imposes are limits on real-time scheduling. The default settings for RLIMIT prohibit any type of real-time scheduling.

Both systems co-exist in a typical Linux install: RTKIT providing default real-time scheduling suitable for consumer audio by default; and the RLIMIT system completely disabling real-time scheduling by default. But enabling the RLIMIT system doesn't break RTKIT.

This article is intended to convince you that you need to enable the RLIMIT real-time priority systems, and then tell you how to enable it. If you want to skip the blather, you can cut to the chase. But you need to do this!

The intentions of RTKIT are noble, if misguided. The intention is to prevent malicious applications or plugins running without privilege from completely locking up a Linux system by "Fork Bombing" the system with processes that consume 100% of available CPU time with real-time threads that prevent the core OS from running. Probably the best feature of RTKIT is a watchdog system that detects when real-time threads are consuming excessive CPU time, and demotes offending threads to non-realtime priority. The problem with RTKIT is that imposes the following additional restrictions:

• It limits any single realtime thread from using more than 20% of a single CPU. That means that plugins in your audio rig can only use one fifth of load they could have otherwise used!
• It limits the number of threads that a single user or single process can run with realtime priority. The default limits can easily be exceeded if you are using convolution reverb, or cab IR effects.
• It limits the number of realtime threads that can be started within a 25 second period to 20. Which makes convolutinon reverb effects totally impractical.
• It limits the maximum priority of realtime threads to +20. One of the useful features of the real-time scheduling system is that realtime audio threads can be scheduled to run at higher priority than interrupt service routes -- specifically network and disk interrupt service routines that can easily cause overruns. But by limiting the maximum priority to +20, instead of +99, this benefit is denied. Pipewire, for example, will run its audio thread at a real-time priority that's higher than interrupt service routines if given the chance. Pipewire actually runs at priority +98 if given a chance, which might be a little over-eager; but at least it won't overrun needlessly.
• Threads and processes that exceed their allotted CPU time are summarily killed, without regard for the consequences of doing so.

While it's true that the default configuration can be changed, doing so is painfully difficult, especially compared to how easy it is to enable RTLIMITS real-time scheduling. (Click the link, I know you want to). The only way to change the default settings is by editing awkwardly documented command-line options into the systemd service file for RTKIT.

In the 14 years since its initial release, this remains the only way to change the default settings. The RTLIMITS system, on the other hand, provides not only the /etc/security/limits.conf file, full of relevant examples, but also the /etc/security/limits.d directory that allows packages to easily make changes to RTLIMITS settings and cleanly remove those changes when uninstalling, while providing a file with no clutter at all that's easily understandable by users who want to customize those settings.

Pipewire, Jack, ALSA and Pulse audio will use either system. They all try to get access to real-time scheduling through the RTLIMITS system, and if they can't, they fall back to the DBus-based RTKIT system, using the restrictions imposed by RTKIT. So all you have to do give your favorite audio stack room to breath is to enable RLIMITS real-time priority permissions. Doing so gives all of these systems five times more CPU time in which to do real-time audio! and substantially increases their ability to run sophisticated compute-intensive plugins that can't run at all under the RTKIT system.

And here's the kicker. Enabling RLIMITS real-time scheduling requires adding a single line to a single file! Pipewire, Jack, ALSA, Pulseaudio, Carla, Reaper, and a host of other audio applications will instantly run faster, with fewer overruns, and better latency.

Here's the change. Edit the file /etc/limits.conf and add the following line:

    yourusername    -     rtprio     98

That's it.

Traditionally, Linux distributions have granted real-time scheduling permissions by group membership. If you are really interested, there are some details here. But functionally, there's not a lot of difference between using a group-based solution, and just granting your user-id the permissions you need directly. Interestingly, the latest version of Mint omits the pre-installed audio group configuration of RLIMITS altogether; and I suspect that might become a trend among other Linux distributions as well in the future..

Things you should know... runaway or broken plugins will NOT lock up your system. The real-time scheduling system set aside a smidgen of CPU time so that processes running at normal priority can intervene and kill the runaway process, or allow you to press the Panic button. The one specific vulnerability is that a deliberately malicious plugin could freeze your system using "Fork Bombing". At which point you'll need to hit the reset button on your computer. But let's be reasonable about this A malicious application could bring a desktop system to its knees by "Fork Bombing" without using real-time scheduling, couldn't it? Or overcommitting memory. Or any of many simpler denial of service attacks. The "Fork Bombing" defense might be important on hardened server-class machines. But hardened server-class machines are going to disable RTKIT anyway.

All of which raises the really tough question: why have draconian default limits provided by RTKIT not been revised in all of the 14 years since its initial release? It's certainly not due to lack of complaints. And it's not because more reasonable defaults which would accommodate pro-audio applications can't be provided. Why?

In the meantime, just go with RLIMITS provisioning, and enjoy!

[nbraud]

Do you think giving PipeWire (and only it) real-timeliness by granting its executable file the SYS_NICE capability, be an effective middle-ground?

Not necessarily for all pro-audio usecases, which might have significant amounts of processing outside the PipeWire (or JACK) daemon, but as something we can ship in general-purpose systems without granting all unprivileged processes the ability to request real-timeliness.

[rerdavies]

I'm not sure what SYS_NICE is, to be perfectly honest. If you mean the ability to bump a processes priority using nice(), then it's a pointless compromise. A process with higher priority via nice() can effectively crash the system by consuming 100% of available CPU bandwidth just as effectively as a process with RT priority can. It doesn't matter whether the priority boost is one or two levels (via nice()), or 70 or 80 levels (via realtime scheduling).

And RT scheduling provides two advantages that threads with priority bumped with nice() don't get:

• the ability to run at higher priority than (say) disk or network adapter interrupt handlers (which run at an RT priority of about 50).
• The ability to prevent crashes by limiting the total CPU use by all RT tasks.

Tragically, RT_KIT gets so close to doing the right thing. The only fault is that it senselessly limits realtime tasks to 20% of total CPU use instead of 90% total CPU use.

As for processing outside of PipeWire or Jack... Jack actually provides a method that allows you to get RT priority without permission. You ask Jack to grant RT permissions to your process, and the Jack daemon has the right to set RT limits on your thread even though you don't. PipeWire (I assume) provides the same functionality when it's emulating Jack. Jack's actual audio loop runs in a thread in your process, and does so using RT permission grants that the Jack Process has applied from its process (where it has permissions to do that).

The problem then becomes that both Jack and Pipewire in default configuration use RT_KIT to gain realtime privileges, and therefore suffer from RT_KIT processing limits in default configuration. Jack and Pipewire do both check to see if they have been granted RLIMIT_RTPRIO permissions, and use that instead of RT_KIT if the system has been so configured. So both useless for realtime pro-audio in default configuration. Which means that any pro-audio package has to open up RTLIMIT_RTPRIO permissions when they are installed so that Jack or Pipewire can do the right thing.

Madness.

[38github]

Interesting!