frontend: Enforce Qt to have a DE-agnostic behavior on Linux

[tytan652]

Description

Enforce Qt to have a DE-agnostic behavior on Linux

I didn't have Plasma screenshot since I have no machine under it.

Before (with Qt 6.8 on GNOME):

After:

Motivation and Context

If the platform theme is not enforced, while Plasma Wayland enforce SSD decoration (which is fine-ish) a certain option if enabled is will lead to some menubar issues. Multiple users have met an issue where OBS had no menubar under Plasma.

Also there is the fact that with the KDE platform we can end up with the Plasma file chooser (not portal) but with OBS style which is not a good visual, xdgdesktopportal enforce the portal which avoid the style issue.

If the wayland decoration is not enforced, under GNOME, Qt itself (since 6.8) will try to enforce a GNOME-like decoration (Fedora already done the same with their own Qt plugins). DO NOT mix it up with GNOME X11 SSD decoration.

There is also third-party Qt plugins (e.g. qt5ct, qgnome) that caused their share of issues in the past, removing them from the list of potential culprit still can be useful.

I think this is not controversial to expect OBS Studio to look the same under any desktop environment.

How Has This Been Tested?

Ran locally OBS (outside of a Flatpak) and no issue were met using file chooser (use portals now rather the GTK 3 dialog) and the window decoration is Bradient.

Types of changes

• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[dimtpap]

I'm not seeing the effect on KDE Plasma
Before

After

[tytan652]

@dimtpap, seems like qt-wayland does something about CSD/SSD or KWin enforces it (I don't know if I can do something…).
At least this PR should prevent any issue related to our menubar and Plasma optional hamburger menu.

Edit: Unlike GTK, Qt has zero stable platform API for this kind of thing.

[robxnano]

On behalf of Gnome users, can I ask you to not change the title bars please? The generic CSDs in Qt are very bad and don't work properly on any desktop, which is why the Gnome-specific ones were created in the first place. Applications that don't use custom title bars have no reason to interfere with the ones provided by Qt.

[tytan652]

which is why the Gnome-specific ones were created in the first place

This decoration was only created and proposed to upstream for almost the same reason that FedoraQt made the qgnome platform plugin.
To force Qt apps CSD to look "like" GNOME apps, where GNOME real stance can be sum up to let apps draw what they what in their own windows.

Applications that don't use custom title bars have no reason to interfere with the ones provided by Qt.

In the GTK ecosystem, you can opt-in to use libadwaita (as a platform), and so your app will look like an GNOME app.
The same point should apply to the Qt ecosystem, so we opt-in to use the same CSD on any DE.
Qt is not a platform, KDE (Breeze, Kirigami) or MauiKit are.

We really had issue with third party Qt plugins being enforced on OBS in the past, this PR is also to prevent to happen again.

[robxnano]

In the GTK ecosystem, you can opt-in to use libadwaita (as a platform), and so your app will look like an GNOME app. The same point should apply to the Qt ecosystem, so we opt-in to use the same CSD on any DE. Qt is not a platform, KDE (Breeze, Kirigami) or MauiKit are.

That would make sense if you were actually using custom title bars like libadwaita applications do. But as it is now, you're just interfering with the upstream Qt choice of which decorations to use on which desktop. Whether window decorations are rendered by the compositor (as with Plasma Wayland and X11) or by the application (Gnome Wayland, Windows) is really just an implementation detail, which is why Qt doesn't present any actual API to change it and you have to resort to hacks like modifying environment variables.

We really had issue with third party Qt plugins being enforced on OBS in the past, this PR is also to prevent to happen again.

The Gnome window decoration plugin isn't third-party though, it's part of Qt and better supported than the generic ones. If anything, the Bradient decorations are more likely to cause issues in the future.

[kkartaltepe]

The generic CSDs in Qt are very bad and don't work properly on any desktop

It would probably be more helpful to list specific issues that you think OBS will be affected by. Preferably to provide screenshots of the issue and platform affected. Right now there are screenshots showing they appear to work exactly as intended on gnome and kde environments.

[robxnano]

Okay, here are some examples. Not all of these are obvious from the screenshots, some of them only become obvious when you're actually using the program with these title bars.

• No visible border on overlapping windows
• Active and inactive title bars are the same color, making it impossible to tell which window is the current focus
• Doesn't follow the setting for window button positions
• Window buttons don't respond to being pressed or hovered
• Double-clicking on title bar doesn't maximize the window
• Resize cursor still appears around the borders of the window when maximized (but clicking it doesn't do anything)
• Maximize button still works on fixed-size dialog boxes

I'd also like to point out that in any other Qt application, the window title at least changes color on inactive windows. So something your code is doing is actually breaking that, whereas the Gnome-style title bars are designed to avoid that by using their own color palettes. So ironically, the Bradient plugin is actually the one causing more issues for your application.

[tytan652]

• No visible border on overlapping windows

Likely the same on other DE.

• Active and inactive title bars are the same color, making it impossible to tell which window is the current focus

Same on other DE.

• Doesn't follow the setting for window button positions

There is no settings portal for that and relying on GNOME specific settings is a no.

• Window buttons don't respond to being pressed or hovered

Same on other DE.

• Double-clicking on title bar doesn't maximize the window

Same on other DE.

• Resize cursor still appears around the borders of the window when maximized (but clicking it doesn't do anything)

Likely the same on other DE.

• Maximize button still works on fixed-size dialog boxes

Same on other DE.

The point of the PR is having OBS to get the same behavior on all DE without exception.
Having a special behavior just for GNOME is a no.

[tytan652]

For testing the correct behavior on GNOME, you need to make sure that FedoraQt plugins are not installed at all so if you are using the Flatpak or using Fedora. Those can be happen to be installed and are not supported by OBS.

[robxnano]

Same on other DE.

Which "other DE" are you using to make this comparison? According to the Wayland protocol explorer, every other non-reference desktop compositor supports the XDG decoration protocol. So in fact, Gnome is the only desktop that uses the CSD decorations.

The point of the PR is having OBS to get the same behavior on all DE without exception. Having a special behavior just for GNOME is a no.

Why do you say that this change gets "the same behavior on all DE without exception", when every other desktop (including KDE, as you already admitted) still uses the SSD decorations? Unless you're going to force the use of the Bradient decorations on every desktop, this justification doesn't make sense.

[tytan652]

This is about having OBS (and mainly Qt) to behave the same on all DE and since unfortunately Qt enforce the use SSD when asked by the compositor (where we wished it does not).
Consider that the in a CSD context bradient should be the default one and it is a stop-gap since we can do nothing to avoid SSD and have CSD all the time for now.

And I repeat, enforcing those values also prevent undesired third-party to change Qt's behavior.

In the long term if we happen to be able to make sure that SSD is never applied, it will be done.