VST fixes again

[DomClark]

Most remarks are in commit messages this time.

e5cc1a1 fixes #4479
4c1d617 fixes #4325
a4e1a0b fixes #4498
8e6ee6f fixes #4306 (see PR #4366)
3af9e26 fixes #4500

b9a8186 will fix window resizing when a plugin is not embedded, but won't work for embedded plugins (things like #4087, #4461); the machinery required to fix this is outside the scope of 1.2 I believe.

I think edf2365 fixes #4117, but it's not entirely clear to me what the point of that issue is; slow VSTs aren't necessarily our problem, but the "many CMD in taskmanager" part is addressed by this commit. @lukas-w requested on Discord that this be implemented differently on the master branch using add_executable.

[SecondFlight]

❤️

[PhysSong]

Great work! At first glance, I can't find any possible regressions.

24c403d, 993803d and 01cf0ee fix #4306 (see PR #4366)

I think you may squash and reauthor three COM/OLE commits before merging if you want.

@lukas-w requested on Discord that this be implemented differently on the master branch using add_executable.

It will/should be handled when syncing branches. I guess what @lukas-w means is to use target_link_libraries like this:

lmms/plugins/vst_base/RemoteVstPlugin/CMakeLists.txt

Line 42 in c44bc4e

target_link_libraries(${EXE_NAME} Qt5::Core)

[PhysSong]

the machinery required to fix this is outside the scope of 1.2 I believe.

@DomClark Could you explain that briefly?

[DomClark]

Could you explain that briefly?

Updating the size of an embedded plugin requires sending a message from the remote plugin back to the host LMMS instance. LMMS isn't currently well equipped to receive messages back from plugins - messages are only processed when the plugin is first started, or when waiting for reply messages. In the current state, the window resize message would most likely be picked up on the processing thread while waiting for processing to complete, which isn't desirable. Even then it would only be received if processing were being done, so if, say, a VST effect didn't have any input at the time and the "process effects without input" setting were turned off, the GUI might not resize itself until audio input came again, which is also obviously undesirable.
Periodically processing messages on the main thread is insufficient as a fix as it doesn't stop messages from being picked up elsewhere where they aren't wanted. One could periodically poll for size changes, sending an 'update me' message and waiting for a reply with a new size, but this would unnecessarily lock the plugin, potentially reducing performance and introducing another spot from which deadlock could arise.
The ideal solution here would be to redo the messaging system so that LMMS is always listening for messages from the remote plugin and dispatching them to the appropriate places. This way any window size updates could be applied as soon as possible, on the main thread, and without holding up processing of anything else. Other advantages here would be that multiple messages that require replies can be waited on in parallel - currently these lock the plugin from sending the initial message until the reply is received, otherwise two of these running at once could eat each other's reply messages and cause deadlock - meaning less delay on the processing thread. There wouldn't be any more problems with the main thread processing events, triggering a slot and trying to lock the plugin mutex twice as it wouldn't need to hold the mutex while processing events.
Such an implementation I think would take a while to write and perfect, and in the short term may cause more trouble than it's worth.
I realise this wasn't exactly brief as requested, and possibly a little over-optimistic about what could be done, but I hope it's clear enough.

[PhysSong]

The ideal solution here would be to redo the messaging system so that LMMS is always listening for messages from the remote plugin and dispatching them to the appropriate places.

It sounds good. I think we should separate real-time(audio processing) and non-real-time(UI, etc.) messages when/if we do so.

Here's just an idea:

• Create a watcher thread like ProcessWatcher which checks for messages from plugin periodically.
• Message consumers interact with the watcher instead of the plugin itself. They should be able to trigger checking for messages manually when needed.
• When received UI messages, post them using queued connections.
• In other cases, store them in queues.

[PhysSong]

Passing WIN32 into add_executable specifies the Windows subsystem. CMake will add -mwindows for MinGW and /subsystem:windows for MSVC.
MSVC looks for WinMain by default when /subsystem:windows is specified. /entry:mainCRTStartup fixes it, but I have no idea which approach is better.
@DomClark Please let me know what you prefer so I can handle it when syncing branches.

[PhysSong]

Merging since @DomClark said okay. I want to know @lukas-w's opinion on the MSVC stuff though.

[lukas-w]

I thought -mwindows behaves exactly the same as subsystem:windows, i.e. it removes the console window and changes the entry point to WinMain. Is this incorrect?

[PhysSong]

@lukas-w MinGW allows using main with -mwindows.

[lukas-w]

I didn't know that. Not sure what the right approach is here. Doesn't matter too much I guess.