Fix some encoding issues (mostly on Windows) #4401

PhysSong · 2018-06-04T05:48:04Z

This pull request fixes several encoding issue. Every issue except STK one are Windows-specific.

ZynAddSubFX setting loading/saving
VST DLL loading
VST setting loading/saving
Sample file decoding
MIDI import
STK rawwave path (incomplete on Windows due to upstream issue)

Issues remaining on Windows:

.pat file loading
.gig file loading
STK rawwave path (which toLocal8Bit fails to handle)

I'm planning to track remaining issues in #1191.

Fixes #3855, Fixes #2662, and almost #1191.

zonkmachine · 2018-06-10T15:37:19Z

Every issue except STK one are Windows-specific

I replicated issue #2662 on LinuxMint/Qt4 . It's indeed fixed by this PR.

Ready to merge?

Wallacoloo · 2018-06-10T21:25:44Z

plugins/MidiImport/MidiImport.cpp

+	if (!file.open(QIODevice::ReadOnly))
+	{
+		return false;
+	}


This looks like you're closing and then reopening the file. Is it possible to just not close the file to begin with, and then replace QByteArray arr = file.readAll(); with QByteArray arr = file().readAll(); further down?

I wrote like this because of the const-ness of file(). I'll add a method to work around.

Wallacoloo · 2018-06-10T21:45:25Z

plugins/vst_base/RemoteVstPlugin.cpp

+static std::wstring toWString(std::string s)
+{
+	/*
+	QString qstr = QString::fromUtf8(s.data(), int(s.size()));


I see this is commented out, so I'm guessing it doesn't work. Did you try

return qstr.toStdWString();

?

Beacause it requires including Qt headers(on Windows). If that don't matter (we already do that), I can switch to QString is needed.

Ah, I see. @lukas-w has offered me advice on where it's okay to use Qt stuff in the past. I think things like QString, at least, are fine everywhere, but hopefully he'll chime in here to confirm that this is OK if he's not too busy.

To be more clear, RemoteVstPlugin use Qt libraries for IPC on Windows. So I think that would be fine.

Edit: Linux build also needs the function, too. That's why I didn't use Qt function here.

@PhysSong is right, this is not a good place to use Qt because that would add MinGW Qt as a dependency on Linux builds. We can however use C++11 functions to convert to UTF-16:

#include <locale> #include <codecvt>

std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter; return converter.from_bytes(s);

This does the same thing, just in fewer lines. I think we already use C++11 for RemoteVstPlugin on stable-1.2, but I didn't check.

because that would add MinGW Qt as a dependency on Linux builds.

It seems like we can use host version Qt, but I don't think it's a good idea because it adds 32bit Qt as a dependency on 64bit Linux.

We can however use C++11 functions to convert to UTF-16:

That doesn't work with our old MinGW because of missing header files(it should work with newest g++-mingw-w64). Those functions are deprecated in C++17, too.

Wallacoloo · 2018-06-10T21:46:30Z

plugins/vst_base/RemoteVstPlugin.cpp

+#define F_OPEN(a, b) _wfopen(toWString(a).c_str(), L##b)
+#else
+#define F_OPEN(a, b) fopen((a).c_str(), b)
+#endif


I see you added these F_OPEN defines, but they don't seem to be used anywhere (?)

Forgot to commit that. Thank you!

Wallacoloo · 2018-06-10T21:51:31Z

plugins/vst_base/RemoteVstPlugin.cpp

@@ -2083,7 +2106,7 @@ int main( int _argc, char * * _argv )
 	}

 	// constructor automatically will process messages until it receives
-	// a IdVstLoadPlugin message and processes it
+    // a IdVstLoadPlugin message and processes it


^ No need to change the indentation here. It was fine before :)

Wallacoloo · 2018-06-10T22:01:53Z

plugins/zynaddsubfx/zynaddsubfx/src/Misc/QtXmlWrapper.cpp

+#else
+    int fd = dup(f.handle());
+#endif
+    f.close();


I see you're using this to open files by their descriptor instead of the filename. Is it gzopen that's giving you problems, fopen, or both? Because if it's only gzopen that's broken, it might be possible to just always fopen, and then call gzdopen(fileno(file), options) if compression is enabled, rather than doing this trickery with duplicating the file handles and such.

Both are broken, because C library only supports characters in local code page.

Oh, oops. I had typed out that comment and after digging deeper to better understand the problem I meant to remove it before posting the review >.<

Yeah, I agree with your conclusion here. It's not pretty, but I don't see any better solution.

Wallacoloo · 2018-06-10T22:25:16Z

plugins/zynaddsubfx/zynaddsubfx/src/Misc/QtXmlWrapper.cpp

+#else
+    int fd = dup(f.handle());
+#endif
+    f.close();


I wonder if you could offload this logic to a helper function,

static int fdopenFromQString(QString filename, QIODevice::OpenMode mode)

to avoid duplicating the body in multiple functins, and then leave a comment near that function explaining why the function is necessary (to most readers, this code will look silly and they may be tempted to simplify it back to the way it was before).

Good suggestion!

Wallacoloo · 2018-06-10T22:31:42Z

src/core/DrumSynth.cpp

+    QFile f(file);
+    f.open(QIODevice::ReadOnly);
+    QByteArray dat = f.readAll().constData();
+    is.str(string(dat.constData(), dat.size()));


The casual reader will see this and think "we're reading the entire file into a string, and then operating on it only as a stream? Why not directly open the file with ifstream?" and will be tempted to change this back to how it was previously. I think there should be some [short] comment explaining why we must open the file with QFile, possibly linking back to this github PR.

Wallacoloo · 2018-06-10T22:37:06Z

src/core/SampleBuffer.cpp

@@ -416,7 +412,10 @@ f_cnt_t SampleBuffer::decodeSampleSF( const char * _f,
 	f_cnt_t frames = 0;
 	bool sf_rr = false;

-	if( ( snd_file = sf_open( _f, SFM_READ, &sf_info ) ) != NULL )
+
+	QFile f(_f);


I'm thinking these should all be commented with something along the lines of

// Open the file with QFile and pass the descriptor to sf_open_fd instead of directly calling sf_open // for better filename encoding support. See https://github.com/LMMS/lmms/pull/4401

so that people know not to revert this change.

Good point.

Wallacoloo · 2018-06-11T04:13:43Z

@PhysSong Thanks for the improvements!

I'm marking this as "approved" now, since you've addressed any concerns I had. But it looks like Lukas wanted to review this too, so you should probably wait until that before merging 🎉

lukas-w · 2018-06-11T07:13:17Z

plugins/stk/mallets/mallets.cpp

+#else
+						.toUtf8().constData() );
+#endif
+


I think we can (should?) use toLocal8Bit on Unix as well. It should be equivalent to toUtf8 on most Linux and macOS systems. Correct me if I'm wrong.

Yes, I think that would be simpler. AFAIK that wouldn't break anything, too.

lukas-w · 2018-06-11T07:17:51Z

plugins/vst_base/RemoteVstPlugin.cpp

+static std::wstring toWString(std::string s)
+{
+	/*
+	QString qstr = QString::fromUtf8(s.data(), int(s.size()));


@PhysSong is right, this is not a good place to use Qt because that would add MinGW Qt as a dependency on Linux builds. We can however use C++11 functions to convert to UTF-16:

#include <locale> #include <codecvt>

std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter; return converter.from_bytes(s);

This does the same thing, just in fewer lines. I think we already use C++11 for RemoteVstPlugin on stable-1.2, but I didn't check.

lukas-w · 2018-06-11T07:29:36Z

plugins/zynaddsubfx/zynaddsubfx/src/Misc/QtXmlWrapper.cpp

@@ -193,9 +216,11 @@ int QtXmlWrapper::dosavefile(const char *filename,
                           int compression,
                           const char *xmldata) const
 {
+   int fd = fdopenFromUtf8(filename, QIODevice::WriteOnly);


Why not use the same method here that's used in RemoteVstPlugin.cpp? We can move the toWString method and the F_OPEN defines to a common header and include that in both files.

I think F_OPEN macro is an ugly workaround for fopen. I can do that if needed, but I don't like the macro approach.

@PhysSong You can make it a function instead.

Something tricky about that: "r" vs L"r" or something like that. That's why I chose macro approach.

@PhysSong Sounds like you need to convert the mode string as well. So something like this:

std::FILE* openFile(std::string filename, std::string mode) { #ifdef LMMS_BUILD_WIN32 return _wfopen(toWString(filename), toWString(mode)); #else return fopen(filename.data(), mode.data()); #endif }

You could, but I don't think there's a need for it, because they're just a couple lines each and operate on an UTF-16 QString instead of an UTF-8 std::string.

I think either lmms_io.h or IoHelper.h will be fine. I'll address this point soon.

Or how about putting them in RemotePlugin.h? I can't use-cases of UTF-8 string filename except in remote plugins.

Edit: It doesn't look like good because Zyn core needs it.

Is it fine for our Zyn to include LMMS' headers at all? I think this will be fine in stable-1.2, but I'm not sure about master's submodule. It would be the last concern for me...

I know it doesn't make sense to build https://github.com/LMMS/zynaddsubfx as a standalone, of course. However, I think it's possible to check if it's building with LMMS or not, though not really needed.

PhysSong · 2018-06-27T12:21:59Z

@lukas-w I addressed these issues and pushed to a separate branch. I'll push this change to the original branch when/if you says okay.

PhysSong · 2018-06-27T12:39:40Z

Oops, accidentally pushed them. I can roll back to original commits whenever needed.

lukas-w · 2018-06-27T13:12:23Z

@PhysSong Please roll back to the original commits, all the comment/review history on GitHub is lost otherwise. It's okay to have one extra commit on top of the original ones, there's no need to rewrite history.

PhysSong · 2018-06-27T13:31:58Z

all the comment/review history on GitHub is lost otherwise.

What do you mean? Review comments are still there unless you commented on specific commits, the are just collapsed.

Anyway, I agree that it's not necessary to rewrite history. I just wanted to keep things cleaner, but that was all. If you're still unhappy with this force-push, I can roll back.

lukas-w · 2018-06-27T13:43:29Z

What do you mean?

Sorry, I was being imprecise. The comments are still there, but the context is lost. Review comments are always on specific commits and you can't view that commits any more, so you can't see the changes the review refers to. Try clicking on the "View changes" button next to a review, this is what you'll get now:

If you're still unhappy with this force-push, I can roll back.

Yes, please do 👍

PhysSong · 2018-06-27T14:05:37Z

I just noticed even a rebasing can cause such a thing. I just accidentally rebased commits because I didn't know that... I'll revert to the revision before rebasing if needed.

PhysSong · 2018-07-01T13:57:12Z

BTW @lukas-w, do you want any more changes in this pull request?

lukas-w · 2018-07-01T15:31:11Z

@PhysSong Apart from restoring the pre-force-push commits, nope.

PhysSong · 2018-07-04T23:55:35Z

Then do you mind if I merge this(probably with cleaner history)?

lukas-w · 2018-07-05T09:44:25Z

I don't mind, go ahead ❤️

Fix plugin loading and setting loading/saving

Still incomplete on Windows due to an upstream issue

PhysSong · 2018-07-05T12:26:13Z

Just found a very rare encoding issues when using temporary directory whose name contains some non-ASCII characters on Linux. In that case VST plugin can't be loaded.
I can fix it, but it requires some dangerous changes(main->wmain and some more) and is easy to work around. So I'll open an issue instead and merge this soon.

lukas-w · 2018-07-06T20:53:32Z

plugins/vst_base/RemoteVstPlugin.cpp

-			int fd = open( _file.c_str(), O_WRONLY | O_BINARY );
-			if ( ::write( fd, chunk, len ) != len )
+			FILE* fp = F_OPEN_UTF8( _file, "wb" );
+			if ( fwrite( chunk, len, 1, fp ) != len )


Only spotted this now: fwritereturns the count of elements successfully written, so in this case it will return 1 on success, not len. Therefore this condition is always leading to a false error message on success (except when coincidentally len == 1).

Thank you! Changed to fwrite( chunk, 1, len, fp ) in 0f3b41f.

Also synced into master via 170a46e.

Fix a regression in 3e538d5 (#4401) that readAllData doesn't read the file from the beginning

Fix a regression in 2dadb15 (LMMS#4401) that readAllData doesn't read the file from the beginning

PhysSong force-pushed the encoding branch from e162527 to e675b2d Compare June 4, 2018 06:01

lukas-w self-requested a review June 10, 2018 17:10

Wallacoloo requested changes Jun 10, 2018

View reviewed changes

PhysSong force-pushed the encoding branch 4 times, most recently from 6cd2d26 to 1f09c05 Compare June 11, 2018 03:59

Wallacoloo approved these changes Jun 11, 2018

View reviewed changes

lukas-w reviewed Jun 11, 2018

View reviewed changes

lukas-w mentioned this pull request Jun 25, 2018

Fix MSVC VST compilation #4421

Closed

PhysSong force-pushed the encoding branch from 1f09c05 to d0b6173 Compare June 27, 2018 12:37

PhysSong force-pushed the encoding branch from d0b6173 to 34816e8 Compare June 27, 2018 14:02

PhysSong added 6 commits July 5, 2018 20:16

Fix file name encoding issues with VST on Windows

792e478

Fix plugin loading and setting loading/saving

Fix file name encoding issues with ZynAddSubFX on Windows

9d0aae2

Fix sample file loading on Windows

ddcae47

Fix WAV exporting on Windows

1af0f08

Fix MIDI import encoding issue on Windows

3e538d5

Improve STK rawwave path encoding handling

62d505b

Still incomplete on Windows due to an upstream issue

PhysSong force-pushed the encoding branch from 34816e8 to 62d505b Compare July 6, 2018 01:23

PhysSong merged commit 62d505b into LMMS:stable-1.2 Jul 6, 2018

PhysSong deleted the encoding branch July 6, 2018 02:07

lukas-w reviewed Jul 6, 2018

View reviewed changes

zonkmachine mentioned this pull request Jul 24, 2018

Free Distortion VST Temper resets values when project is re-opened #4500

Closed

PhysSong mentioned this pull request Jul 27, 2018

MIDI import: Reset the seek pointer after parsing magic bytes. #4503

Closed

PhysSong added a commit that referenced this pull request Jul 27, 2018

Fix broken MIDI import

112d342

Fix a regression in 3e538d5 (#4401) that readAllData doesn't read the file from the beginning

DomClark mentioned this pull request Aug 13, 2018

Synth1 preset settings are lost at upgrade from 1.1.3 to 1.2.0-RC5 #4084

Closed

DomClark mentioned this pull request Dec 9, 2018

Some problems with VST. Again. #3677

Closed

DomClark mentioned this pull request Jan 16, 2019

VST changes preset value on load #3847

Closed

PhysSong mentioned this pull request Oct 1, 2019

Assertion failure loading missing sample in MSVC debug build #5213

Closed

PhysSong mentioned this pull request Oct 10, 2019

RemoteVstPlugin crashes on loading/saving presets if it fails to open the file #5234

Closed

PhysSong mentioned this pull request Aug 24, 2020

Update portsmf + some MIDI loading updates #5653

Merged

sdasda7777 pushed a commit to sdasda7777/lmms that referenced this pull request Jun 28, 2022

Fix broken MIDI import

bc2530c

Fix a regression in 2dadb15 (LMMS#4401) that readAllData doesn't read the file from the beginning

Fix some encoding issues (mostly on Windows) #4401

Fix some encoding issues (mostly on Windows) #4401

Conversation

PhysSong commented Jun 4, 2018

zonkmachine commented Jun 10, 2018

Choose a reason for hiding this comment

PhysSong Jun 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PhysSong Jun 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wallacoloo commented Jun 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PhysSong Jun 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PhysSong commented Jun 27, 2018

PhysSong commented Jun 27, 2018

lukas-w commented Jun 27, 2018

PhysSong commented Jun 27, 2018

lukas-w commented Jun 27, 2018

PhysSong commented Jun 27, 2018

PhysSong commented Jul 1, 2018

lukas-w commented Jul 1, 2018

PhysSong commented Jul 4, 2018

lukas-w commented Jul 5, 2018

PhysSong commented Jul 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PhysSong Jun 10, 2018 •

edited

Loading

PhysSong Jun 10, 2018 •

edited

Loading

PhysSong Jun 25, 2018 •

edited

Loading