<Unknown device> as device name with javapackager bundle, or under OpenJDK 11

[iaguilera]

My app works fine in Mac Os X until I package it with javapackager, selfcontaining Java 8 runtime. Then, some device names are reported as "Unknow devices". If I extract the .jar files from the package and run them, MIDI device names are ok...

IAC Driver device show name ok, only fails with my USB MIDI controllers...

So is an issue that only happends when creating a .app with javapackager.

Any help please?

[brunchboy]

Hello, thanks for bringing this to my attention. I have also been using javapackager to build self-contained Mac application bundles for my Beat Link Trigger, and I never even noticed it was causing this problem—but I can reproduce it, even when bundling OpenJDK 11 from Amazon Coretto.

(As an aside I just want to make sure you realize that Oracle Java 8 is past end-of-life, and if you are distributing software—even free software—using it, you are legally required to purchase a commercial license from Oracle at great expense. To protect yourself I would recommend you migrate to using OpenJDK instead, which has no such expensive licensing requirements nor hordes of angry lawyers to back them up. Amazon is offering long term support for both JDK 8 and 11 in their Coretto offering.)

However, javapackager itself is also past end-of-life and is no longer distributed in current Java distributions by Oracle or anyone else. So even though I plan to investigate this a little bit, the problem may well turn out to be javapackager, with nothing we can do about it in CoreMidi4J. Since we can’t fix javapackager, and there will be no further releases or bug fixes from Oracle, the answer may be that we have to stop using javapackager. The replacement is being worked on by the OpenJDK project as JEP 343, but it keeps slipping into future releases, and I have no idea when we will be able to use it.

If you can find a way to fix this in your own investigations before I do, a pull request would be very welcome.

[brunchboy]

A bit more testing confirms that it is only the CoreMidi4J devices whose names are unknown when running in the app-bundled JDK; the com.sun.media ones still show their expected names—which suggests we may have some hope of fixing this.

It’s going to be extraordinarily tedious to debug, though, since I am going to have to package an überjar and build it into an application bundle every time I want to try something. So, @DerekCook do you have any guesses as to what might be going on, or what I might want to add as debugging statements, to help reduce the iteration count?

[DerekCook]

Hi. Not really. Can't say I have seen anything like this or understand why JavaPacker causes this issue. Note I do not use JavaPacker, so do not have a clue as to where to start with it.

In the autumn I will be looking at if I can migrate to a later version of Java now for my librarians including packaging the JRE (now I have decent broadband via a 4G package), so obviously will have an interest in this when I get there.

I was also thinking it might be time to rebuild CoreMIDI4J for later Java versions, now JRE 7 and 8 are end of support, but let's sort this issue first.

[iaguilera]

I have found that the problem is the native launcher that javapackager creates inside the app. I extract the content of the app, and then noticed that if launch the .jar double clicking it everything worked fine, but if I launch with the native launcher, it failed.

So I replaced the native launcher by a bash script with a java -jar ... command, and repackage using Packages application. This solution worked for me...

[iaguilera]

And thanks for the info, we will move to OpenJDK...

[DerekCook]

OK, that is good news that you have a solution. I need to move to OpenJDK myself, but first need to investigate a problem that screws up MIDI names on Windows, which stopped my from migrating to J9. Hopefully they have fixed it now.

If you're happy that you have a solution, I will close this issue, especially as James points out that JavaPackager is no longer supported.

[iaguilera]

Yes this solution is ok for me, thanks for your support....

[DerekCook]

Thanks. I will close this issue then. If you need to come back to it, then we can always reopen it.

[brunchboy]

@iaguilera could you please give me more details of exactly how you implemented your solution? I would like to try it myself. What is the native launcher inside the package, and how do you get the package to run java -jar instead? When you do that, does the icon and other package-related features still work? Are you able to code sign the resulting package still, so that macOS is happy with it as trusted code?

[brunchboy]

And @DerekCook you should be able use Amazon’s Coretto version of OpenJDK without moving past JDK 8 if you want to, they are offering free long-term support for that as well, even though Oracle has stopped. Check whether it is compatible with your license, though.

[iaguilera]

Yes of course, I did this steps:

1. First I installed the app with the installer created by javapackager.
2. Then I searched the app in the Applications folder, and copied the Contents to a folder outside the app.
3. In the MacOS folder inside Contents, you can see the native launcher, it's a binary with the name of your application, that launch the Java app.
4. Remove the native launcher and create a bash script with the same name. In my case I put this content.

#!/bin/sh
basedir=$(cd "$(dirname "$0")"; pwd)
"$basedir/../Plugins/Java.runtime/Contents/Home/jre/bin/java" -Xdock:icon=$basedir/../Resources/[YOURAPPICON].icns -jar "$basedir/../Java/[YOURAPPJAR].jar"

Give the file permissions to execute.

The Xdock:icon parameter is needed so the Java app shows the app icon on the dock when executing, and not the default Java icon.

6. Contents\PlugIns\Java.runtime\Contents\Home\jre contains the jre partially, but I didnt find the bin folder with the java command. I created the bin folder and copied the java command from the full jre.

7. Then I put copied the modified Contents folder into other folder, that I renamed as [NAMEOFMYAPP].app. It maintained the app icon.

8. Then I created an installer of this .app with Packages application that is free.

9 I didn't codesign the package, but I think you can do it with Packages, I don't think it's a problem...

10. After created the package, delete the .app file (you still have the Contents folder, backup it). Don't keep the .app, at least in my case, it seems the installer overrides the .app when installing it, so it doesn't go to Application folder if you don't delete it.

Hope that helps

[iaguilera]

And what if I am using an old version of the JDK / JRE, it applies the license of the time it was published isn't true? I want to self content the JRE, but not the latest version, but a version of past year that I checked and I know my app is working perfectly.

[brunchboy]

Thank you for those steps, they give me a good starting point for my own experiments. Is this the Packages you are talking about?

As far as JDK 8 licensing and distribution goes, I am not a lawyer, and it is very complex. I don’t believe you will be able to obtain current versions from Oracle with security patches for distribution unless you purchase a commercial license. If you are willing to ship older versions with known security vulnerabilities, you may be fine, but I personally switched to OpenJDK last year to be safe (and remain current with security updates). The good news is that it worked just fine for me.

[iaguilera]

Yes that's the Packages software I used to create the installer for the app...

And of course the future is move to a new version of OpenJDK we will do the same...

[brunchboy]

Unfortunately, the workaround didn’t work for me, and in fact the problem seems to be deeper.

In my application bundle, there is no JRE (because JDK 11 and later no longer have this concept), so the path had to be slightly different. But also, javapackager did not even include the JDK’s bin directory, so I had to create one and copy over the java executable (which is very small, since these things all use dynamically loaded libraries these days).

Once I had done that, I was able to get the application to launch using your script approach, but sadly that did not bring back my device names. Then I started digging deeper, and I discovered that even if I do not use javapackager, but am using JDK 11 as my JAVA_HOME, CoreMidi4J is unable to determine device names for anything other than my IAC Driver devices. If I drop down to Oracle’s Java 10 or Java 8 JDKs, then I see the actual device names again.

So I am going to reopen this because it remains a problem for me, and is one even without javapackager involved in OpenJDK 11, so it is something we are going to have to figure out. But that last part also means it will be easier to test and debug. I will try creating a CoreMidi4J build with Derek’s debugging output re-enabled in the CoreMidiProvider, and see if that sheds any more light.

[brunchboy]

The plot thickens… After uncommenting the debug statements in CoreMidiDeviceProvider_getMidiDeviceInfo() OpenJDK 11 crashes in that function (see line 11 below):

Thread 32 Crashed:: Java: AWT-EventQueue-0
0 libsystem_kernel.dylib 0x00007fff687a82c6 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff68863bf1 pthread_kill + 284
2 libsystem_c.dylib 0x00007fff687126a6 abort + 127
3 libjvm.dylib 0x000000010a3f4df6 os::abort(bool, void*, void const*) + 22
4 libjvm.dylib 0x000000010a550f2b VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long) + 2799
5 libjvm.dylib 0x000000010a550426 VMError::report_and_die(Thread*, unsigned int, unsigned char*, void*, void*, char const*, ...) + 166
6 libjvm.dylib 0x000000010a550f79 VMError::report_and_die(Thread*, unsigned int, unsigned char*, void*, void*) + 33
7 libjvm.dylib 0x000000010a3f8e16 JVM_handle_bsd_signal + 816
8 libjvm.dylib 0x000000010a3f685b signalHandler(int, __siginfo*, void*) + 45
9 libsystem_platform.dylib 0x00007fff68858b5d _sigtramp + 29
10 ??? 0x00000000131432ea 0 + 320090858
11 libCoreMidi4J.dylib 0x000000017ed191ea Java_uk_co_xfactorylibrarians_coremidi4j_CoreMidiDeviceProvider_getMidiDeviceInfo + 887 (ostream:864)
12 ??? 0x0000000113143950 0 + 4615059792
13 ??? 0x000000011313da00 0 + 4615035392
14 ??? 0x000000011313dca7 0 + 4615036071
15 ??? 0x000000011313dca7 0 + 4615036071
16 ??? 0x000000011313dca7 0 + 4615036071
17 ??? 0x000000011313dca7 0 + 4615036071
18 ??? 0x000000011313da00 0 + 4615035392
19 ??? 0x000000011313dae2 0 + 4615035618

[brunchboy]

Ok, I have figured this out. It is an actual bug in CoreMidi4J. We are using CFStringGetCStringPtr to obtain C string pointers from Core Foundation String objects. Unfortunately, as documented in the above link, while this is a very convenient function when it works, you cannot assume it will work, it is free to return NULL if it cannot efficiently operate without allocating or copying memory. From the documentation:

Whether or not this function returns a valid pointer or NULL depends on many factors, all of which depend on how the string was created and its properties. In addition, the function result might change between different releases and on different platforms. So do not count on receiving a non-NULL result from this function under any circumstances.

And, for whatever reason, from JDK 11 and javapackager it was returning NULL. This is what was causing Derek’s debug print statements to crash on me when I uncommented them as well—it was returning NULL even for the "**NULL pointer**" failsafe CFString!

I have modified the code to use a safer function which allocates memory for the strings and always works, but of course this means we need to free them as well after we pass them to the Java constructor.

This fix is working for me. I will clean it up a bit more and push it to GitHub and Maven Central as a snapshot of 1.2 so you two can test it as well. Then if it looks good, we can release it.

[brunchboy]

All right, the snapshot build of release 1.2 is uploading to Maven Central now. Please let me know when you have a chance to see if it lets you use the normal native launcher, @iaguilera

[brunchboy]

Whoops, I made one more small change, so that (like in the original code) it uses the system encoding for the CFStrings, rather than assuming UTF-8, which the sample code I’d based the solution on was doing. I have updated the snapshot on Maven central as well.

[DerekCook]

Thanks, James. Sounds like you’ve been productive on this! I reopened the issue as well given that it was deeper than originally thought, but then noticed you had already reopened and closed it! On reflection I’d like to keep it open until I get a chance to check the new build on my system. Which might not be for a few days (more likely the weekend)

[brunchboy]

I’d really like to get a release out today so that I can finalize the release of Beat Link Trigger that I planned for this long weekend (it’s Labor Day in the USA), but I also realized overnight that I want to scan for anywhere else we might have used the problematic function.

I’ll leave the issue open regardless. If you’re going to be spending the time to test on your own system, I think I will merge the build modernization branch (switching from javah to using the -h argument of javac) in as well so you can test it all in one pass.

If I do end up releasing 1.2 for incorporation in Beat Link Trigger and then you find new issues in your testing, I can release 1.3 once we fix those. But I may also just hold off on my own release. There are other features and fixes I’d love to make available to people who don’t install their own Java environment, but another week won’t kill me. 😄

[brunchboy]

Ok, I have merged in the branch that fixes #30 as well, and everything seems to be working nicely, I look forward to hearing how both of your testing goes when you have time.

[brunchboy]

Oh, just as a reminder, my previous comment means that the minimum JDK for using the Maven build process is now 1.8, though it still compiles for use in 1.7 or later. I should figure out how to update the pom.xml to make that explicit.

[DerekCook]

Thanks, James.

Lots going on in work right now, and I'm in the middle of a pervasive problem in writing a librarian for a Roland FC300 foot controller, where I don't want to lose what is in my head (a first in first out buffer of small size these days ! :) ). So I do not think I will get around to it for a little while.

Once I have my FC300 editor under control and ready for release, then I want to revisit the Windows MIDI Port Name issues I was having on JRE9 and see if things are better on later JREs. If not we might need a CoreMIDI type SPI for Windows as well!

So I will do a quick check of your fixes as a working build as soon as I can, and then perhaps see what we want to do next (if needed) slightly longer term.

PS: It's hard to believe that CoreMIDI4J has passed its 3rd birthday now, and nearly at the 4th ( I was working on the first implementation over Christmas 2015)! Notwithstanding this new issue, it's stood the test of time pretty well!

[brunchboy]

Indeed it has, I was actually surprised to find anything to fix in it, and delighted to hear that more people are using it.

Oh @iaguilera if you are willing to test the fix, in case you aren’t familiar with how to work with Maven snapshots: update the dependency version of uk.co.xfactory-librarians/coremidi4 in your project from 1.1 (the last official release) to 1.2-SNAPSHOT, and then it will build using the latest code I have been describing here. You will probably also need to update the version number of your project itself to include a -SNAPSHOT suffix, because Maven protects against building non-snapshot releases with any snapshot dependencies.

With that, I believe you should just be able to use javapackager and the resulting application should work without any of the messing around with package contents that you have been using as a workaround for the problem. Please let us know if you can try that.

Once we release a final 1.2 release, you will want to update your dependency to that, rather than the snapshot, and you can then remove the -SNAPSHOT suffix from your own new version.

[brunchboy]

(And speaking of old projects, Derek, I heard from someone today asking if I was going to update DisplayWatcher, which I had not touched since 2004 and had quite forgotten about, to work with macOS Catalina! I need to migrate that from CVS to git, and publish it on github, so it won’t depend on my single copy of the source, or availability of time.)

[brunchboy]

I also just fixed another issue I found when building the project under JDK 11. In that circumstance we need to specify the source version for the javadoc tool as well, or we get errors relating to JDK 9 modules. I also discovered that we really should be building our javadoc under JDK 11 because it adds a lovely and highly useful search box! I will be making this change to my other open source projects shortly.

[iaguilera]

I would like to try the fixes, but I'm not using Maven, I added the .jar dependency to my Netbeans project...

[brunchboy]

Ah, well let me see if I can put a jar here.
How stupid. It is making me ZIP the Jar (which is already a ZIP file!)
coremidi4j-1.2-SNAPSHOT.jar.zip

[iaguilera]

Thanks! I will test tomorrow and give you feedback

[brunchboy]

Excellent! I will hold off on a release until you can do that. (And, I think you probably know this, but you will want to unzip the jar I uploaded before substituting it for the one in your netbeans build.)

[iaguilera]

Yes I unziped the file, it's ok

[brunchboy]

Great. Derek, I also just made a tweak to the pom.xml so you get a nice build failure explaining you need at least java 1.8 to build now, rather than a cryptic complaint that compilation failed because of an unrecognized -h argument to javac.

[iaguilera]

I have tried to pack my application with javapackager and coremidi4j-1.2, and it works perfectly with this version, name of the MIDI devices are shown now without problems...

[brunchboy]

That’s great news, thank you for verifying my fix, and hopefully that will save you some trouble in the future!

We will close this issue once there is a non-snapshot build released, and you can update at that time, but you should be fine using the snapshot build until then.

[iaguilera]

Ok thanks a lot for your support :-)

[DerekCook]

@iaguilera thanks very much for confirming, and of course kudos to James for finding the cause and finding it so quickly. It will be a little while before I get the time to look at it. Hopefulyl I wil get to try the new build on the weekend, but I have every confidence it will be fine.

[DerekCook]

Hi,

I finally got some downtime tonight, so have checked the snapshot build out with my an.factory librarian against my AN1x - the most portable synth I can bring to my Mac!! - and also with my new fc.factory librarian in development for my FC300 MIDI foot controller.

SYSEX import and export on both are looking good, so I think we can go for release based on our collective testing.

Later in the autumn, when I have a bit more time, I think we then need to look at "what next"

Cheers
Derek

[brunchboy]

Excellent, thanks so much, Derek, that was fast! 😄 I will publish a release tonight, and then can release Beat Link Trigger this weekend.

I would also be happy to help think about next steps, although I must say I am very happy with the library as it exists today.

[brunchboy]

All right, Version 1.2 is released and published to Maven Central. @iaguilera if you would like to update to the official release version, you can find it on the releases page.

[iaguilera]

Thanks a lot, I'm going to update my project with the official release version :-)