Apple M1 architecture

[jasonnerothin]

Zstd builds and seems stable on Apple M1. I'd like to provide a PR.

What would the logistics need to be? (No Actions for M1 yet on Github.)

[luben]

Thanks for confirming there are not problems on M1. The current issue is the lack of M1 support on Github actions - I use the build artifacts from the GH actions (and GCC compile farm) and include them in the final jars.

Let's hope the M1 build problems get resolved sooner: actions/runner-images#2187, actions/runner#805

Another option is to cross-compile it. I have to research a little bit about this, though the latest installed xcode (12.2) should support it. Edit: it looks it support creating universal binaries but I don't think this will work for my use case.

[jasonnerothin]

Seems like cross-compile is going to be the only option for a while, even after Github fixes that one - sbt (and most things JVM) are mostly broken. Same thing for golang and therefore Docker.

[luben]

I have a build script that uses just the cc and produces the binary, I use it for cross-compiling and building on more exotic architectures. Downside is that I don't run the test suite on them.

[jasonnerothin]

Could you put it up on a new branch?

My exotic architecture is a 2015 MBP (sdkman -> JVMs, SBTs, scala-langs...) and a 2020 MBA ("native" homebrew -> cmake zstd))

Could it be adapted to run the sbt tests on an sbt-container image prior to packaging up the dylibs from an as-yet-nonexistent m1 action?

[luben]

@jasonnerothin , looks I was able to cross-compile it with Xcode using github actions, here is the result:

~/zstd-jni(master) » file target/classes/darwin/aarch64/libzstd-jni.dylib
target/classes/darwin/aarch64/libzstd-jni.dylib: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>

One question, can you check the architecture folder in the JAR you compiled on M1? It should be darwin/arm64 or darwin/aarch64 or something similar. As I am cross-compiling it, so I don't know what will be the value for architecture in the JVM.

[jasonnerothin]

This program prints this output:

OS: mac_os_x arch: aarch64

Note: I'm running Azul Systems' JVM. It is the only one that runs, last time I checked.

[luben]

Great, thanks. So next release should support M1. I will be releasing later today with v1.4.7 upstream.

[luben]

Hmm, I will wait for facebook/zstd#2428 to be resolved as it looks dangerous.

[luben]

@jasonnerothin pushed v1.4.7-1 to maven-central, it also contains the fix for the above bug. Can you test it?

[luben]

Better try 1.4.7-3

[dongjoon-hyun]

Thank you for releasing 1.4.7-3, but Facebook seems to release a hot fix 1.4.8.

• facebook/zstd@7c495e8

[luben]

@dongjoon-hyun , yes I already have the same fix. Doesn't hurt to release also 1.4.8

[dongjoon-hyun]

Nice! Thanks for confirming, @luben !

[luben]

@dongjoon-hyun , I just pushed v1.4.8-1 the only functional change is adding an assert for the buffer alignment.

[jasonnerothin]

This is what I'm seeing: Unsupported OS/arch, cannot find /darwin/aarch64/libzstd-jni.dylib or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in your system.

[luben]

@jasonnerothin , There should be other error strings on the previous lines. Can you tell me what they say. Also which line is the exception thrown?

Double checked the native artifacts packed in the jar and there is nothing suspicious there:

 » file libzstd-jni.dylib.*                                                                                                                                                                                                  luben@zenbook
libzstd-jni.dylib.1.4.5:   Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>
libzstd-jni.dylib.1.4.8:   Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>
libzstd-jni.dylib.aarch64: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>

[luben]

Another thing that may help is to extract the library from the jar, e.g. 'jar -xf zstd-jni.jar' and find if there is some break in the linking with ldd libzstd-jni.dylib (I hope it works the same on Mac).

[jasonnerothin]

created: darwin/
created: darwin/x86_64/
inflated: darwin/x86_64/libzstd-jni.dylib
...
created: linux/
created: linux/aarch64/
inflated: linux/aarch64/libzstd-jni.so

[jasonnerothin]

jasonnerothin@wibble x86_64 % pwd
/tmp/darwin/x86_64
jasonnerothin@wibble x86_64 % otool -L libzstd-jni.dylib
libzstd-jni.dylib:
	/Users/runner/work/zstd-jni/zstd-jni/target/classes/darwin/x86_64/libzstd-jni.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

[luben]

@jasonnerothin , any help with the error message? It should at least have line number, but also there may be another error messages before the Unsupported OS/arch, cannot find /darwin/aarch64/libzstd-jni.dylib ... line

[jasonnerothin]

^^ two comments back is partial output of jzr xvf zstd-jni-1.4.5-12.jar: The dylib is going to the linux directory and the so file is ending up in the darwin directory.

^ last comment says that the dylib file is pointing to a non-existent directory.

[luben]

zstd-jni-1.4.5-12 is not build for M1. I started including the M1 native binary in the 1.4.7 (thought it was fast followed by 1.4.8).

I don't think the dylib is going the the linux directory:

$ jar tf zstd-jni-1.4.8-1.jar | grep dylib
darwin/aarch64/libzstd-jni.dylib
darwin/x86_64/libzstd-jni.dylib

I am not sure about the link to "/User/runner/..." : I don't think this is an issue as 1.4.5 was build using the old gcc toolchain and it's working. With 1.4.7 I switched the toolchain to use the xcode compiler in order to support the cross-compilation to M1.

Can you try with v1.4.8-1?

[jasonnerothin]

Okay, so it looks like the issue is that Apple has moved its "User Cache" to somewhere else.

otool is reporting that the darwin dylib is looking for /usr/lib/libSystem.B.dylib. find / -type f ...etc turns nothing up in the default OS.

However, installing XCode installs one for each of tvOS, iOS, watchOS.

I am symlinking to see if I can get the scala tests to run from IntelliJ.

[l0rinc]

Seems to be working with latest 1.4.9-1 - can this issue be closed?

[luben]

Thanks for confirming. I am not sure where the fix came from - most probably something changed on the GH workers.

@jasonnerothin , can you confirm 1.4.9-1 is working for you also?