Unable to publish NativeAOT project for linux-bionic-arm64

[emmauss]

Description

Building a simple Hello World sample works for the linux-bionic-x64 and linux-bionic-arm64 rids, but publishing AOT fails for linux-bionic-arm64, but works for linux-bionic-x64. The following error is thrown immediately when attempting the build,

/home/emmaus/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-preview.6.23307.4/build/Microsoft.NETCore.Native.Publish.targets(75,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [/home/emmaus/test/test.csproj]

Reproduction Steps

A simple project with the required nuget packages already referenced here, https://github.com/emmauss/test-aot.
Restore and build it with
dotnet publish -r linux-bionic-arm64 -c Debug -p:SysRoot=/.tools/android-rootfs/android-ndk-r21/sysroot. The error is immediately thrown.

Expected behavior

The project quickly builds and a compatible executable is generated.

Actual behavior

The following error is displayed,

/home/emmaus/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-preview.6.23307.4/build/Microsoft.NETCore.Native.Publish.targets(75,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [/home/emmaus/test/test.csproj]

Regression?

No response

Known Workarounds

No response

Configuration

OS: Fedora 38 on WSL2 running on Windows 11 x64 host
.Net Version: 8.0.0-preview.6.23307.4 daily build

Other information

No response

[ghost]

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Building a simple Hello World sample works for the linux-bionic-x64 and linux-bionic-arm64 rids, but publishing AOT fails for linux-bionic-arm64, but works for linux-bionic-x64. The following error is thrown immediately when attempting the build,

/home/emmaus/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-preview.6.23307.4/build/Microsoft.NETCore.Native.Publish.targets(75,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [/home/emmaus/test/test.csproj]

Reproduction Steps

A simple project with the required nuget packages already referenced here, https://github.com/emmauss/test-aot.
Restore and build it with
dotnet publish -r linux-bionic-arm64 -c Debug -p:SysRoot=/.tools/android-rootfs/android-ndk-r21/sysroot. The error is immediately thrown.

Expected behavior

The project quickly builds and a compatible executable is generated.

Actual behavior

The following error is displayed,

/home/emmaus/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-preview.6.23307.4/build/Microsoft.NETCore.Native.Publish.targets(75,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [/home/emmaus/test/test.csproj]

Regression?

No response

Known Workarounds

No response

Configuration

OS: Fedora 38 on WSL2 running on Windows 11 x64 host
.Net Version: 8.0.0-preview.6.23307.4 daily build

Other information

No response

Author:	emmauss
Assignees:	-
Labels:	untriaged, area-NativeAOT-coreclr
Milestone:	-

[MichalStrehovsky]

This is not expected to work yet. A fix that should get rid of the error you're seeing just went in a day a two ago but didn't make into installer yet. The fix was for building iOS but it should make progress with Bionic too. I didn't test Bionic with it yet but I was waiting until it merges to make more progress on Bionic.

[emmauss]

Thank you for the reply. I'll wait then and test it when it's ready.

[MichalStrehovsky]

It's possible to try this out now:

• Download the latest installer build from dotnet/installer repo. I just download the ZIP variant and unpack it somewhere.
• Download Android NDK and unpack it somewhere. If you have Android Studio, you probably already have this somewhere.
• Then (these instructions are for Windows, but basically just replace set with export for Linux):

  # Put the downloaded installer on PATH (update the path as necessary)
  $ set path=c:\net8;%path%
  # Put the downloaded Android NDK on PATH (update path as necessary) - we need Android clang to be first on PATH
  $ set path=C:\Programs\android-ndk-r21b\toolchains\llvm\prebuilt\windows-x86_64\bin;%path%
  # Make a new console app. Shared libraries should also work but I didn't test.
  # If you get a failure during restore, you need to create a NuGet.config as described
  # in the root README.md of the dotnet/installer repo.
  $ dotnet new console -o HelloBionic --aot
  $ cd HelloBionic
  # Publish for ARM64 Bionic. Current gotchas:
  # 1. Needs to be a Debug build. This is fixed, but the fix didn't reach installer repo yet.
  # 2. Need to specify StripSymbols=false. Didn't look into this yet.
  # 3. Need to specify InvariantGlobalization=true (the `dotnet new console --aot` template does this) or you get a FailFast. Didn't look into this yet.
  $ dotnet publish -r linux-bionic-arm64 -c Debug -p:DisableUnsupportedError=true -p:PublishAotUsingRuntimePack=true -p:StripSymbols=false

• You should have a binary under bin\Debug\net8.0\linux-bionic-arm64\publish. Copy this to an Android device. Either adb push or using some GUI.
• You can probably run it with adb shell, but I used Termux: open Termux, give it access to file system by running termux-setup-storage. This will give you access to phone storage under ~/storage. Copy the binary from ~/storage/... to ~ (internal storage is not executable and you won't be able to run stuff from it). Then chmod +x HelloBionic and ./HelloBionic. You should see Hello World. There will also be a warning, that one is also fixed and will be in dotnet/installer in a day or two.

The more interesting variation of this is building shared libraries that you can call into with JNI. There's a sample for that at https://github.com/josephmoresena/NativeAOT-AndroidHelloJniLib. The instructions obviously don't explain how to use this with today's bits, but @josephmoresena should be able to greatly simplify it soon (or you can send a PR).

Cc @kekekeks @maxkatz6

[josephmoresena]

The more interesting variation of this is building shared libraries that you can call into with JNI. There's a sample for that at https://github.com/josephmoresena/NativeAOT-AndroidHelloJniLib. The instructions obviously don't explain how to use this with today's bits, but @josephmoresena should be able to greatly simplify it soon (or you can send a PR).

I can't access a Windows/Linux machine at the moment (I only have a MacBook Air for now), so I guess I can't test this. I will of course accept any PR with these instructions for .NET8.
As I see, all that is needed is to add to the current path the android toolchain directory.

The only simplification of the instructions I've done on that repository was the use of BFlat.

I would think it will be a while before I can test this but I think this is excellent news. Could I do something similar from MacOS?

[MichalStrehovsky]

I would think it will be a while before I can test this but I think this is excellent news. Could I do something similar from MacOS?

I would be surprised if the above instructions didn't work for macOS. I see there's some extra notes around macOS in the NDK downloads ("Only a DMG is provided for direct download for macOS because it is required for Catalina and newer. In most cases it will be easier to use the Android SDK manager to install the NDK.") but I assume those are just the general macOS pains that everyone bought into that ecosystem is already accustomed to :).

[emmauss]

Thanks. A sample Hello World project builds and runs fine on termux.

[emmauss]

Moving on to building a much larger library with many dependencies, one of which rely on System.Security.Cryptography. Building is successful, but when loaded in a native app, crashes with a SIGABRT on calling one of said dependency's methods.
The crash occurs at

 XXlib/arm/LibSample.so (InitializeOpenSSLShim+) ()
 XXlib/arm/LibSample.so ()
 /apex/com.android.runtime/lib/bionic/libc.so (pthread_once+) ()
 XXlib/arm/LibSample.so (CryptoNative_EnsureOpenSslInitialized+) ()
 XXlib/arm/LibSample.so ()

Is it possible to link openssl?

[MichalStrehovsky]

We don't build/provide an openssl build because we don't want to be in the business of maintaining it and doing all the necessary security servicing. There's one we use in testing at https://netcorenativeassets.blob.core.windows.net/resource-packages/external/android/openssl-1.1.1l-beta-1.zip (you'll have to look around this repo for how it's consumed - I don't actually know).

[emmauss]

Following hints found in https://github.com/josephmoresena/NativeAOT-AndroidHelloJniLib , I'm unable to load anative shared library built. Logcat error is as follows;

digest=============org.sample.kjava.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__android_log_print" referenced by "XXlib/arm/libsample.so"...
  at java.lang.Runtime.load(Runtime.java:XX)
  at java.lang.System.load(System.java:XX)

When building an ndk library in android studio, they are linked at compile time to the android log library. Those binaries load fine using the same method as when loading the dotnet nativeaot shared library.
@josephmoresena were you able to load your sample jni library?

[MichalStrehovsky]

cannot locate symbol "__android_log_print" referenced by "XXlib/arm/libsample.so"...

Can you try adding this to the project file, delete obj and bin directories and publish again?

<ItemGroup>
  <LinkerArg Include="-llog" />
</ItemGroup>

[emmauss]

cannot locate symbol "__android_log_print" referenced by "XXlib/arm/libsample.so"...

Can you try adding this to the project file, delete obj and bin directories and publish again?

<ItemGroup>
  <LinkerArg Include="-llog" />
</ItemGroup>

Adding this works, and I'm able to call functions from the library. Thanks.

[MichalStrehovsky]

Thanks for confirming - I created #87862 to include this by default when targeting Bionic. Once that merges and propagates through the systems, the workaround won't be needed.

[MichalStrehovsky]

#87900 was the last fix.

The only remaining issue is to document how to set up the environment, and the gotcha with openssl.

[emmauss]

Nice. Am I right to assume we can publish Release builds now? And that we can strip symbols?

[MichalStrehovsky]

Nice. Am I right to assume we can publish Release builds now? And that we can strip symbols?

Yes and yes. #87900 might need to flow into installer still, but if you're not on Windows, that one shouldn't be needed and latest installer should have everything I think.

[emmauss]

What's the expected behavior of OperatingSystem/RuntimeInformation apis when running on android when called in a bionic library? They both detect it as a Linux system, and not specifically Android, with RuntimeInformation,OSDescription returning the linux kernel version.

I'm not able to call any functions in a shared library when built as Release using the following;
dotnet publish -r linux-bionic-arm64 -c Release -p:DisableUnsupportedError=true -p:PublishAotUsingRuntimePack=true -p:StripSymbols=false
Running a debug library works fine.

[maxkatz6]

I wonder if something like RuntimeInformation.IsOSPlatform(OSPlatform.Create("BIONIC")) makes sense.

[emmauss]

We would still have to handle OperatingSystem.IsAndroid. The only OS, to my knowledge, that you'd run bionic binaries on is Android.

[MichalStrehovsky]

What's the expected behavior of OperatingSystem/RuntimeInformation apis when running on android when called in a bionic library? They both detect it as a Linux system, and not specifically Android, with RuntimeInformation,OSDescription returning the linux kernel version.

Android typically means the Android RID with Java interop support, etc. What would you do differently based on knowing this is Bionic? (Right now this feels like no different from whether this is Ubuntu, or Red Hat or Gentoo - they're all Linux but still quite different). You should see Bionic in RuntimeInformation.RuntimeIdentifier.

I'm not able to call any functions in a shared library when built as Release using the following

How is it failing? Is there a failing stack you could share?

[emmauss]

I'm not able to call any functions in a shared library when built as Release using the following

How is it failing? Is there a failing stack you could share?

No stack, the app just crashes

[MichalStrehovsky]

I'm not able to call any functions in a shared library when built as Release using the following

How is it failing? Is there a failing stack you could share?

No stack, the app just crashes

Did you try running it under a debugger to see if there's a stack at the point of failure? (E.g. File -> Profile or Debug APK in Android Studio, or equivalent).

[emmauss]

Yes. It's an Android project that's consuming the library, and I debug with Android Studio. Simply replacing the debug library with no code change on the kotlin project, and it falls on the first function call to the library.

[emmauss]

The debugger doesn't break, and logcat prints this,

Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x1db0967d0 in tid 11029 (org.sample.k), pid 11029 (org.sample.k)
crash_dump.cpp:512] failed to attach to thread 11029, already traced by 19932 (/data/data/org.sample.k/lldb/bin/lldb-server)
crash_dump helper failed to exec, or was killed

[MichalStrehovsky]

There's probably something that is needed for Android Studio to break into the debugger when the SIGSEGV happens. Quick search seems to indicate there's a UI option to enable native debugging. Maybe something along the lines of this? https://stackoverflow.com/questions/28602260/how-to-get-breakpoint-in-ndk-native-code-and-debug-native-code-in-android-studio

(Sorry I don't actually have much Android experience myself, but there should be a stack associated with a SIGSEGV - it's an out of range attempt to access memory and there needs to be something that did it).

[MichalPetryka]

Android typically means the Android RID with Java interop support, etc. What would you do differently based on knowing this is Bionic? (Right now this feels like no different from whether this is Ubuntu, or Red Hat or Gentoo - they're all Linux but still quite different). You should see Bionic in RuntimeInformation.RuntimeIdentifier.

If Unity decides to use Bionic for Android Apps when switching to NativeAOT, the users will probably expect to see Android being returned. JNI support won't matter there.

[jkotas]

We can deal with that problem once/if Unity runs into it. This is not the only place where Android treatment is debatable today. For example, OperatingSystem.IsLinux() returns false for Android RID with Java interop today.

[josephmoresena]

var stringPtr = newString(jEnv, jString, new JBooleanRef(false)); throws the SIGSEGV. It works fine on Debug build. It is possible to instead create the helper in an ndk project, and p/invoke to that as a workaround.

I see you're using the JNI templete NativeAOT-HelloWorldJNI. Please open a discussion at that repo.

I'm working on a high level project from this. You're getting the Utf8 Chats from VM but you need to encode them to UTF16 or retrieve directly UTF16 chars from VM.

[emmauss]

On moving to use the windows daily releases, I came across 2 issues that caused build to fail;

1. Newer sdk versions aren't supported. They break with a invalid linker name in argument '-fuse-ld=bfd' error in the clang call. Ndk version 21, which is old, no longer supported and nor part of modern android sdk packages, works.
2. chmod is called during generating the .so file. This doesn't exist on windows system. I used a no-op batch file to replace this command in PATH to workaround this.

[josephmoresena]

If you remove -fuse-ld=bfd does it work?
Which file calls chmod? Why is it called when generating .so file?

[emmauss]

If you remove -fuse-ld=bfd does it work? Which file calls chmod? Why is it called when generating .so file?

I don't know how to modify the clang commands, as this was run with just dotnet publish.
The chmod error log is as follows

  
  Libsample -> D:\sample\sample\src\Libsample\bin\Debug\net8.0\linux-bionic-arm64\Libsample.dll
  'chmod' is not recognized as an internal or external command,
  operable program or batch file.
C:\nuget\packages\microsoft.dotnet.ilcompiler\8.0.0-preview.7.23325.2\build\Microsoft.NETCore.Native.targets(373,5): er
ror MSB3073: The command "chmod 644 "bin\Debug\net8.0\linux-bionic-arm64\native\Libsample.so"" exited with code 9009.
[D:\sample\sample\src\Libsample\Libsample.csproj]

Right after this point on a working build, will be the text Generating native code in the console output, and then the trim and ilc stages run.
I also had to move my nuget packages directory to a path with no whitespace as my user profile's name is two words. This caused a failure when starting the ilcompiler. I no longer have the logs from that time.

[MichalStrehovsky]

I have a fix for the chmod issue in #88174.

I'll need to think about what to do for the linker issue. Does adding <LinkerFlavor>lld</LinkerFlavor> resolve that problem?

(Cc @am11 for thoughts/opinions - maybe we can somehow detect this?)

[am11]

We make some decisions based on the linker flavor and version, so we want to know the linker's name the toolchain is going to use. Detecting which linker is used by a given toolchain (clang or gcc) is quite rough, so we are using this approximate mapping

runtime/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets

Lines 24 to 25 in cc585f6

	<LinkerFlavor Condition="'$(LinkerFlavor)' == '' and '$(_targetOS)' == 'freebsd'">lld</LinkerFlavor>
	<LinkerFlavor Condition="'$(LinkerFlavor)' == '' and '$(_targetOS)' == 'linux'">bfd</LinkerFlavor>

I think, in the unlikely event, when this (platform->[default]linker) list becomes large or unmaintainable, we can replace it with the detection but for now™️, lets add the third one for linux-bionic. :)

[MichalStrehovsky]

I think, in the unlikely event, when this (platform->[default]linker) list becomes large or unmaintainable, we can replace it with the detection but for now™️, lets add the third one for linux-bionic. :)

Sounds good, thank you! @emmauss if you can confirm <LinkerFlavor>lld</LinkerFlavor> resolves this, I can make this the default for Bionic.

[emmauss]

I think, in the unlikely event, when this (platform->[default]linker) list becomes large or unmaintainable, we can replace it with the detection but for now™️, lets add the third one for linux-bionic. :)

Sounds good, thank you! @emmauss if you can confirm <LinkerFlavor>lld</LinkerFlavor> resolves this, I can make this the default for Bionic.

Setting lld makes it build successfully with ndk 25

[MichalStrehovsky]

Setting lld makes it build successfully with ndk 25

Thanks for confirming! #88179 should fix this.