[build] Optimize managed <-> java type lookups (#3992)

Typemap data is used to correlate JNI type names to .NET Assembly-
Qualified Type Names, and vice versa:

	java/lang/Object <=> Java.Lang.Object, Mono.Android

Typemap data is used from `JNIEnv.GetJniName()` for managed-to-JNI
lookups, and from `TypeManager.GetJavaToManagedType()` for
JNI-to-managed lookups.

When [typemap files were first introduced][0], they relied on:

 1. A string-oriented mapping from Java type names to .NET Assembly
    Qualified names and vice versa; and

 2. A binary search via **bsearch**(3) over this table to find the
    associated type, using the source type as the "key".

(The introduction of `libxamarin-app.so` (decfbcc) merely moved the
(formerly separate) typemap data into `libxamarin-app.so` for Release
config builds -- Debug builds continued using separate typemap files --
but didn't otherwise change how these mappings work.)

This approach works very well at the expense of data size -- shorter
strings are 0-padded to a common width -- and slightly degraded
performance because of the requirement to perform string comparisons.
Furthermore, the managed-to-JNI lookup required that Reflection is
used to obtain the Assembly Qualified type name
(`Type.AssemblyQualifiedName`), while the JNI-to-managed lookup
likewise requires some Reflection to obtain `Type` instances (via
`Type.GetType()`).

Rework the typemap data in an effort to reduce Reflection use:
For the managed-to-JNI mapping, use the combination of
`type.Module.ModuleVersionId` and `Type.MetadataToken` -- a GUID
and an int -- instead of using `Type.AssemblyQualifiedName`.  This
allows us to perform the binary search over a set of 20 bytes (16
bytes for the UUID and 4 bytes for the token ID).

JNI-to-managed lookups still need to rely on a binary search across
strings, but instead of mapping the JNI name to an Assembly-Qualified
Type Name and using `Type.GetType()`, we instead map the JNI name to
the same GUID+token pair via a new internal call which uses
`mono_class_get()` & `mono_type_get_object()` to return the `Type`.

As a result of this fundamental change, `libxamarin-app.so` decreases
in size, and app startup time is reduced.  For a Release configuration
build of `tests/Xamarin.Forms-Performance-Integration`,
`libs/arm64-v8a/libxamarin-app.so` shrinks from 377KB to 104KB (!),
and on a Pixel 3 XL app the `ActivityTaskManager: Displayed` time was
reduced from 805ms to 789ms (`$(AndroidEnablePreloadAssemblies)`=True),
a nearly 10% improvement.

Build time is also minimally impacted; `<GenerateJavaStubs/>` task
time is reduced from 389ms to 247ms for the Xamarin.Forms build.


~~ Fast Deployment ~~

When Xamarin.Android Fast Deployment is *not* used for Debug builds
(which is the case for OSS builds of xamarin-android), the typemap
generation and deployment is identical for both Release and Debug
builds: `libxamarin-app.so` contains the new typemap information.

In commercial Xamarin.Android builds which use Fast Deployment, the
typemap data is instead stored in two sets of files:

  * `typemap.index`: stores the mapping from module GUIDs to
    assembly filenames.

  * `*.typemap`: One file per .NET *module*, contain both the JNI-to-
    managed and managed-to-JNI maps, the latter using indexes into
    the Java to managed maps.

All of these files are loaded during Debug app startup and used to
construct a dataset which is then searched during all the lookups.


~~ File Formats ~~

All data in all file formats is little-endian.

The `typemap.index` file stores the mapping from GUIDs to module
filenames such as `Mono.Android.dll`.  The file format in pseudo-C++:

	struct TypemapIndexHeader {
	    byte                magic [4];              // "XATI"
	    uint32_t            format_version;
	    uint32_t            entry_count;
	    uint32_t            module_filename_width;
	    TypemapIndexEntry   entries [entry_count];
	};

	struct TypemapIndexEntry {
	    UUID        module_uuid;  // 16 bytes
	    byte        file_name [TypemapIndexHeader::module_filename_width];
	};

`TypemapIndexHeader::module_filename_width` is the maximum filename
length of any entry within `TypemapIndexEntry::file_name` + 1 for a
terminating `NUL`.

There is no order required within `TypemapIndexHeader::entries`.

`TypemapIndexEntry::file_name` is `NUL` padded, filling the entire
array until the next `TypemapIndexEntry` entry.


The `*.typemap` file stores the mappings from JNI type names to
module GUID and type token pairs.  The file format in pseudo-C++:

	struct TypemapFileHeader {
	    byte                            magic [4];              // "XATM"
	    uint32_t                        format_version;
	    GUID                            module_uuid;
	    uint32_t                        entry_count;
	    uint32_t                        duplicate_count;
	    uint32_t                        jni_name_width;
	    uint32_t                        assembly_name_size;
	    byte                            assembly_name [assembly_name_size];
	    TypemapFileJavaToManagedEntry   java_to_managed [entry_count];
	    TypemapFileManagedToJavaEntry   managed_to_java [entry_count];
	    TypemapFileManagedToJavaEntry   duplicates [duplicate_count];
	};

	struct TypemapFileJavaToManagedEntry {
	    byte        jni_name [TypemapFileHeader::jni_name_width];
	    uint32_t    managed_type_token;
	};

	struct TypemapFileManagedToJavaEntry {
	    uint32_t    managed_type_token;
	    uint32_t    java_to_managed_index;
	};

`TypemapFileHeader::duplicate_count` may be 0.

`TypemapFileJavaToManagedEntry::jni_name` is `NUL` padded.

`TypemapFileJavaToManagedEntry::managed_type_token` is the value of
`Type.MetadataToken`.

`TypemapFileManagedToJavaEntry::java_to_managed_index` is the index
within `TypemapFileHeader::java_to_managed` that contains the JNI name.

[0]: xamarin/monodroid@e69b76e
[1]: https://github.com/xamarin/java.interop/blob/3226a4b57ad84574a69a151a310b077cfe69ee19/src/Java.Interop.Tools.JavaCallableWrappers/Java.Interop.Tools.JavaCallableWrappers/TypeNameMapGenerator.cs#L16-L56

Documentation/guides/messages/README.md

@@ -104,6 +104,7 @@ ms.date: 01/24/2020
 + [XA4305](xa4305.md): MultiDex is enabled, but '{nameof (MultiDexMainDexListFile)}' was not specified.
 + [XA4306](xa4306.md): R8 does not support \`@(MultiDexMainDexList)\` files when android:minSdkVersion >= 21
 + [XA4307](xa4307.md): Invalid ProGuard configuration file.
++ [XA4308](xa4308.md): Failed to generate type maps
 
 ## XA5xxx: GCC and toolchain
 

Documentation/guides/messages/xa4308.md

@@ -0,0 +1,18 @@
+---
+title: Xamarin.Android error XA4308
+description: XA4308 error code
+ms.date: 02/07/2020
+---
+# Xamarin.Android error XA4308
+
+## Issue
+
+The `GenerateJavaStubs` task was unable to generate type maps. Detailed diagnostic will be found before this
+error in the build log.
+
+## Solution
+
+Consider submitting a [bug][bug] if you are getting this warning under
+normal circumstances.
+
+[bug]: https://github.com/xamarin/xamarin-android/wiki/Submitting-Bugs,-Feature-Requests,-and-Pull-Requests

build-tools/xaprepare/xaprepare/Steps/Step_DownloadMonoArchive.cs

@@ -60,26 +60,39 @@ async Task<bool> DownloadMonoArchive (Context context)
 				localPath = Path.Combine (context.Properties.GetRequiredValue (KnownProperties.AndroidToolchainCacheDirectory), archiveFileName);
 			}
 
-			bool result = await DownloadAndUpackIfNeeded (
-				context,
-				"Mono",
-				context.MonoArchiveCustomUrl,
-				localPath,
-				archiveFileName,
-				Configurables.Paths.MonoSDKSOutputDir
-			);
+			bool result = false;
+			for (uint i = 0; i < 3; i++) {
+				result = await DownloadAndUpackIfNeeded (
+					context,
+					"Mono",
+					context.MonoArchiveCustomUrl,
+					localPath,
+					archiveFileName,
+					Configurables.Paths.MonoSDKSOutputDir
+				);
+
+				if (result)
+					break;
+			}
 
 			if (!result)
 				return false;
 
-			return await DownloadAndUpackIfNeeded (
-				context,
-				"Windows Mono",
-				customUrl: null,
-				localPath: Configurables.Paths.MonoArchiveWindowsLocalPath,
-				archiveFileName: Configurables.Paths.MonoArchiveWindowsFileName,
-				destinationDirectory: Configurables.Paths.BCLWindowsOutputDir
-			);
+			for (uint i = 0; i < 3; i++) {
+				result = await DownloadAndUpackIfNeeded (
+					context,
+					"Windows Mono",
+					customUrl: null,
+					localPath: Configurables.Paths.MonoArchiveWindowsLocalPath,
+					archiveFileName: Configurables.Paths.MonoArchiveWindowsFileName,
+					destinationDirectory: Configurables.Paths.BCLWindowsOutputDir
+				);
+
+				if (result)
+					break;
+			}
+
+			return result;
 		}
 
 		async Task<bool> DownloadAndUpackIfNeeded (Context context, string name, string customUrl, string localPath, string archiveFileName, string destinationDirectory)
@@ -109,7 +122,7 @@ async Task<bool> DownloadAndUpackIfNeeded (Context context, string name, string
 				await Download (context, url, localPath, $"{name} Archive", archiveFileName, downloadStatus);
 
 				if (!File.Exists (localPath)) {
-					Log.InfoLine ($"Download of {name} archive from {url} failed, Mono will be rebuilt");
+					Log.InfoLine ($"Download of {name} archive from {url} failed");
 					return false;
 				}
 			}
@@ -118,7 +131,8 @@ async Task<bool> DownloadAndUpackIfNeeded (Context context, string name, string
 			if (!await Utilities.Unpack (localPath, tempDir, cleanDestinatioBeforeUnpacking: true)) {
 				Utilities.DeleteFileSilent (localPath);
 				Utilities.DeleteDirectorySilent (destinationDirectory);
-				Log.WarningLine ($"Failed to unpack {name} archive {localPath}, Mono will be rebuilt");
+				Log.WarningLine ($"Failed to unpack {name} archive {localPath}");
+				Utilities.DeleteFileSilent (localPath);
 				return false;
 			}
 

src/Mono.Android/Android.Runtime/AndroidRuntime.cs

@@ -231,9 +231,9 @@ protected override IEnumerable<Type> GetTypesForSimpleReference (string jniSimpl
 
 		protected override string GetSimpleReference (Type type)
 		{
-			var j = JNIEnv.monodroid_typemap_managed_to_java (type.FullName + ", " + type.Assembly.GetName ().Name);
-			if (j != IntPtr.Zero) {
-				return Marshal.PtrToStringAnsi (j);
+			string j = JNIEnv.TypemapManagedToJava (type);
+			if (j != null) {
+				return j;
 			}
 			if (JNIEnv.IsRunningOnDesktop) {
 				return JavaNativeTypeManager.ToJniName (type);
@@ -243,9 +243,9 @@ protected override string GetSimpleReference (Type type)
 
 		protected override IEnumerable<string> GetSimpleReferences (Type type)
 		{
-			var j = JNIEnv.monodroid_typemap_managed_to_java (type.FullName + ", " + type.Assembly.GetName ().Name);
-			if (j != IntPtr.Zero) {
-				yield return Marshal.PtrToStringAnsi (j);
+			string j = JNIEnv.TypemapManagedToJava (type);
+			if (j != null) {
+				yield return j;
 			}
 			if (JNIEnv.IsRunningOnDesktop) {
 				yield return JavaNativeTypeManager.ToJniName (type);

src/Mono.Android/Android.Runtime/JNIEnv.cs

@@ -57,10 +57,14 @@ public static partial class JNIEnv {
 		static UncaughtExceptionHandler defaultUncaughtExceptionHandler;
 
 		internal static bool IsRunningOnDesktop;
+		internal static bool LogTypemapMissStackTrace;
 
 		static AndroidRuntime androidRuntime;
 		static BoundExceptionType BoundExceptionType;
 
+		[ThreadStatic]
+		static byte[] mvid_bytes;
+
 		internal    static      AndroidValueManager AndroidValueManager;
 
 		[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
@@ -146,6 +150,8 @@ internal static unsafe void Initialize (JnienvInitializeArgs* args)
 				partial_timing_sequence = monodroid_timing_start (null);
 			}
 
+			LogTypemapMissStackTrace = (args->logCategories & (uint)LogCategories.Assembly) != 0;
+
 			gref_gc_threshold = args->grefGcThreshold;
 
 			java_vm = args->javaVm;
@@ -632,16 +638,55 @@ public static string GetClassNameFromInstance (IntPtr jobject)
 		}
 
 		[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
-		internal static extern IntPtr monodroid_typemap_managed_to_java (string managed);
+		static extern IntPtr monodroid_typemap_managed_to_java (byte[] mvid, int token);
+
+		internal static void LogTypemapTrace (StackTrace st)
+		{
+			string trace = st.ToString ()?.Trim ();
+			if (String.IsNullOrEmpty (trace))
+				return;
+
+			monodroid_log (LogLevel.Warn, LogCategories.Assembly, "typemap: called from");
+			foreach (string line in trace.Split ('\n')) {
+				monodroid_log (LogLevel.Warn, LogCategories.Assembly, line);
+			}
+		}
+
+		internal static string TypemapManagedToJava (Type type)
+		{
+			if (mvid_bytes == null)
+				mvid_bytes = new byte[16];
+
+			Span<byte> mvid = new Span<byte>(mvid_bytes);
+			byte[] mvid_slow = null;
+			if (!type.Module.ModuleVersionId.TryWriteBytes (mvid)) {
+				monodroid_log (LogLevel.Warn, LogCategories.Default, $"Failed to obtain module MVID using the fast method, falling back to the slow one");
+				mvid_slow = type.Module.ModuleVersionId.ToByteArray ();
+			}
+
+			IntPtr ret = monodroid_typemap_managed_to_java (mvid_slow == null ? mvid_bytes : mvid_slow, type.MetadataToken);
+
+			if (ret == IntPtr.Zero) {
+				if (LogTypemapMissStackTrace) {
+					monodroid_log (LogLevel.Warn, LogCategories.Default, $"typemap: failed to map managed type to Java type: {type.AssemblyQualifiedName} (Module ID: {type.Module.ModuleVersionId}; Type token: {type.MetadataToken})");
+					LogTypemapTrace (new StackTrace (true));
+				}
+
+				return null;
+			}
+
+			return Marshal.PtrToStringAnsi (ret);
+		}
 
 		public static string GetJniName (Type type)
 		{
 			if (type == null)
 				throw new ArgumentNullException ("type");
-			var java  = monodroid_typemap_managed_to_java (type.FullName + ", " + type.Assembly.GetName ().Name);
-			return java == IntPtr.Zero
+
+			string java = TypemapManagedToJava (type);
+			return java == null
 				? JavaNativeTypeManager.ToJniName (type)
-				: Marshal.PtrToStringAnsi (java);
+				: java;
 		}
 
 		public static IntPtr ToJniHandle (IJavaObject value)

src/Mono.Android/Java.Interop/TypeManager.cs

@@ -2,6 +2,7 @@
 using System.Collections.Generic;
 using System.Reflection;
 using System.Linq;
+using System.Runtime.CompilerServices;
 using System.Runtime.InteropServices;
 using Java.Interop.Tools.TypeNameMappings;
 
@@ -203,22 +204,25 @@ static Exception CreateJavaLocationException ()
 				return new JavaLocationException (loc.ToString ());
 		}
 
-		[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
-		internal static extern IntPtr monodroid_typemap_java_to_managed (string java);
+		[MethodImplAttribute(MethodImplOptions.InternalCall)]
+		static extern Type monodroid_typemap_java_to_managed (string java_type_name);
 
 		internal static Type GetJavaToManagedType (string class_name)
 		{
-			var t = monodroid_typemap_java_to_managed (class_name);
-			if (t != IntPtr.Zero)
-				return Type.GetType (Marshal.PtrToStringAnsi (t));
+			Type type = monodroid_typemap_java_to_managed (class_name);
+			if (type != null)
+				return type;
 
 			if (!JNIEnv.IsRunningOnDesktop) {
+				// Miss message is logged in the native runtime
+				if (JNIEnv.LogTypemapMissStackTrace)
+					JNIEnv.LogTypemapTrace (new System.Diagnostics.StackTrace (true));
 				return null;
 			}
 
 			__TypeRegistrations.RegisterPackages ();
 
-			var type    = (Type) null;
+			type = null;
 			int ls      = class_name.LastIndexOf ('/');
 			var package = ls >= 0 ? class_name.Substring (0, ls) : "";
 			List<Converter<string, Type>> mappers;

src/Mono.Android/Properties/AssemblyInfo.cs.in

@@ -27,3 +27,7 @@ using System.Runtime.CompilerServices;
 [assembly: System.Runtime.CompilerServices.TypeForwardedToAttribute(typeof(System.Drawing.Size))]
 [assembly: System.Runtime.CompilerServices.TypeForwardedToAttribute(typeof(System.Drawing.SizeF))]
 [assembly: System.Runtime.CompilerServices.TypeForwardedToAttribute(typeof(System.Drawing.SystemColors))]
+[assembly: InternalsVisibleTo("Mono.Android-Tests, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
+[assembly: InternalsVisibleTo("Java.Interop-Tests, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
+[assembly: InternalsVisibleTo("Mono.Android-TestsMultiDex, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]
+[assembly: InternalsVisibleTo("Mono.Android-TestsAppBundle, PublicKey=0024000004800000940000000602000000240000525341310004000011000000438ac2a5acfbf16cbd2b2b47a62762f273df9cb2795ceccdf77d10bf508e69e7a362ea7a45455bbf3ac955e1f2e2814f144e5d817efc4c6502cc012df310783348304e3ae38573c6d658c234025821fda87a0be8a0d504df564e2c93b2b878925f42503e9d54dfef9f9586d9e6f38a305769587b1de01f6c0410328b2c9733db")]

src/Mono.Android/Test/Java.Interop-Tests/Java.Interop-Tests.csproj

@@ -17,6 +17,8 @@
     <GenerateSerializationAssemblies>Off</GenerateSerializationAssemblies>
     <TargetFrameworkVersion>v10.0</TargetFrameworkVersion>
     <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
+    <SignAssembly>true</SignAssembly>
+    <AssemblyOriginatorKeyFile>..\..\..\..\product.snk</AssemblyOriginatorKeyFile>
   </PropertyGroup>
   <Import Project="..\..\..\..\Configuration.props" />
   <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
@@ -48,12 +50,11 @@
     <Reference Include="Mono.Android" />
     <Reference Include="Xamarin.Android.NUnitLite" />
     <Reference Include="Mono.Linq.Expressions">
-      <HintPath>..\..\..\..\packages\mono.linq.expressions.2.0.0\lib\netstandard2.0\Mono.Linq.Expressions.dll</HintPath>
+      <HintPath>..\..\..\..\packages\Mono.Linq.Expressions.2.0.0\lib\netstandard2.0\Mono.Linq.Expressions.dll</HintPath>
     </Reference>
   </ItemGroup>
   <ItemGroup>
     <Compile Include="Resources\Resource.designer.cs" />
-    <Compile Include="Properties\AssemblyInfo.cs" />
     <Compile Include="Java.InteropTests\JavaInterop_Tests_Reference.cs" />
   </ItemGroup>
   <ItemGroup>

src/Mono.Android/Test/Java.Interop/JnienvTest.cs

@@ -1,5 +1,6 @@
 using System;
 using System.Linq;
+using System.Runtime.CompilerServices;
 using System.Runtime.InteropServices;
 using System.Threading;
 
@@ -373,32 +374,27 @@ public void MoarThreadingTests ()
 			Assert.IsNull (ignore_t2, string.Format ("No exception should be thrown [t2]! Got: {0}", ignore_t2));
 		}
 
-		[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
-		static extern IntPtr monodroid_typemap_java_to_managed (string java);
-
 		[Test]
 		public void JavaToManagedTypeMapping ()
 		{
-			var m = monodroid_typemap_java_to_managed ("android/content/res/Resources");
-			Assert.AreNotEqual (IntPtr.Zero, m);
-			m = monodroid_typemap_java_to_managed ("this/type/does/not/exist");
-			Assert.AreEqual (IntPtr.Zero, m);
+			Type m = Java.Interop.TypeManager.GetJavaToManagedType ("android/content/res/Resources");
+			Assert.AreNotEqual (null, m);
+			m = Java.Interop.TypeManager.GetJavaToManagedType ("this/type/does/not/exist");
+			Assert.AreEqual (null, m);
 		}
 
 		[DllImport ("__Internal", CallingConvention = CallingConvention.Cdecl)]
-		static extern IntPtr monodroid_typemap_managed_to_java (string java);
-
-		string GetTypeName (Type type)
-		{
-			return type.FullName + ", " + type.Assembly.GetName ().Name;
-		}
+		static extern IntPtr monodroid_typemap_managed_to_java (byte[] mvid, int token);
 
 		[Test]
 		public void ManagedToJavaTypeMapping ()
 		{
-			var m = monodroid_typemap_managed_to_java (GetTypeName (typeof (Activity)));
+			Type type = typeof(Activity);
+			var m = monodroid_typemap_managed_to_java (type.Module.ModuleVersionId.ToByteArray (), type.MetadataToken);
 			Assert.AreNotEqual (IntPtr.Zero, m, "`Activity` subclasses Java.Lang.Object, it should be in the typemap!");
-			m = monodroid_typemap_managed_to_java (GetTypeName (typeof (JnienvTest)));
+
+			type = typeof (JnienvTest);
+			m = monodroid_typemap_managed_to_java (type.Module.ModuleVersionId.ToByteArray (), type.MetadataToken);
 			Assert.AreEqual (IntPtr.Zero, m, "`JnienvTest` does *not* subclass Java.Lang.Object, it should *not* be in the typemap!");
 		}
 

src/Mono.Android/Test/Mono.Android-Test.Library/Mono.Android-Test.Library.csproj

@@ -46,7 +46,6 @@
   </ItemGroup>
   <ItemGroup>
     <Compile Include="Resources\Resource.designer.cs" />
-    <Compile Include="Properties\AssemblyInfo.cs" />
     <Compile Include="CustomTextView.cs" />
   </ItemGroup>
   <ItemGroup>