[Mono.Android] Generate Mono.Android.xml

Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus are woefully out of date.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained a `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to extract the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `anddroid-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)

.gitmodules

@@ -12,8 +12,8 @@
     branch = master
 [submodule "external/Java.Interop"]
     path = external/Java.Interop
-    url = https://github.com/xamarin/java.interop.git
-    branch = master
+    url = https://github.com/jonpryor/java.interop.git
+    branch = jonp-generator-javadoc-xml
 [submodule "external/lz4"]
     path = external/lz4
     url = https://github.com/lz4/lz4.git

Xamarin.Android.sln

@@ -142,6 +142,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Xamarin.SourceWriter", "ext
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "apksigner", "src\apksigner\apksigner.csproj", "{9A9EF774-6EA6-414F-9D2F-DCD66C56B92A}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "java-source-utils", "external\Java.Interop\tools\java-source-utils\java-source-utils.csproj", "{37FCD325-1077-4603-98E7-4509CAD648D6}"
+EndProject
 Global
 	GlobalSection(SharedMSBuildProjectFiles) = preSolution
 		src\Xamarin.Android.NamingCustomAttributes\Xamarin.Android.NamingCustomAttributes.projitems*{3f1f2f50-af1a-4a5a-bedb-193372f068d7}*SharedItemsImports = 4
@@ -376,7 +378,7 @@ Global
 		{071D9096-65BB-4359-822E-09788439F210}.Debug|AnyCPU.ActiveCfg = Debug|Any CPU
 		{071D9096-65BB-4359-822E-09788439F210}.Debug|AnyCPU.Build.0 = Debug|Any CPU
 		{071D9096-65BB-4359-822E-09788439F210}.Release|AnyCPU.ActiveCfg = Release|Any CPU
-		{071D9096-65BB-4359-822E-09788439F210}.Release|AnyCPU.Build.0 = Release|Any CPU	EndGlobalSection
+		{071D9096-65BB-4359-822E-09788439F210}.Release|AnyCPU.Build.0 = Release|Any CPU
 		{2CE4CD4B-B7B7-4EAE-A9BE-2699824D6096}.Debug|AnyCPU.ActiveCfg = Debug|Any CPU
 		{2CE4CD4B-B7B7-4EAE-A9BE-2699824D6096}.Debug|AnyCPU.Build.0 = Debug|Any CPU
 		{2CE4CD4B-B7B7-4EAE-A9BE-2699824D6096}.Release|AnyCPU.ActiveCfg = Release|Any CPU
@@ -389,6 +391,10 @@ Global
 		{9A9EF774-6EA6-414F-9D2F-DCD66C56B92A}.Debug|AnyCPU.Build.0 = Debug|Any CPU
 		{9A9EF774-6EA6-414F-9D2F-DCD66C56B92A}.Release|AnyCPU.ActiveCfg = Release|Any CPU
 		{9A9EF774-6EA6-414F-9D2F-DCD66C56B92A}.Release|AnyCPU.Build.0 = Release|Any CPU
+		{37FCD325-1077-4603-98E7-4509CAD648D6}.Debug|AnyCPU.ActiveCfg = Debug|Any CPU
+		{37FCD325-1077-4603-98E7-4509CAD648D6}.Debug|AnyCPU.Build.0 = Debug|Any CPU
+		{37FCD325-1077-4603-98E7-4509CAD648D6}.Release|AnyCPU.ActiveCfg = Release|Any CPU
+		{37FCD325-1077-4603-98E7-4509CAD648D6}.Release|AnyCPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -454,6 +460,7 @@ Global
 		{2CE4CD4B-B7B7-4EAE-A9BE-2699824D6096} = {04E3E11E-B47D-4599-8AFC-50515A95E715}
 		{86A8DEFE-7ABB-4097-9389-C249581E243D} = {04E3E11E-B47D-4599-8AFC-50515A95E715}
 		{9A9EF774-6EA6-414F-9D2F-DCD66C56B92A} = {04E3E11E-B47D-4599-8AFC-50515A95E715}
+		{37FCD325-1077-4603-98E7-4509CAD648D6} = {864062D3-A415-4A6F-9324-5820237BA058}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {53A1F287-EFB2-4D97-A4BB-4A5E145613F6}

build-tools/installers/create-installers.targets

@@ -132,6 +132,7 @@
     <_MSBuildFiles Include="$(MSBuildSrcDir)\illinkanalyzer.pdb"  ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Irony.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\java-interop.jar" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\java-source-utils.jar" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\javadoc-to-mdoc.exe" ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\javadoc-to-mdoc.pdb" ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Interop.dll" />
@@ -149,6 +150,8 @@
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Interop.Tools.Generator.pdb"  ExcludeFromAndroidNETSdk="true" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Interop.Tools.JavaCallableWrappers.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Interop.Tools.JavaCallableWrappers.pdb" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Interop.Tools.JavaSource.dll" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Interop.Tools.JavaSource.pdb" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Runtime.Environment.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Runtime.Environment.pdb" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Java.Runtime.Environment.dll.config" Condition=" '$(HostOS)' != 'Windows' " />

build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/AndroidToolchain.cs

@@ -62,6 +62,8 @@ public AndroidToolchain ()
 				new AndroidPlatformComponent ("platform-29_r01",   apiLevel: "29", pkgRevision: "1"),
 				new AndroidPlatformComponent ("platform-30_r01",   apiLevel: "30", pkgRevision: "1"),
 
+				new AndroidToolchainComponent ("sources-30_r01",   destDir: Path.Combine ("platforms", $"android-30", "src"), pkgRevision: "1", dependencyType: AndroidToolchainComponentType.BuildDependency),
+
 				new AndroidToolchainComponent ("docs-24_r01",                                       destDir: "docs", pkgRevision: "1", dependencyType: AndroidToolchainComponentType.BuildDependency),
 				new AndroidToolchainComponent ("android_m2repository_r47",                          destDir: Path.Combine ("extras", "android", "m2repository"), pkgRevision: "47.0.0", dependencyType: AndroidToolchainComponentType.BuildDependency),
 				new AndroidToolchainComponent ($"x86_64-29_r07-{osTag}",                            destDir: Path.Combine ("system-images", "android-29", "default", "x86_64"), relativeUrl: new Uri ("sys-img/android/", UriKind.Relative), pkgRevision: "7", dependencyType: AndroidToolchainComponentType.EmulatorDependency),

build-tools/xaprepare/xaprepare/ThirdPartyNotices/Java.Interop.cs

@@ -12,6 +12,7 @@ class JavaInterop_External_Dependencies_Group : ThirdPartyNoticeGroup
 		public override List<ThirdPartyNotice> Notices => new List <ThirdPartyNotice> {
 			new JavaInterop_xamarin_Java_Interop_TPN (),
 			new JavaInterop_gityf_crc_TPN (),
+			new JavaInterop_javaparser_javaparser_TPN (),
 			new JavaInterop_jbevain_mono_linq_expressions_TPN (),
 			new JavaInterop_mono_csharp_TPN (),
 			new JavaInterop_mono_LineEditor_TPN (),
@@ -69,6 +70,18 @@ POSSIBILITY OF SUCH DAMAGE.
 ";
 	}
 
+	// via: https://github.com/xamarin/java.interop/blob/b588ef502d8d3b4c32e0ad731115e1b71fd56b5c/tools/java-source-utils/build.gradle#L33-L34
+	class JavaInterop_javaparser_javaparser_TPN : ThirdPartyNotice
+	{
+		static readonly Uri    url         = new Uri ("https://github.com/javaparser/javaparser/");
+		static readonly string licenseFile = Path.Combine (Configurables.Paths.ExternalJavaInteropDir, "LICENSE");
+
+		public override string LicenseFile => CommonLicenses.Apache20Path;
+		public override string Name        => "javaparser/javaparser";
+		public override Uri    SourceUrl   => url;
+		public override string LicenseText => String.Empty;
+	}
+
 	// git submodules of Java.Interop
 	class JavaInterop_jbevain_mono_linq_expressions_TPN : ThirdPartyNotice
 	{

src/Mono.Android/Mono.Android.csproj

@@ -24,6 +24,16 @@
     <ProduceReferenceAssembly>true</ProduceReferenceAssembly>
   </PropertyGroup>
 
+  <PropertyGroup>
+    <!-- TODO: Default to False? Depends on impact on build times -->
+    <IncludeAndroidJavadoc Condition=" '$(IncludeAndroidJavadoc)' == '' ">True</IncludeAndroidJavadoc>
+  </PropertyGroup>
+
+  <PropertyGroup Condition=" '$(IncludeAndroidJavadoc)' == 'True' ">
+    <DocumentationFile>$(OutputPath)Mono.Android.xml</DocumentationFile>
+    <NoWarn>CS1573;CS1591</NoWarn>
+  </PropertyGroup>
+
   <PropertyGroup Condition=" '$(TargetFramework)' == 'monoandroid10' ">
     <TargetFrameworkIdentifier>MonoAndroid</TargetFrameworkIdentifier>
     <TargetFrameworkVersion>v1.0</TargetFrameworkVersion>
@@ -344,7 +354,8 @@
     <!-- Explicitly pass the target framework of the project so we don't have conflicts with the multiple targets in this file. -->
     <ProjectReference Include="..\..\build-tools\api-merge\api-merge.csproj" ReferenceOutputAssembly="false" SkipGetTargetFrameworkProperties="true" AdditionalProperties="TargetFramework=net472" />
     <ProjectReference Include="..\..\build-tools\jnienv-gen\jnienv-gen.csproj" ReferenceOutputAssembly="false" SkipGetTargetFrameworkProperties="true" AdditionalProperties="TargetFramework=net472" />
-    <ProjectReference Include="..\..\external\Java.Interop\tools\generator\generator.csproj" ReferenceOutputAssembly="false" SkipGetTargetFrameworkProperties="true" AdditionalProperties="TargetFramework=net472"/>
+    <ProjectReference Include="..\..\external\Java.Interop\tools\generator\generator.csproj" ReferenceOutputAssembly="false" SkipGetTargetFrameworkProperties="true" AdditionalProperties="TargetFramework=net472" />
+    <ProjectReference Include="..\..\external\Java.Interop\tools\java-source-utils\java-source-utils.csproj" ReferenceOutputAssembly="false" SkipGetTargetFrameworkProperties="true" AdditionalProperties="TargetFramework=net472" />
     <ProjectReference Include="..\..\external\Java.Interop\tools\jcw-gen\jcw-gen.csproj" ReferenceOutputAssembly="false" SkipGetTargetFrameworkProperties="true" AdditionalProperties="TargetFramework=net472" />
     <ProjectReference Include="..\..\src\java-runtime\java-runtime.csproj" ReferenceOutputAssembly="false" />
     <ProjectReference Include="..\r8\r8.csproj" ReferenceOutputAssembly="False" />

src/Mono.Android/Mono.Android.targets

@@ -53,6 +53,50 @@
         Replacements="@PACKAGE_VERSION@=$(_PackageVersion);@PACKAGE_VERSION_BUILD@=$(_PackageVersionBuild);@PACKAGE_HEAD_REV@=$(XAVersionHash);@PACKAGE_HEAD_BRANCH@=$(XAVersionBranch)">
     </ReplaceFileContents>
   </Target>
+  <PropertyGroup>
+    <_JavaSourceUtilsJar>$(XAInstallPrefix)xbuild\Xamarin\Android\java-source-utils.jar</_JavaSourceUtilsJar>
+    <_AndroidStableSrcDir>$(AndroidSdkDirectory)\platforms\android-$(AndroidLatestStableApiLevel)\src</_AndroidStableSrcDir>
+    <_AndroidJavadocXml>..\..\bin\Build$(Configuration)\android-javadoc.xml</_AndroidJavadocXml>
+  </PropertyGroup>
+  <Target Name="_BuildAndroidJavadocXml"
+      Condition=" '$(IncludeAndroidJavadoc)' == 'True' "
+      BeforeTargets="CoreCompile"
+      Inputs="$(MSBuildThisFile);$(_AndroidStableSrcDir)\source.properties;$(_JavaSourceUtilsJar)"
+      Outputs="$(_AndroidJavadocXml)">
+    <ItemGroup>
+      <_Doclink Include="--doc-copyright" />
+      <_Doclink Include="$(MSBuildThisFileDirectory)javadoc-copyright.xml" />
+      <_Doclink Include="--doc-url-prefix" />
+      <_Doclink Include="https://developer.android.com/reference" />
+      <_Doclink Include="--doc-url-style" />
+      <_Doclink Include="developer.android.com/reference@2020-Nov" />
+    </ItemGroup>
+    <ItemGroup>
+      <_AndroidSources Include="$(_AndroidStableSrcDir)\android\**\*.java" />
+      <_AndroidSources Include="$(_AndroidStableSrcDir)\java\**\*.java" />
+      <_AndroidSources Include="$(_AndroidStableSrcDir)\javax\**\*.java" />
+      <_AndroidSources Include="$(_AndroidStableSrcDir)\org\**\*.java" />
+      <_AndroidSources Remove="$(_AndroidStableSrcDir)\**\*.annotated.java" />
+    </ItemGroup>
+    <PropertyGroup>
+      <_Filenames>$(IntermediateOutputPath)\java-sources.txt</_Filenames>
+    </PropertyGroup>
+    <WriteLinesToFile
+       File="$(_Filenames)"
+       Lines="@(_Doclink);@(_AndroidSources)"
+       Overwrite="True"
+    />
+    <ItemGroup>
+      <_JSIArg Include="-v" />
+      <_JSIArg Include="--source &quot;$(_AndroidStableSrcDir)&quot;" />
+      <_JSIArg Include="--output-javadoc &quot;$(_AndroidJavadocXml)&quot;" />
+      <_JSIArg Include="@$(_Filenames)" />
+    </ItemGroup>
+    <Exec
+        Command="&quot;$(JavaPath)&quot; -jar &quot;$(_JavaSourceUtilsJar)&quot; @(_JSIArg, ' ')"
+    />
+    <Touch Files="$(_AndroidJavadocXml)" />
+  </Target>
   <Target Name="_BuildJNIEnv"
       BeforeTargets="CoreCompile"
       Inputs="..\..\bin\Build$(Configuration)\jnienv-gen.exe"
@@ -113,12 +157,14 @@
       <_TypeMap>--type-map-report=$(IntermediateOutputPath)mcw\type-mapping.txt</_TypeMap>
       <_Api>$(IntermediateOutputPath)mcw\api.xml</_Api>
       <_Dirs>--enumdir=$(IntermediateOutputPath)mcw</_Dirs>
+      <_WithJavadocXml Condition=" '$(IncludeAndroidJavadoc)' == 'True' ">"--with-javadoc-xml=$(_AndroidJavadocXml)"</_WithJavadocXml>
       <_FullIntermediateOutputPath>$([System.IO.Path]::GetFullPath('$(IntermediateOutputPath)'))</_FullIntermediateOutputPath>
       <_LangFeatures>--lang-features=nullable-reference-types</_LangFeatures>
       <_LangFeatures Condition="$(AndroidApiLevel) &gt;= 30">$(_LangFeatures),default-interface-methods,nested-interface-types,interface-constants</_LangFeatures>
     </PropertyGroup>
     <Exec
-        Command="$(ManagedRuntime) $(ManagedRuntimeArgs) $(Generator) $(_GenFlags) $(_ApiLevel) $(_Out) $(_Codegen) $(_Fixup) $(_Enums1) $(_Enums2) $(_Versions) $(_Annotations) $(_Assembly) $(_TypeMap) $(_LangFeatures) $(_Dirs) $(_Api)"
+        Command="$(ManagedRuntime) $(ManagedRuntimeArgs) $(Generator) $(_GenFlags) $(_ApiLevel) $(_Out) $(_Codegen) $(_Fixup) $(_Enums1) $(_Enums2) $(_Versions) $(_Annotations) $(_Assembly) $(_TypeMap) $(_LangFeatures) $(_Dirs) $(_Api) $(_WithJavadocXml)"
+        IgnoreStandardErrorWarningFormat="True"
     />
     <ItemGroup>
       <Compile Include="$(_FullIntermediateOutputPath)\mcw\**\*.cs" KeepDuplicates="False" />

src/Mono.Android/javadoc-copyright.xml

@@ -0,0 +1,11 @@
+<para>
+  Portions of this page are modifications based on work created
+  and shared by the
+  <format type="text/html">
+    <a href="https://developers.google.com/terms/site-policies" title="Android Open Source Project">Android Open Source Project</a>
+  </format>
+  and used according to terms described in the
+  <format type="text/html">
+    <a href="https://creativecommons.org/licenses/by/2.5/" title="Creative Commons 2.5 Attribution License">Creative Commons 2.5 Attribution License.</a>
+  </format>
+</para>