Bump to mono:2019-10 (xamarin#7192)

* Fixed
`/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/builds/mono-ios-sdk-destdir/ios-sources/external/linker/src/linker/Linker.Steps/OutputStep.cs(110,15): error CS0246: The type or namespace name ‘OutputException’ could not be found (are you missing a using directive or an assembly reference?) [/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tools/mmp/mmp.csproj]`
* Changed the name of the method that is used from linker. Because of this commit dotnet/linker@6be2677
* Added `OutputException.cs` file on `mtouch.csproj`.
* Removing enter_gc_safe and exit_gc_safe because now it's already gc_safe in this part of code, after a mono change.
* Added known exceptions to LLVM exception list.
* Needs `ifdef` because of this mono/mono#17260.
* Bump MIN_MONO_VERSION to 6.8.0.41 and point MIN_MONO_URL to the PR.
* Add ENABLE_IOS=1 and ENABLE_MAC=1.
* Added switch to disable packaged mono build
* [Tests] Ignore tests that fail on 32b.
    Ignore the test on 32b, and filled issue: mono/mono#17752
* [Tests] Ignore a couple of tests causing OOM.
    Hopefully fixes xamarin/maccore#1659 for good.
* Ignore `MM0135` test on Catalina+ because it needs Xcode 9.4.
* [monotouch-test] Add null checks for teardown when test didn't run because of a too early OS version.
* [CFNetwork]: Http 2.0 requires OS X 10.11 or later.
    Check whether `_HTTPVersion2_0` is available and fallback to HTTP 1.1 otherwise.

* xamarin#7346
* This bumps Mono to use mono/mono#17645 (which is the 2019-10 backport
of mono/mono#17628).
* The big user-visible change is in regards to certificate validation, everything below are just
some minor adjustments to tests.

CoreFX uses a completely new `HttpClientHandler` implementation called `SocketsHttpHandler`,
which you can find at https://github.com/dotnet/corefx/tree/release/3.0/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler.

Since this is not based on the web stack anymore, it does not use any of the related APIs such
as `ServicePointManager` or `WebException`.

There is a new API called `HttpClientHandler.ServerCertificateCustomValidationCallback`.
- https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.8
- https://github.com/dotnet/corefx/blob/c1778515a3bee34cc09c757b5563d0af0c8b1e99/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Unix.cs#L154
- https://github.com/dotnet/corefx/blob/c1778515a3bee34cc09c757b5563d0af0c8b1e99/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Windows.cs#L383

The `ServicePointManager.ServerCertificateValidationCallback` is no longer invoked and on
certificate validation failure, `AuthenticationException` (from `System.Security.Authentication`)
is thrown instead of `WebException`.

At the moment, the `NSUrlSessionHandler` still uses it's own validation callback and also still
throws `WebException` on failure; we should probably look into making this consistent with the
other handlers.

* `HttpContent.SerializeToStreamAsync()` is now `protected` (changed from `protected internal`).
  - src/Foundation/NSUrlSessionHandler.cs: changed overload accordingly.
  - src/System.Net.Http/CFContentStream.cs: likewise.

* `HttpHeaders.GetKnownHeaderKind()` is an internal Mono API.
   There is a new internal API called `System.Net.Http.PlatformHelper.IsContentHeader(key)`
   which exists in both the old as well as the new implementation.
   The correct way of doing it with the CoreFX handler is
   `HeaderDescriptor.TryGet (key, out var descriptor) && descriptor.HeaderType == HttpHeaderType.Content`

* `HttpClientHandler.MaxRequestContentBufferSize` is now longer supported, you can set it to
  any non-negative value, the getter will always return 0.
  See https://github.com/dotnet/corefx/blob/c1778515a3bee34cc09c757b5563d0af0c8b1e99/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Core.cs#L18.
  - tests/linker/ios/link sdk/HttpClientHandlerTest.cs: removed assertion from test.

* `HttpMessageInvoker.handler` is a `protected private` field - in the CoreFX handler, it is
  called `_handler` and `private`.  This is accessed via reflection by some of the tests, which are
  now using the new name.
  - tests/mmptest/src/MMPTest.cs: here
  - tests/mtouch/MTouch.cs: here

* tests/monotouch-test/System.Net.Http/MessageHandlers.cs:
  Adjust `RejectSslCertificatesServicePointManager` to reflect the certificate validation
  changes described above.
  - FIXME: There was an `Assert.Ignore()` related to `NSUrlSessionHandler` and macOS 10.10;
    I removed that to reenable the test because the description linked to an old issue in
    the private repo that was referenced by several "Merged" PR's, so it looked to me that
    this might have already been fixed - and I also didn't see why it would fail there.

Make.config

@@ -69,9 +69,10 @@ include $(TOP)/mk/mono.mk
 MONO_HASH := $(NEEDED_MONO_VERSION)
 
 # Minimum Mono version for building XI/XM
-MIN_MONO_VERSION=6.6.0.117
-MAX_MONO_VERSION=6.6.99
-MIN_MONO_URL=https://xamjenkinsartifact.azureedge.net/build-package-osx-mono/2019-08/117/617f399efca133f357cb8207bc7f58b1120f12f1/MonoFramework-MDK-6.6.0.117.macos10.xamarin.universal.pkg
+
+MIN_MONO_VERSION=6.8.0.0
+MAX_MONO_VERSION=6.8.99
+MIN_MONO_URL=https://xamjenkinsartifact.azureedge.net/build-package-osx-mono/2019-10/65/18920a83f423fb864a2263948737681968f5b2c8/MonoFramework-MDK-6.8.0.61.macos10.xamarin.universal.pkg
 
 # Minimum Mono version for Xamarin.Mac apps using the system mono
 MIN_XM_MONO_VERSION=6.4.0.94

builds/Makefile

@@ -109,6 +109,9 @@ $(SDK_CONFIG):
 	echo "DISABLE_ANDROID=1" > $@
 	echo "DISABLE_DESKTOP=1" >> $@
 	echo "DISABLE_WASM=1" >> $@
+	echo "ENABLE_IOS=1" >> $@
+	echo "ENABLE_MAC=1" >> $@
+
 
 ifdef DISABLE_BUILDS_MAKEFILE_DEP
 BUILDS_MAKEFILE_DEP =

mk/mono.mk

@@ -1,5 +1,5 @@
-NEEDED_MONO_VERSION := 062f0ab8cae60d1d347e4d722884c065c9f34484
-NEEDED_MONO_BRANCH := 2019-08
+NEEDED_MONO_VERSION := 18920a83f423fb864a2263948737681968f5b2c8
+NEEDED_MONO_BRANCH := 2019-10
 
 MONO_DIRECTORY := mono
 MONO_MODULE    := https://github.com/mono/mono

runtime/monotouch-main.m

@@ -476,9 +476,7 @@ - (void) memoryWarning: (NSNotification *) sender
 
 		mono_domain_set_config (mono_domain_get (), base_dir, config_file_name);
 
-		MONO_ENTER_GC_SAFE;
 		rv = xamarin_extension_main (argc, argv);
-		MONO_EXIT_GC_SAFE;
 		break;
 	case XamarinLaunchModeApp:
 		rv = mono_jit_exec (mono_domain_get (), assembly, managed_argc, managed_argv);

src/CFNetwork/CFHTTPMessage.cs

@@ -80,9 +80,12 @@ static IntPtr GetVersion (Version version)
 				return _HTTPVersion1_1.Handle;
 			else if (version.Equals (HttpVersion.Version10))
 				return _HTTPVersion1_0.Handle;
-			else if (version.Major == 2 && version.Minor == 0)
-				return _HTTPVersion2_0.Handle;
-			else
+			else if (version.Major == 2 && version.Minor == 0) {
+				if (_HTTPVersion2_0 != null && _HTTPVersion2_0.Handle != IntPtr.Zero)
+					return _HTTPVersion2_0.Handle;
+				// HTTP 2.0 requires OS X 10.11 or later.
+				return _HTTPVersion1_1.Handle;
+			} else
 				throw new ArgumentException ();
 		}
 

src/Foundation/NSUrlSessionHandler.cs

@@ -831,7 +831,7 @@ protected override void Dispose (bool disposing)
 				base.Dispose (disposing);
 			}
 
-			protected internal override Task SerializeToStreamAsync (Stream stream, TransportContext context)
+			protected override Task SerializeToStreamAsync (Stream stream, TransportContext context)
 			{
 				if (contentCopied) {
 					if (!content.CanSeek) {

src/System.Net.Http/CFContentStream.cs

@@ -125,7 +125,7 @@ public void Close ()
 			data_event.Set ();
 		}
 
-		protected internal override async Task SerializeToStreamAsync (Stream stream, TransportContext context)
+		protected override async Task SerializeToStreamAsync (Stream stream, TransportContext context)
 		{
 			while (data_event.WaitOne ()) {
 				data_mutex.WaitOne ();

src/System.Net.Http/CFNetworkHandler.cs

@@ -358,7 +358,7 @@ void HandleHasBytesAvailableEvent (object sender, CFStream.StreamEventArgs e)
 					var key = entry.Key.ToString ();
 					var value = entry.Value == null ? string.Empty : entry.Value.ToString ();
 					HttpHeaders item_headers;
-					if (HttpHeaders.GetKnownHeaderKind (key) == Headers.HttpHeaderKind.Content) {
+					if (HeaderDescriptor.TryGet (key, out var descriptor) && descriptor.HeaderType == HttpHeaderType.Content) {
 						item_headers = response_msg.Content.Headers;
 					} else {
 						item_headers = response_msg.Headers;

tests/bcl-test/common-monotouch_corlib_xunit-test.dll.ignore

@@ -414,4 +414,13 @@ System.SpanTests.ReadOnlySpanTests.BinarySearch_Double
 System.Reflection.Tests.MethodBaseTests.Test_GetCurrentMethod_Inlineable
 
 # device issues, it was reported here: https://github.com/mono/mono/issues/14761
-System.Text.Tests.StringBuilderTests.Append_CharPointer_Null_ThrowsNullReferenceException
+System.Text.Tests.StringBuilderTests.Append_CharPointer_Null_ThrowsNullReferenceException
+
+# mono issue: https://github.com/mono/mono/issues/17752
+# Exception messages: System.OutOfMemoryException : Insufficient memory to continue the execution of the program.
+Platform32:System.Reflection.Tests.RuntimeReflectionExtensionsTests.GetRuntimeMethod
+
+# maccore issue: https://github.com/xamarin/maccore/issues/1659
+# random OOMs
+Platform32:System.Collections.Tests.HashtableTests.Ctor_Int_Int_GenerateNewPrime
+Platform32:System.Tests.BitConverterTests.ToString_ByteArrayTooLong_Throws

tests/bcl-test/macOSModern-xammac_net_4_5_System_xunit-test.dll.ignore

@@ -115,4 +115,4 @@ System.Net.Security.Tests.ApmSslStreamSystemDefaultTest.ClientAndServer_OneOrBot
 System.Net.Security.Tests.ClientDefaultEncryptionTest.ClientDefaultEncryption_ServerRequireEncryption_ConnectWithEncryption
 System.Net.Security.Tests.ClientDefaultEncryptionTest.ClientDefaultEncryption_ServerAllowNoEncryption_ConnectWithEncryption
 System.Net.Security.Tests.ServerAllowNoEncryptionTest.ServerAllowNoEncryption_ClientAllowNoEncryption_ConnectWithEncryption
-KLASS:System.Net.Security.Tests.SslStreamSniTest
+KLASS:System.Net.Security.Tests.SslStreamSniTest 

tests/linker/ios/link sdk/HttpClientHandlerTest.cs

@@ -30,7 +30,6 @@ public void HttpClient ()
 				Assert.That (handler.AutomaticDecompression, Is.EqualTo (DecompressionMethods.None), "AutomaticDecompression");
 				Assert.That (handler.ClientCertificateOptions, Is.EqualTo (ClientCertificateOption.Manual), "ClientCertificateOptions");
 				Assert.That (handler.MaxAutomaticRedirections, Is.EqualTo (50), "MaxAutomaticRedirections");
-				Assert.That (handler.MaxRequestContentBufferSize, Is.EqualTo (Int32.MaxValue), "MaxRequestContentBufferSize");
 				Assert.Null (handler.Proxy, "Proxy");
 				Assert.True (handler.SupportsAutomaticDecompression, "SupportsAutomaticDecompression");
 				Assert.True (handler.SupportsProxy, "SupportsProxy");

tests/mmptest/src/MMPTest.cs

@@ -474,9 +474,9 @@ public void HttpClientHandler (string mmpHandler, string expectedHandler)
 					References = " <Reference Include=\"System.Net.Http\" />",
 					TestCode = $@"
 			var client = new System.Net.Http.HttpClient ();
-			var field = client.GetType ().BaseType.GetField (""handler"", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
+			var field = client.GetType ().BaseType.GetField (""_handler"", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
 			if (field == null)
-				throw new System.Exception (""Could not find the field 'handler' in HttpClient's base type (which should be 'HttpMessageInvoker')."");
+				throw new System.Exception (""Could not find the field '_handler' in HttpClient's base type (which should be 'HttpMessageInvoker')."");
 			var fieldValue = field.GetValue (client);
 			if (fieldValue == null)
 				throw new System.Exception (""Unexpected null value found in 'HttpMessageInvoker.handler' field."");

tests/mmptest/src/WarningTests.cs

@@ -14,6 +14,8 @@ public void MM0135 ()
 
 			if (!Directory.Exists (oldXcode))
 				Assert.Ignore ("This test requires Xcode 9.4 (or updated to a newer one that still warns MM0135).");
+			else if (Environment.OSVersion.Version.Major >= 19 /* macOS 10.15+ */)
+				Assert.Ignore ("Xcode 9.4 does not work on Catalina or later."); // This can check can be removed after switching to a newer Xcode than 9.4.
 
 			MMPTests.RunMMPTest (tmpDir => {
 				TI.UnifiedTestConfig test = new TI.UnifiedTestConfig (tmpDir);

tests/monotouch-test/Network/NWEstablishmentReportTest.cs

@@ -75,8 +75,8 @@ public void Init ()
 		[TestFixtureTearDown]
 		public void Dispose()
 		{
-			report.Dispose ();
-			connection.Dispose ();
+			report?.Dispose ();
+			connection?.Dispose ();
 		}
 
 		[Test]

tests/monotouch-test/Network/NWParametersTest.cs

@@ -58,9 +58,11 @@ public void Init ()
 		[TestFixtureTearDown]
 		public void Dispose()
 		{
-			connection.Dispose ();
-			foreach (var i in interfaces)
-				i.Dispose ();
+			connection?.Dispose ();
+			if (interfaces != null) {
+				foreach (var i in interfaces)
+					i.Dispose ();
+			}
 		}
 
 		[SetUp]

tests/monotouch-test/Network/NWProtocolIPOptionsTest.cs

@@ -66,8 +66,8 @@ public void Init ()
 		[TestFixtureTearDown]
 		public void Dispose()
 		{
-			connection.Dispose ();
-			stack.Dispose ();
+			connection?.Dispose ();
+			stack?.Dispose ();
 		}
 
 		[SetUp]

tests/monotouch-test/Network/NWProtocolStackTest.cs

@@ -56,10 +56,12 @@ public void Init ()
 		[TestFixtureTearDown]
 		public void Dispose()
 		{
-			connection.Dispose ();
-			stack.Dispose ();
-			foreach (var o in options)
-				o.Dispose ();
+			connection?.Dispose ();
+			stack?.Dispose ();
+			if (options != null) {
+				foreach (var o in options)
+					o.Dispose ();
+			}
 		}
 
 		[SetUp]

tests/monotouch-test/System.Net.Http/MessageHandlers.cs

@@ -14,6 +14,7 @@
 
 using NUnit.Framework;
 using System.Net.Http.Headers;
+using System.Security.Authentication;
 using System.Text;
 using Foundation;
 #if MONOMAC
@@ -174,24 +175,35 @@ public void RejectSslCertificatesServicePointManager (Type handlerType)
 				Assert.Ignore ("Fails on macOS 10.10: https://github.com/xamarin/maccore/issues/1645");
 #endif
 
-			bool servicePointManagerCbWasExcuted = false;
+			bool validationCbWasExecuted = false;
+			bool customValidationCbWasExecuted = false;
+			bool invalidServicePointManagerCbWasExcuted = false;
 			bool done = false;
 			Exception ex = null;
+			Type expectedExceptionType = null;
 			HttpResponseMessage result = null;
 
 			var handler = GetHandler (handlerType);
-			if (handler is NSUrlSessionHandler ns) {
-				ns.TrustOverride += (a,b) => {
-					servicePointManagerCbWasExcuted = true;
+			if (handler is HttpClientHandler ch) {
+				expectedExceptionType = typeof (AuthenticationException);
+				ch.ServerCertificateCustomValidationCallback = (sender, certificate, chain, errors) => {
+					validationCbWasExecuted = true;
 					// return false, since we want to test that the exception is raised
 					return false;
 				};
-			} else {
 				ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, errors) => {
-					servicePointManagerCbWasExcuted = true;
+					invalidServicePointManagerCbWasExcuted = true;
+					return false;
+				};
+			} else if (handler is NSUrlSessionHandler ns) {
+				expectedExceptionType = typeof (WebException);
+				ns.TrustOverride += (a,b) => {
+					validationCbWasExecuted = true;
 					// return false, since we want to test that the exception is raised
 					return false;
 				};
+			} else {
+				Assert.Fail ($"Invalid HttpMessageHandler: '{handler.GetType ()}'.");
 			}
 
 			TestRuntime.RunAsync (DateTime.Now.AddSeconds (30), async () =>
@@ -213,11 +225,14 @@ public void RejectSslCertificatesServicePointManager (Type handlerType)
 			if (!done) { // timeouts happen in the bots due to dns issues, connection issues etc.. we do not want to fail
 				Assert.Inconclusive ("Request timedout.");
 			} else {
+				// the ServicePointManager.ServerCertificateValidationCallback will never be executed.
+				Assert.False(invalidServicePointManagerCbWasExcuted);
+				Assert.True(validationCbWasExecuted);
 				// assert the exception type
 				Assert.IsNotNull (ex, (result == null)? "Expected exception is missing and got no result" : $"Expected exception but got {result.Content.ReadAsStringAsync ().Result}");
 				Assert.IsInstanceOfType (typeof (HttpRequestException), ex);
 				Assert.IsNotNull (ex.InnerException);
-				Assert.IsInstanceOfType (typeof (WebException), ex.InnerException);
+				Assert.IsInstanceOfType (expectedExceptionType, ex.InnerException);
 			}
 		}
 

tests/mtouch/MTouch.cs

@@ -3940,9 +3940,9 @@ public class HandlerTest
 	public void Test ()
 	{{
 		var client = new System.Net.Http.HttpClient ();
-		var field = client.GetType ().BaseType.GetField (""handler"", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
+		var field = client.GetType ().BaseType.GetField (""_handler"", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
 		if (field == null)
-			throw new System.Exception (""Could not find the field 'handler' in HttpClient's base type (which should be 'HttpMessageInvoker')."");
+			throw new System.Exception (""Could not find the field '_handler' in HttpClient's base type (which should be 'HttpMessageInvoker')."");
 		var fieldValue = field.GetValue (client);
 		if (fieldValue == null)
 			throw new System.Exception (""Unexpected null value found in 'HttpMessageInvoker.handler' field."");

tests/mtouch/SdkTest.cs

@@ -302,10 +302,23 @@ static string [] GetWatchOSAssemblies ()
 				"LLVM failed for '<SendFrameFallbackAsync>d__56.MoveNext': non-finally/catch/fault clause.",
 				"LLVM failed for '<ReceiveAsyncPrivate>d__61`2.MoveNext': non-finally/catch/fault clause.",
 				"LLVM failed for '<ReceiveAsyncPrivate>d__61`2.MoveNext': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.Read': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.Write': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.BeginRead': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.EndRead': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.BeginWrite': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.EndWrite': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.ReadAsync': non-finally/catch/fault clause.",
+				"LLVM failed for 'NetworkStream.WriteAsync': non-finally/catch/fault clause.",
 			}) },
 			{ "System.Core.dll", new Tuple<int, string[]> (0, new string [] {
 				"LLVM failed for 'EnterTryCatchFinallyInstruction.Run': non-finally/catch/fault clause.",
 			}) },
+			{ "System.Net.Http.dll", new Tuple<int, string[]> (0, new string [] {
+				"LLVM failed for 'HttpContent.CopyToAsync': non-finally/catch/fault clause.",
+				"LLVM failed for 'HttpContent.LoadIntoBufferAsync': non-finally/catch/fault clause.",
+				"LLVM failed for '<CopyToAsyncCore>d__47.MoveNext': non-finally/catch/fault clause.",
+			}) },
 			{ "mscorlib.dll", new Tuple<int, string[]> (0, new string [] {
 				"LLVM failed for 'Console.Write': opcode arglist",
 				"LLVM failed for 'Console.WriteLine': opcode arglist",

tools/linker/MobileSweepStep.cs

@@ -49,12 +49,12 @@ protected void SweepMainModule (ModuleDefinition main)
 			// or they can point to nothing which will break later (e.g. when re-loading for stripping IL)
 			// reference: https://bugzilla.xamarin.com/show_bug.cgi?id=36577
 			if (main.HasExportedTypes)
-				SweepCollectionNonAttributable (main.ExportedTypes);
+				SweepCollectionMetadata (main.ExportedTypes);
 			// only when linking should we remove module references, if we (re)save the assembly then
 			// the entrypoints (for p/invokes) will be required later
 			// reference: https://bugzilla.xamarin.com/show_bug.cgi?id=35372
 			if (main.HasModuleReferences && (CurrentAction == AssemblyAction.Link))
-				SweepCollectionNonAttributable (main.ModuleReferences);
+				SweepCollectionMetadata (main.ModuleReferences);
 		}
 	}
 }

tools/mmp/mmp.csproj

@@ -166,6 +166,9 @@
     <Compile Include="..\..\builds\mono-ios-sdk-destdir\ios-sources\external\linker\src\linker\Linker\XApiReader.cs">
       <Link>Linker\XApiReader.cs</Link>
     </Compile>
+    <Compile Include="..\..\builds\mono-ios-sdk-destdir\ios-sources\external\linker\src\linker\Linker\OutputException.cs">
+      <Link>Linker\OutputException.cs</Link>
+    </Compile>
     <Compile Include="..\..\builds\mono-ios-sdk-destdir\ios-sources\external\linker\src\linker\Linker.Steps\BaseStep.cs">
       <Link>Linker.Steps\BaseStep.cs</Link>
     </Compile>

tools/mtouch/mtouch.csproj

@@ -154,6 +154,9 @@
     <Compile Include="..\..\builds\mono-ios-sdk-destdir\ios-sources\external\linker\src\linker\Linker\MethodAction.cs">
       <Link>Linker\MethodAction.cs</Link>
     </Compile>
+    <Compile Include="..\..\builds\mono-ios-sdk-destdir\ios-sources\external\linker\src\linker\Linker\OutputException.cs">
+      <Link>Linker\OutputException.cs</Link>
+    </Compile>
     <Compile Include="..\..\builds\mono-ios-sdk-destdir\ios-sources\external\linker\src\linker\Linker.Steps\BaseStep.cs">
       <Link>Linker.Steps\BaseStep.cs</Link>
     </Compile>