[build] enable One .NET MSBuild tests #4820

jonathanpeppers · 2020-06-15T15:35:02Z

This will enable all MSBuild tests in
Xamarin.Android.Build.Tests.dll except for the new categories:
DotNetIgnore, AOT, FSharp, LibraryProjectZip, MkBundle,
MonoSymbolicate, PackagesConfig, and StaticProject.

Reasoning:

AOT - need Mono AOT support in .NET 5.
FSharp - Xamarin.Android.FSharp.ResourceProvider support
needed on .NET 5.
LibraryProjectZip - will need future support in Xamarin.Android.
MkBundle - may not ever come to .NET 5.
MonoSymbolicate - need symbolication support in .NET 5.
PackagesConfig - n/a for .NET 5.
StaticProject a few tests that were migrated to
Xamarin.Android.Build.Tests in d447aa6 are ignored for now.

This also reworks our build pipeline to run the MSBuild tests across 12
CI machines:

Windows - Node 1 - Legacy
Windows - Node 2 - Legacy
Windows - Node 3 - Legacy
Windows - Node 1 - One .NET
Windows - Node 2 - One .NET
Windows - Node 3 - One .NET
macOS - Node 1 - Legacy
macOS - Node 2 - Legacy
macOS - Node 3 - Legacy
macOS - Node 1 - One .NET
macOS - Node 2 - One .NET
macOS - Node 3 - One .NET

Since this is a lot of machines:

I moved all the One .NET tests to their own phase.
The phase only runs when RunAllTests=true. This means the One .NET tests will run on master, release branches, and Mono bumps. You can manually queue a build to enable RunAllTests on a PR.

To be completed in another PR, there are still more test assemblies
that need to be run under a dotnet context:

Xamarin.Android.Build.Tests.Commercial.dll
MSBuildDeviceIntegration.dll

jpobst · 2020-08-06T14:29:22Z

My preference would be to move these to a new Test - One .NET stage like Regression Tests is. I think this will help keep the graph more readable, particularly if we're skipping them by default on PR runs:

I would put the stage between Test and Regression Tests.

There are various tests in `Xamarin.Android.Build.Tests` that execute MSBuild tasks directly. This means the changes in dotnet#4820 are not quite complete. We need to actually *run* the tests under .NET Core to see if these tests would pass under a One .NET context. The solution (you would think), would be to make the test assembly `netstandard2.0` so you could run either: > nunit3-console Xamarin.Android.Build.Tests.dll > dotnet test Xamarin.Android.Build.Tests.dll `nunit3-console` would execute the tests under .NET framework/Mono and `dotnet test` would run under .NET Core or .NET 5+. Unfortunately, `nunit3-console` can't run tests from a `netstandard2.0` library! So we multi-target: <TargetFrameworks>net472;net5.0</TargetFrameworks> Unfortunately, the `net5.0` causes MSBuild to fail with: > msbuild Xamarin.Android.sln ... (GetReferenceAssemblyPaths target) -> C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(1177,5): error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks I think there is some expectation to use `dotnet build` for .NET 5 instead. We don't want to move our *entire* build over to `dotnet build` yet (we probably can't!), so instead we can do: <TargetFrameworks>net472;netcoreapp3.1</TargetFrameworks> This allows `msbuild Xamarin.Android.sln` to continue to work. This might also be something that gets fixed in a future MSBuild version. I updated various `.yaml` definitions so we can run NUnit tests via `nunit3-console` or `dotnet test` as needed. Other things that broke here: * We need `Microsoft.NET.Test.Sdk` for `dotnet test` to work. * `Builder.UseDotNet` can now be set based on a `#if` instead of an NUnit parameter. * `Xamarin.Android.Tools.Aidl` needs to be `netstandard2.0`. This required including a `System.CodeDom` package. * `Xamarin.Android.Build.Tasks` can't use `FSharp.Core` in `netcoreapp3.1`. I just removed it, since it is not used. * Updated various paths to `Xamarin.Android.Build.Tests.dll`.

This will enable all MSBuild tests in `Xamarin.Android.Build.Tests.dll` except for the new categories: `DotNetIgnore`, `AOT`, `FSharp`, `LibraryProjectZip`, `MkBundle`, `MonoSymbolicate`, `PackagesConfig`, and `StaticProject`. Reasoning: * `AOT` - need Mono AOT support in .NET 5. * `FSharp` - [Xamarin.Android.FSharp.ResourceProvider][0] support needed on .NET 5. * `LibraryProjectZip` - will need future support in Xamarin.Android. * `MkBundle` - may not ever come to .NET 5. * `MonoSymbolicate` - need symbolication support in .NET 5. * `PackagesConfig` - n/a for .NET 5. * `StaticProject` a few tests that were migrated to `Xamarin.Android.Build.Tests` in d447aa6 are ignored for now. This also reworks our build pipeline to run the MSBuild tests across 12 CI machines: * Windows - Node 1 - Legacy * Windows - Node 2 - Legacy * Windows - Node 3 - Legacy * Windows - Node 1 - One .NET * Windows - Node 2 - One .NET * Windows - Node 3 - One .NET * macOS - Node 1 - Legacy * macOS - Node 2 - Legacy * macOS - Node 3 - Legacy * macOS - Node 1 - One .NET * macOS - Node 2 - One .NET * macOS - Node 3 - One .NET Since this is a lot of machines: 1. I moved all the `One .NET` tests to their own phase. 2. The phase only runs when `RunAllTests=true`. This means the `One .NET` tests will run on master, release branches, and Mono bumps. You can manually queue a build to enable `RunAllTests` on a PR. To be completed in another PR, there are still more test assemblies that need to be run under a `dotnet` context: * `Xamarin.Android.Build.Tests.Commercial.dll` * `MSBuildDeviceIntegration.dll` [0]: https://github.com/xamarin/Xamarin.Android.FSharp.ResourceProvider

jonathanpeppers · 2020-08-06T15:13:02Z

I moved it around so it will look like this now:

I'll need to manually queue one with RunAllTests=true to make sure that works.

jonathanpeppers · 2020-08-06T16:15:25Z

The PR build here looks OK so far, started a RunAllTests=true build here: https://build.azdo.io/3952696

jonpryor · 2020-08-07T14:32:16Z

*Invert* the MSBuild unit tests & One .NET Relationship: previously,
we would selectively *enable* our MSBuild tests to run under .NET 5+.
Now, run *all* tests under .NET 5+, *except* for the categories:

  * `AOT` and `MonoSymbolicate`: requires Mono AOT support in .NET 5
    and packaging of required tooling; see also, but not limited to,
    <https://github.com/dotnet/runtime/issues/35852>
    <https://github.com/dotnet/runtime/issues/36758>

  * `FSharp`: [Xamarin.Android.FSharp.ResourceProvider][0] support
    is needed to run on .NET 5.

  * `LibraryProjectZip`, `StaticProject`: Require future support
    within Xamarin.Android, but don't need to be done *before*
    inverting the unit test regime.

  * `MkBundle`: `mkbundle.exe` might not make it to .NET 5 -- see
    also dotnet/runtime#35852 -- and it may be moot with d236af54.

  * `PackagesConfig`: .NET 5+ won't support `packages.config` files.

Also rework our build pipeline to run the MSBuild tests across 12
CI machines:

  * Windows - Node 1 - Legacy
  * Windows - Node 2 - Legacy
  * Windows - Node 3 - Legacy
  * Windows - Node 1 - One .NET
  * Windows - Node 2 - One .NET
  * Windows - Node 3 - One .NET
  * macOS - Node 1 - Legacy
  * macOS - Node 2 - Legacy
  * macOS - Node 3 - Legacy
  * macOS - Node 1 - One .NET
  * macOS - Node 2 - One .NET
  * macOS - Node 3 - One .NET

Since this is a lot of machines:

 1. I moved all the `One .NET` tests to their own phase.
 2. The phase only runs when `RunAllTests=true`.  This means the
    `One .NET` tests will run on master, release branches, and
    Mono bumps.

    You can manually queue a build to enable `RunAllTests` on a PR.

To be completed in another PR, there are still more test assemblies
that need to be run under a `dotnet` context:

  * `Xamarin.Android.Build.Tests.Commercial.dll`
  * `MSBuildDeviceIntegration.dll`

[0]: https://github.com/xamarin/Xamarin.Android.FSharp.ResourceProvider

There are various tests in `Xamarin.Android.Build.Tests` that execute MSBuild tasks directly. This means the changes in dotnet#4820 are not quite complete. We need to actually *run* the tests under .NET Core to see if these tests would pass under a One .NET context. The solution (you would think), would be to make the test assembly `netstandard2.0` so you could run either: > nunit3-console Xamarin.Android.Build.Tests.dll > dotnet test Xamarin.Android.Build.Tests.dll `nunit3-console` would execute the tests under .NET framework/Mono and `dotnet test` would run under .NET Core or .NET 5+. Unfortunately, `nunit3-console` can't run tests from a `netstandard2.0` library! So we multi-target: <TargetFrameworks>net472;net5.0</TargetFrameworks> Unfortunately, the `net5.0` causes MSBuild to fail with: > msbuild Xamarin.Android.sln ... (GetReferenceAssemblyPaths target) -> C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(1177,5): error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks I think there is some expectation to use `dotnet build` for .NET 5 instead. We don't want to move our *entire* build over to `dotnet build` yet (we probably can't!), so instead we can do: <TargetFrameworks>net472;netcoreapp3.1</TargetFrameworks> This allows `msbuild Xamarin.Android.sln` to continue to work. This might also be something that gets fixed in a future MSBuild version. I updated various `.yaml` definitions so we can run NUnit tests via `nunit3-console` or `dotnet test` as needed. Other things that broke here: * We need `Microsoft.NET.Test.Sdk` for `dotnet test` to work. * `Builder.UseDotNet` can now be set based on a `#if` instead of an NUnit parameter. * `Xamarin.Android.Tools.Aidl` needs to be `netstandard2.0`. This required including a `System.CodeDom` package. * `Xamarin.Android.Build.Tasks` can't use `FSharp.Core` in `netcoreapp3.1`. I just removed it, since it is not used. * Updated various paths to `Xamarin.Android.Build.Tests.dll`.

There are various tests in `Xamarin.Android.Build.Tests` that execute MSBuild tasks directly. This means the changes in dotnet#4820 are not quite complete. We need to actually *run* the tests under .NET Core to see if these tests would pass under a One .NET context. The solution (you would think), would be to make the test assembly `netstandard2.0` so you could run either: > nunit3-console Xamarin.Android.Build.Tests.dll > dotnet test Xamarin.Android.Build.Tests.dll `nunit3-console` would execute the tests under .NET framework/Mono and `dotnet test` would run under .NET Core or .NET 5+. Unfortunately, `nunit3-console` can't run tests from a `netstandard2.0` library! So we multi-target: <TargetFrameworks>net472;net5.0</TargetFrameworks> Unfortunately, the `net5.0` causes MSBuild to fail with: > msbuild Xamarin.Android.sln ... (GetReferenceAssemblyPaths target) -> C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(1177,5): error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks I think there is some expectation to use `dotnet build` for .NET 5 instead. We don't want to move our *entire* build over to `dotnet build` yet (we probably can't!), so instead we can do: <TargetFrameworks>net472;netcoreapp3.1</TargetFrameworks> This allows `msbuild Xamarin.Android.sln` to continue to work. This might also be something that gets fixed in a future MSBuild version. I updated various `.yaml` definitions so we can run NUnit tests via `nunit3-console` or `dotnet test` as needed. Other things that broke here: * We need `Microsoft.NET.Test.Sdk` for `dotnet test` to work. * `Builder.UseDotNet` can now be set based on a `#if` instead of an NUnit parameter. * `Xamarin.Android.Tools.Aidl` needs to be `netstandard2.0`. This required including a `System.CodeDom` package. * `Xamarin.Android.Build.Tasks` can't use `FSharp.Core` in `netcoreapp3.1`. I just removed it, since it is not used. * Updated various paths to `Xamarin.Android.Build.Tests.dll`. * Usage of System.Drawing crashed on macOS. I replaced this with ImageSharp: https://www.nuget.org/packages/SixLabors.ImageSharp/

There are various tests in `Xamarin.Android.Build.Tests` that execute MSBuild tasks directly. This means the changes in #4820 are not quite complete. We need to actually *run* the tests under .NET Core to see if these tests would pass under a One .NET context. The solution (you would think), would be to make the test assembly `netstandard2.0` so you could run either: > nunit3-console Xamarin.Android.Build.Tests.dll > dotnet test Xamarin.Android.Build.Tests.dll `nunit3-console` would execute the tests under .NET framework/Mono and `dotnet test` would run under .NET Core or .NET 5+. Unfortunately, `nunit3-console` can't run tests from a `netstandard2.0` library! So we multi-target: <TargetFrameworks>net472;net5.0</TargetFrameworks> Unfortunately, the `net5.0` causes MSBuild to fail with: > msbuild Xamarin.Android.sln ... (GetReferenceAssemblyPaths target) -> C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(1177,5): error MSB3644: The reference assemblies for .NETFramework,Version=v5.0 were not found. To resolve this, install the Developer Pack (SDK/Targeting Pack) for this framework version or retarget your application. You can download .NET Framework Developer Packs at https://aka.ms/msbuild/developerpacks I think there is some expectation to use `dotnet build` for .NET 5 instead. We don't want to move our *entire* build over to `dotnet build` yet (we probably can't!), so instead we can do: <TargetFrameworks>net472;netcoreapp3.1</TargetFrameworks> This allows `msbuild Xamarin.Android.sln` to continue to work. This might also be something that gets fixed in a future MSBuild version. I updated various `.yaml` definitions so we can run NUnit tests via `nunit3-console` or `dotnet test` as needed. Other things that broke here: * We need `Microsoft.NET.Test.Sdk` for `dotnet test` to work. * `Builder.UseDotNet` can now be set based on a `#if` instead of an NUnit parameter. * `Xamarin.Android.Tools.Aidl` needs to be `netstandard2.0`. This required including a `System.CodeDom` package. * `Xamarin.Android.Build.Tasks` can't use `FSharp.Core` in `netcoreapp3.1`. I just removed it, since it is not used. * Updated various paths to `Xamarin.Android.Build.Tests.dll`. * Usage of System.Drawing crashed on macOS. I replaced this with ImageSharp: https://www.nuget.org/packages/SixLabors.ImageSharp/

jonathanpeppers changed the title ~~[build] enable all One .NET MSBuild tests~~ [build] enable One .NET MSBuild tests Jun 15, 2020

jonathanpeppers force-pushed the dotnet-msbuild-tests branch 2 times, most recently from 3abc19c to 13aa00a Compare June 23, 2020 13:45

jonathanpeppers force-pushed the dotnet-msbuild-tests branch 5 times, most recently from 7676bbf to 3570fae Compare July 1, 2020 19:57

jonathanpeppers force-pushed the dotnet-msbuild-tests branch 4 times, most recently from 73b9e93 to 639062f Compare July 13, 2020 18:09

jonathanpeppers force-pushed the dotnet-msbuild-tests branch 9 times, most recently from 23dcb4c to 95dc5b8 Compare July 21, 2020 14:54

jonathanpeppers force-pushed the dotnet-msbuild-tests branch 9 times, most recently from 07d4864 to 5cc8ed7 Compare July 23, 2020 19:28

jonathanpeppers force-pushed the dotnet-msbuild-tests branch from 3695c2b to 015daae Compare August 6, 2020 15:11

jpobst approved these changes Aug 6, 2020

View reviewed changes

jonpryor merged commit 3c4c8de into dotnet:master Aug 7, 2020

jonathanpeppers deleted the dotnet-msbuild-tests branch August 7, 2020 14:35

brendanzagaeski added this to the In Preview: 11.2 (d16-9) milestone Nov 10, 2020

github-actions bot locked and limited conversation to collaborators Jan 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build] enable One .NET MSBuild tests #4820

[build] enable One .NET MSBuild tests #4820

jonathanpeppers commented Jun 15, 2020 •

edited

Loading

jpobst commented Aug 6, 2020

jonathanpeppers commented Aug 6, 2020

jonathanpeppers commented Aug 6, 2020

jonpryor commented Aug 7, 2020 •

edited

Loading

[build] enable One .NET MSBuild tests #4820

[build] enable One .NET MSBuild tests #4820

Conversation

jonathanpeppers commented Jun 15, 2020 • edited Loading

jpobst commented Aug 6, 2020

jonathanpeppers commented Aug 6, 2020

jonathanpeppers commented Aug 6, 2020

jonpryor commented Aug 7, 2020 • edited Loading

jonathanpeppers commented Jun 15, 2020 •

edited

Loading

jonpryor commented Aug 7, 2020 •

edited

Loading