[ci] Use AZDO built-in parallelization strategy.

[jpobst]

Previously, we split our MSBuildIntegration unit tests to run across multiple CI test agents. While a huge improvement, there are a few downsides to the approach we went with at the time:

• The number of agents is hardcoded by copy/pasting steps in the CI YAML.
• The tests must be manually load-balanced across agents with [Category ("Node-X")] in code which is cumbersome and hard to keep updated.

As we are at the point where our tests have expanded well past the 1 hour target we are faced with needing to increase our parallelization.

This time we're going to invest in a better fix for the above issues.

First, we can remove the YAML duplication by using AZDO's built-in parallel strategy, allowing us to control the number of agents to use with a single number:

  - job: 
    strategy:
      parallel: 4

This leaves us the issue of automatically splitting our tests among an unknown number of test agents. A clever solution is provided in the AZDO docs:

• Use dotnet test --list-tests to find all the tests
• Use a script to calculate which tests the agent should run, based on $(System.JobPositionInPhase) and $(System.TotalJobsInPhase)
• Pass those test names into dotnet test as a filter

Unfortunately there are issues with the provided approach:

• dotnet test --list-tests is not compatible with --filter so you have to run all tests in the test assembly which is not desirable for us. (Option --list-tests should take into account --filter option sdk#8643)
• Passing test names (including test parameters) on the command line hits limitations with escaping certain characters and argument limits.

So the approach is good, but we have to do all the work ourselves. Enter dotnet-test-slicer.

This dotnet global tool:

• Uses the NUnit.Engine NuGet package to find all tests in a test assembly using the desired filter query.
• Slices the test list for the current test agent.
• Outputs the desired tests into an NUnit-specific .runsettings file that can be passed to dotnet test --settings foo.runsettings, bypassing command line arguments.

Voila! Now we can automatically scale our test agents up or down as needed by changing a single variable in our YAML file.

Limitation: The tests are sliced in a round robin fashion. This can produce uneven results if test durations are uneven. A future optimization would be to store approximate test durations so tests can be load balanced more intelligently.

[jonathanpeppers]

I was comparing two PR builds:

• This PR: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=7363883&view=ms.vss-test-web.build-test-results-tab
• Random green PR: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=7367450&view=ms.vss-test-web.build-test-results-tab

This PR ran 182,624 tests and the other one has 182,627 tests.

Is it worth checking if we somehow lost 3 tests? How can we tell?

[pjcollins]

Is it worth checking if we somehow lost 3 tests? How can we tell?

The 3 test difference appears to be in the emulator tests, though I haven't looked through all of them yet:

vs

The additional jobs ultimately use more machine time as well, since we still have a somewhat expensive "provisioning" process for each test job that could use some optimization. Not having to tag every test/fixture with a node is definitely a plus though.

[jpobst]

One of those tests (NativeAssemblyCacheWithSatelliteAssemblies) is added in that PR.

The other two are the permutations of XASdkDeployTests.DotNetDebug ([Values("net6.0-android", "net7.0-android")] string targetFramework), which was set to both "Node-3" and "Node-4", so it is currently being run on both in main:

https://github.com/xamarin/xamarin-android/pull/7804/files#diff-1c999605eb5b10629c064935812b504e78b6b0a8e6530a7beb0f8840a77f0edeL159-L160

The additional jobs ultimately use more machine time as well, since we still have a somewhat expensive "provisioning" process for each test job that could use some optimization.

Yep, this is something I am planning on looking at next.

[jonathanpeppers]

I feel like we could try this in main for a while and see how it goes? Anyone else have concerns?