Add JIT stress mode testing using libraries tests #36486

BruceForstall · 2020-05-15T02:29:57Z

Add JIT stress mode testing using libraries tests

There are three separate pipelines, with different sets of JIT
stress modes enabled for each:

libraries-jitstress.yml
libraries-jitstress2-jitstressregs.yml
libraries-jitstressregs.yml

The live built Release libraries are used with a Checked CoreCLR.

The non-stress configuration is tested during normal PR testing.

The libraries sendtohelix.proj file is changed to take an optional
list of CoreCLR stress scenarios to run. If given, a set of environment
setting scripts is created and added to the Helix correlation payload.
Then, an auxiliary sendtohelixhelp.proj file is invoked for each
scenario (or just the default, empty scenario) to create and run the
Helix tasks.

Remove remnants of old corefx-on-coreclr test run infrastructure.

Disable tests:
src/libraries/System.Numerics.Vectors/tests/Matrix3x2Tests.cs
#36587
Matrix3x2CreateRotationCenterTest()

#36587
Matrix3x2CreateScaleCenterTest1()

#36587
Matrix3x2CreateScaleCenterTest3()

src/libraries/System.Numerics.Vectors/tests/Matrix4x4Tests.cs
#36586
Matrix4x4CreateFromAxisAngleTest()

ghost · 2020-05-15T02:30:02Z

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

ViktorHofer · 2020-05-15T06:07:05Z

Would be nice if we could get rid of the corefx and replace it with libraries there.

BruceForstall · 2020-05-16T04:48:21Z

@safern @ViktorHofer This PR will enable 3 new AzDO pipelines to test checked CoreCLR with various stress modes against the libraries test assets. Because these haven't been run in 6-9 months, there are some test failures. How should I disable these failing tests temporarily? Based on https://github.com/dotnet/runtime/blob/master/docs/workflow/testing/libraries/filtering-tests.md it looks like I should annotate the failing tests with something like [ActiveIssue("https://github.com/dotnet/runtime/issues/33993")], but that will disable it for all runs, not just JIT stress mode runs. That seems ok, but maybe there is something more specific I should use?

safern · 2020-05-16T06:31:01Z

We should use SkipOnCoreClrAttribute that one has support for adding a string with an issue and also to disable on specific stress scenarios.

BruceForstall · 2020-05-18T02:28:55Z

Would be nice if we could get rid of the corefx and replace it with libraries there.

I're renamed the pipelines. I'd like to rename the files as a separate PR that only does the rename.

BruceForstall · 2020-05-18T02:32:13Z

This is ready to be reviewed. I've triggered the pipelines, and they all seem to work as expected. I've had to up the timeout. I don't know if that's because I can't create/submit the helix jobs all at once due to a Helix synchronization issue, or if the jobs just take a long time. There are still some test failures that I need to repro and disable, if possible, especially on arm/arm64.

safern · 2020-05-19T18:22:13Z

src/libraries/sendtohelix-scenarios.proj

+      <_Scenarios Include="$(_Scenarios.Split(','))" />
+
+      <!-- MSBuild creates a new instance of the project for each %(_Scenarios.Identity) and can build them in parallel. -->
+      <_ProjectsToBuild Include="$(MSBuildProjectFile)">


Why not instead use sendtohelix.proj? that way we don't have to duplicate all the helix targets and properties?

I would want to avoid so much duplication of the code as, as we move and add more features we would need to maintain both this projects (for example to add more helix data to kusto info, or other stuff).

The issue is that my new sendtohelix-scenarios.proj is quite a bit more complicated than the existing sendtohelix.proj. So I did it as-is to avoid perturbing (and potentially breaking) the existing queues, at least while I was doing development,

I can certainly make everyone use my new system, and have the existing pipelines use a default or empty scenario list. This would require more testing (maybe drastically more testing) to ensure existing pipelines aren't broken.

I didn't mean augmenting sendtohelix.proj -- I meant, calling into sendtohelix.proj from sendtohelix-scenarios.proj.

So basically sendtohelix-scenarios does the fan out of the scenarios and expands into sendtohelix.proj foreach scenario present. Then that way we avoid much duplication.

Then stuff specific to multiple scenarios like env vars and stuff like that, we could put it in a targets file and import that file when scenario is not null in sendtohelix.proj. Does that makes it easier/cleaner?

I'll look into this; I've thought before about whether that would be easier, but was following the previous model (this same basic code exists in the coreclr sendtohelix equivalent).

Yeah, but now that we share the same sources, I think it makes sense to avoid duplication and this brings a lot of duplication, like the targets to zip the testhost and to build the helix workitems, etc.

jashook · 2020-05-19T18:23:27Z

/cc @naricc

safern · 2020-05-19T18:24:14Z

eng/pipelines/libraries/helix-scenarios.yml

@@ -0,0 +1,34 @@
+parameters:


I don't think we should introduce a new template just for this.

Maybe make the project a parameter?

parameters: helixProject: '$(Build.SourcesDirectory)/src/libraries/sendtohelix.proj' ... steps: - script: ${{ parameters.msbuildScript }} ${{ parameters.helixProject }} ...

I could do this, but if we decide to just augment the existing sendtohelix.proj with the new scenarios functionality, it won't be necessary.

There are three separate pipelines, with different sets of JIT stress modes enabled for each: - libraries-jitstress.yml - libraries-jitstress2-jitstressregs.yml - libraries-jitstressregs.yml The live built Release libraries are used with a Checked CoreCLR. The non-stress configuration is tested during normal PR testing. The libraries `sendtohelix.proj` file is changed to take an optional list of CoreCLR stress scenarios to run. If given, a set of environment setting scripts is created and added to the Helix correlation payload. Then, an auxiliary `sendtohelixhelp.proj` file is invoked for each scenario (or just the default, empty scenario) to create and run the Helix tasks.

``` src/libraries/System.Numerics.Vectors/tests/Matrix3x2Tests.cs dotnet#36587 Matrix3x2CreateRotationCenterTest() dotnet#36587 Matrix3x2CreateScaleCenterTest1() dotnet#36587 Matrix3x2CreateScaleCenterTest3() src/libraries/System.Numerics.Vectors/tests/Matrix4x4Tests.cs dotnet#36586 Matrix4x4CreateFromAxisAngleTest() ```

BruceForstall · 2020-06-12T04:41:14Z

@safern @ViktorHofer @jashook This is ready to be reviewed again.

The main thing I've done is split the libraries sendtohelix.proj into two parts:

the "driver", still in sendtohelix.proj. This constructs and compresses the Helix correlation payload. In the case where a set of coreclr stress scenarios is given, it also constructs a set of environment-setting batch files to use, and includes them all in the payload (they are named by the scenario name).
the part that creates Helix work items and causes the jobs to be submitted. I put this in sendtohelixhelp.proj. This is called once per scenario (in parallel).

All libraries jobs go through this path, and the "no scenarios given" case, which is the default, should behave as before.

I've disabled a few tests, but I noticed while testing this today that there are additional regressions, so the test runs will not be clean. That's ok; we can clean those up after this gets checked in.

Triggered stress runs:

jitstress: https://dev.azure.com/dnceng/public/_build/results?buildId=684206&view=results
jitstressregs: https://dev.azure.com/dnceng/public/_build/results?buildId=684207&view=results
jitstress2-jitstressregs: https://dev.azure.com/dnceng/public/_build/results?buildId=684210&view=results

trylek

Great improvements and cleanup, thanks Bruce!

safern · 2020-06-16T16:28:11Z