Switch over JIT/Methodical tests to run using merged wrappers

[trylek]

fyi @jkoritzinsky

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

fyi @jkoritzinsky

Author:	trylek
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

/azp run runtime-extra-platforms

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jkoritzinsky]

Two things that immediately popped out to me:

1. We can remove the range_d.csproj and range_r.csproj projects as the tests they include are covered by Methodical_d.csproj and Methodical_r.csproj respectively.
2. We need to update the generator to automatically exclude all <RequiresProcessIsolation>true</RequiresProcessIsolation> tests on any platforms where we can't launch new processes. Given the small test coverage loss there, I think that's okay (many of the tests that are going to have RequiresProcessIsolation set to true already don't run on mobile platforms due to other reasons).

[trylek]

@BruceForstall - once I pass initial outerloop validation on the change, I plan to start triggering Crossgen2 and stress runs to further validate the change. Please let me know which pipelines you find the most important to start with.

[trylek]

@jkoritzinsky - thanks for your immediate feedback. I believe my change should include removal of the wrappers under Array you previously added, if it doesn't, it will likely fail the build.

[trylek]

Hmm, in fact it won't fail the build, it only fails local test run because the Python code for producing the summary table hits a duplicate dictionary key. I'm not sure whether that's exercised in lab pipeline runs, I guess we'll see in an hour or two.

[jkoritzinsky]

Yes, it looks like they have been removed. GitHub's diff was not showing it to me due to the number of changed files, but I validated in the GitHub Codespaces view that the Range merged wrappers are gone.

[trylek]

@jkoritzinsky - while this change is still WIP, I have tried to maintain some consistency w.r.t. its contents; I believe the particular bit we're discussing here should be covered in the commit

e4f2590

Thanks

Tomas

[BruceForstall]

First would be runtime-coreclr jitstress. Most others do the same thing but set different environment variables, so maybe it's not necessary to run them all? Another: runtime-coreclr ilasm (round-trip tests). Also runtime-coreclr r2r and/or runtime-coreclr r2r-extra. And it might not hurt to run runtime-coreclr gcstress0x3-gcstress0xc (but unfortunately I don't think it's clean due to gcstress0x3 timeouts)

[trylek]

@jkoritzinsky - Hehe, I was curious why we're not observing the removal of the range_d and range_r projects you previously added. Turns out GitHub decided the newly added Methodical_* projects are their renames.

[jkoritzinsky]

Ah Git rename detection. It always kicks in when you don't need it to and rarely kicks in when you actually want it to

[trylek]

One additional bit of debt in this change is the lack of issues.targets migration regarding the tests involved; I plan to work on that next.

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Pull request contains merge conflicts.

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

/azp run runtime-coreclr jitstress

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

/azp run runtime-coreclr outerloop

[trylek]

/azp run runtime-extra-platforms

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

Based on Andrew's suggestion I have experimentally combined this change with a subtle tweak to src/tests/run.sh making event log opt-in instead of opt-out. The purpose of this experiment is to validate whether it fixes the failures seen in the r2r-extra pipeline on Unix. You can find more detail in the tracking issue #64263.

[trylek]

Hmm, in fact I don't think there's any chance this might work as the script is not actually exercised in the lab pipelines at all; instead I'll add targeted instrumentation letting us see at runtime whether the event log has been activated (in some other manner) directly in the offending runs.

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

/azp run runtime-extra-platforms

[trylek]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[trylek]

I have verified that in the latest r2r-extra run I no longer see the hangs; some unrelated tests failed and two merged wrappers crashed on a runtime JIT assertion that is a known issue according to Bruce. I plan to merge this in after retesting unless some new catastrophic failure appears.