Setup OSX.1100.ARM64.* helix queues #47919

sdmaclea · 2021-02-05T17:54:21Z

ghost · 2021-02-05T17:54:23Z

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	sdmaclea
Assignees:	sdmaclea
Labels:	`area-Infrastructure`
Milestone:	6.0.0

eng/pipelines/libraries/helix-queues-setup.yml

sdmaclea · 2021-02-05T21:14:53Z

The current version should enable osx-arm64 test for CI runs. Is there a way to trigger them before merging? Or should this be done in a infra branch.

safern · 2021-02-05T21:21:20Z

The current version should enable osx-arm64 test for CI runs. Is there a way to trigger them before merging? Or should this be done in a infra branch.

You can queue a manual build from your branch.

sdmaclea · 2021-02-05T21:37:48Z

Helix M1 queues don't have any machines yet

OK I'll hold off testing till the queues are populated.

However in the runtime pipeline we can't merge with any failures as that pipeline we track it's pass rate

I can disable the unstable tests before merging.

mangod9 · 2021-02-23T17:55:00Z

Hey @sdmaclea, the queues should be hydrated now, could you please try running tests on them?

steveisok · 2021-02-23T20:49:41Z

I ran libraries tests on #47864

Looks like there are various python problems. Not sure if we're going to hit the same things:

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-47864-merge-39d501d2b0984d9ea6/System.Buffers.Tests/console.fba4d13d.log?sv=2019-07-07&se=2021-03-15T18%3A41%3A56Z&sr=c&sp=rl&sig=Vl6KbD0L0CdGQOe5CJ%2F42cXOHHO8n090lOIm9ZVaVAw%3D

mangod9 · 2021-02-24T01:23:33Z

Thanks @steveisok for the info. Is anyone looking into the python script issues? I do see a mention of script issues being worked on here: https://github.com/dotnet/core-eng/issues/11937#issuecomment-780183832. @parose1 is this something you are still looking into?

parose1 · 2021-02-24T16:09:17Z

@mangod9 The python issue I was looking into was blocking the helix script from completing due to a conflict with the python version homebrew installs. This prevented the machines from being added to helix. The solution was to just uninstall the homebrew python and let the helix script install it instead, which allowed the script to complete.

sdmaclea · 2021-02-24T17:11:19Z

/azp run runtime-coreclr jitstress

azure-pipelines · 2021-02-24T17:11:35Z

Azure Pipelines successfully started running 1 pipeline(s).

sdmaclea · 2021-02-24T17:13:37Z

@parose1 Can. you provide more info? What version of python is installed on the machines now? The initial plan was to have the universal binary version of python 3.9.1 installed. Is that still installed and on the default PATH?

parose1 · 2021-02-24T17:37:31Z

@sdmaclea Running "python3 --version" I see python 3.9.1 is installed on the machines. The only difference is that the helix script installs it instead of homebrew. @ulisesh made the suggestion to uninstall the homebrew version and let the helix script install to get me unblocked, so I could add these into the helix queue, so he may know more on the reasoning as to why the homebrew version was causing issues with the helix script.

ulisesh · 2021-02-24T20:27:25Z

@parose1 Can. you provide more info? What version of python is installed on the machines now? The initial plan was to have the universal binary version of python 3.9.1 installed. Is that still installed and on the default PATH?

The default python in the PATH has Intel binaries. We had problems using Python with universal binaries because there were some package (like cryptography) that don't support M1

@sdmaclea can you please help us understand how having Intel based Python affects your scenario?

Talking about https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-47864-merge-39d501d2b0984d9ea6/System.Buffers.Tests/console.fba4d13d.log?sv=2019-07-07&se=2021-03-15T18%3A41%3A56Z&sr=c&sp=rl&sig=Vl6KbD0L0CdGQOe5CJ%2F42cXOHHO8n090lOIm9ZVaVAw%3D it looks like the test reporter is having problems running on Python 3.9 and the recommendation here is to upgrade setuptools. @parose1 could you run python3 -m pip install --upgrade setuptools in all the machines?

parose1 · 2021-02-24T21:39:13Z

Talked with @ulisesh on teams and changed the command a bit to "sudo -H python3 -m pip install --no-user --upgrade setuptools"

I completed running that on all the machines in the osx.1100.arm64.open and osx.1100.arm64 helix queues. Let me know if you need anything else, thanks.

ulisesh · 2021-02-24T21:41:24Z

/azp run runtime-coreclr jitstress

azure-pipelines · 2021-02-24T21:41:42Z

Azure Pipelines successfully started running 1 pipeline(s).

sdmaclea · 2021-02-24T21:54:42Z

/azp run runtime-coreclr outerloop

azure-pipelines · 2021-02-24T21:55:01Z

Azure Pipelines failed to run 1 pipeline(s).

sdmaclea · 2021-02-24T21:55:40Z

/azp list

azure-pipelines · 2021-02-24T21:55:44Z

CI/CD Pipelines for this repository: coreclr-gc-longrunning coreclr-gc-simulator runtime-coreclr outerloop runtime-coreclr jitstress runtime-coreclr jitstressregs runtime-coreclr jitstress2-jitstressregs runtime-coreclr r2r-extra runtime-coreclr jitstress-isas-x86 runtime-coreclr jitstress-isas-arm runtime-coreclr jitstressregs-x86 runtime-coreclr libraries-jitstressregs runtime-coreclr libraries-jitstress2-jitstressregs runtime-coreclr r2r runtime-coreclr runincontext runtime-coreclr crossgen2 runtime-libraries-coreclr outerloop runtime-libraries-coreclr outerloop-windows runtime-libraries-coreclr outerloop-linux runtime-libraries-coreclr outerloop-osx runtime runtime-libraries enterprise-linux runtime-libraries stress-http runtime-libraries stress-ssl runtime-dev-innerloop runtime-coreclr crossgen2 outerloop coreclr-release-outerloop-nightly sync-runtime-to-mono runtime-coreclr crossgen2-composite runtime-jit-experimental runtime-coreclr libraries-jitstress dotnet-linker-tests runtime-coreclr ilasm runtime-coreclr clrinterpreter runtime-coreclr crossgen2-composite gcstress runtime-libraries-mono outerloop runtime-staging

sdmaclea · 2021-02-24T22:07:44Z

/azp run runtime-coreclr jitstress

This reverts commit 03d414b.

sdmaclea · 2021-02-26T05:40:24Z

/azp run runtime-coreclr jitstress

azure-pipelines · 2021-02-26T05:40:37Z

Azure Pipelines successfully started running 1 pipeline(s).

sdmaclea · 2021-02-26T15:23:14Z

/azp run runtime-coreclr outerloop

azure-pipelines · 2021-02-26T15:23:34Z

Azure Pipelines successfully started running 1 pipeline(s).

sdmaclea · 2021-02-26T15:24:10Z

/azp run runtime-coreclr jitstressregs

azure-pipelines · 2021-02-26T15:24:26Z

Azure Pipelines successfully started running 1 pipeline(s).

mangod9 · 2021-02-26T20:09:35Z

Are tests actually running on osx-arm64? I am noticing this failure in the jitstress runs:

Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 6.0.0-preview.2.21117.5)
  Discovering: tracing.eventcounter.XUnitWrapper
  Discovered:  tracing.eventcounter.XUnitWrapper
  Starting:    tracing.eventcounter.XUnitWrapper
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.<>c.<.cctor>b__17_0(Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_1(Object state)
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

sdmaclea · 2021-02-26T20:16:11Z

Yes. That test suite had run in earlier runs. It looks like a spurious failure in that leg of that particular run.

If you remove the failed+aborted filter in the test tab, you can see all the tests run and passed.

I looked at outerloop and some of the other jitstress legs/runs and they look fine.

I think this is close to ready to merge.

@hoyosjs @safern Any objections?

sdmaclea · 2021-02-26T20:19:59Z

I wanted to trigger a libraries-outer fullMatrix run on this before merging. The one triggered through azp run doesn't trigger the osx-arm64 legs, because it is not full mattrix.

The ilasm legs have a similar issue since the PR trigger limits them to PR which change ilasm/ildasm source code.

hoyosjs · 2021-02-26T20:39:52Z

I'll take a look at this later.

sdmaclea · 2021-02-26T21:53:47Z

Triggered libraries outerloop here https://dev.azure.com/dnceng/public/_build/results?buildId=1014515&view=results

This one looks pretty good. There is one macOS failure. It could easily be spurious. x64 also has one failure presumably a different one.

sdmaclea · 2021-02-26T21:56:23Z

I triggered the ilasm pipeline here https://dev.azure.com/dnceng/public/_build/results?buildId=1014527&view=results

This one also looks good for Apple Silicon

sdmaclea · 2021-02-26T21:59:53Z

The coreclr-release-outerloop-nightly has been broken on all other platform for a while. I didn't retrigger it. I can fix it in a separate PR (It is even possible I broke it 9mo ago...)

The various other jitstress runs I didn't retrigger. The earlier runs were very similar to each other.

sdmaclea · 2021-03-01T17:18:43Z

I plan to merge this today.

ViktorHofer · 2021-03-01T17:25:19Z

@safern can you please take a look at the yml changes in particular?

safern

It seems like libraries tests are only going to be run for outerloop... do we want to add a libraries test run for this on innerloop libraries tests that run only when IsFullMatrix == true?

safern · 2021-03-01T21:31:53Z

eng/pipelines/libraries/helix-queues-setup.yml

@@ -85,6 +85,11 @@ jobs:
        # Limiting interp runs as we don't need as much coverage.
        - Debian.9.Amd64.Open

+    # OSX arm64
+    - ${{ if eq(parameters.platform, 'OSX_arm64') }}:
+      - ${{ if eq(parameters.jobParameters.isFullMatrix, true) }}:


I know we discussed this earlier, but I believe we should remove this condition for libraries and then instead condition the test runs in the yml entry point?

sdmaclea · 2021-03-02T02:20:25Z

I plan several other PR to gradually increase coverage. I am going to merge this so we begin getting test coverage. I'll put up a PR to for library tests tomorrow.

sdmaclea added the area-Infrastructure label Feb 5, 2021

sdmaclea added this to the 6.0.0 milestone Feb 5, 2021

sdmaclea requested review from hoyosjs and safern February 5, 2021 17:54

sdmaclea self-assigned this Feb 5, 2021

mangod9 reviewed Feb 5, 2021

View reviewed changes

eng/pipelines/libraries/helix-queues-setup.yml Show resolved Hide resolved

sdmaclea force-pushed the AppleSiliconCI branch from 21e396b to 01cc0b3 Compare February 24, 2021 22:05

sdmaclea added 3 commits February 26, 2021 00:37

Fixup libraries outerloop

965efb3

Disable failing Apple Silicon PAL tests

763770d

Revert "Apple Silicon disable failing PAL tests (dotnet#48782)"

144d483

This reverts commit 03d414b.

sdmaclea force-pushed the AppleSiliconCI branch from 384fdd7 to 144d483 Compare February 26, 2021 05:38

This was referenced Feb 26, 2021

Apple Silicon disable failing PAL tests #48782

Merged

Review Apple Silicon PAL tests disabled by #47919 #48783

Closed

Base automatically changed from master to main March 1, 2021 09:07

safern reviewed Mar 1, 2021

View reviewed changes

sdmaclea merged commit b5c809f into dotnet:main Mar 2, 2021

sdmaclea mentioned this pull request Mar 2, 2021

Allow libraries outerloop OSX_arm64 to be triggered from PR #49018

Merged

ghost locked as resolved and limited conversation to collaborators Apr 1, 2021

Setup OSX.1100.ARM64.* helix queues #47919

Setup OSX.1100.ARM64.* helix queues #47919

Conversation

sdmaclea commented Feb 5, 2021

ghost commented Feb 5, 2021

sdmaclea commented Feb 5, 2021

safern commented Feb 5, 2021

sdmaclea commented Feb 5, 2021

mangod9 commented Feb 23, 2021

steveisok commented Feb 23, 2021

mangod9 commented Feb 24, 2021

parose1 commented Feb 24, 2021

sdmaclea commented Feb 24, 2021

azure-pipelines bot commented Feb 24, 2021

sdmaclea commented Feb 24, 2021

parose1 commented Feb 24, 2021

ulisesh commented Feb 24, 2021

parose1 commented Feb 24, 2021

ulisesh commented Feb 24, 2021

azure-pipelines bot commented Feb 24, 2021

sdmaclea commented Feb 24, 2021

azure-pipelines bot commented Feb 24, 2021

sdmaclea commented Feb 24, 2021

azure-pipelines bot commented Feb 24, 2021

sdmaclea commented Feb 24, 2021

sdmaclea commented Feb 26, 2021

azure-pipelines bot commented Feb 26, 2021

sdmaclea commented Feb 26, 2021

azure-pipelines bot commented Feb 26, 2021

sdmaclea commented Feb 26, 2021

azure-pipelines bot commented Feb 26, 2021

mangod9 commented Feb 26, 2021

sdmaclea commented Feb 26, 2021

sdmaclea commented Feb 26, 2021

hoyosjs commented Feb 26, 2021

sdmaclea commented Feb 26, 2021 • edited Loading

sdmaclea commented Feb 26, 2021 • edited Loading

sdmaclea commented Feb 26, 2021 • edited Loading

sdmaclea commented Mar 1, 2021

ViktorHofer commented Mar 1, 2021

safern left a comment

Choose a reason for hiding this comment

safern Mar 1, 2021

Choose a reason for hiding this comment

sdmaclea commented Mar 2, 2021

sdmaclea commented Feb 26, 2021 •

edited

Loading

sdmaclea commented Feb 26, 2021 •

edited

Loading

sdmaclea commented Feb 26, 2021 •

edited

Loading