Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh intermittent failures #51318

Closed
runfoapp bot opened this issue Apr 15, 2021 · 11 comments
Closed

readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh intermittent failures #51318

runfoapp bot opened this issue Apr 15, 2021 · 11 comments
Labels
arch-arm64 area-ReadyToRun-coreclr os-macos-bigsur (macOS11) untriaged New issue has not been triaged by the area owner

Comments

@runfoapp
Copy link

runfoapp bot commented Apr 15, 2021 •
edited
Loading

Runfo Tracking Issue: readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh intermittent failures

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1089963 runtime Rolling CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open console.log core dump runclient.py

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 1 1
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 15, 2021
@BruceForstall
Copy link
Member

Another failure: https://dev.azure.com/dnceng/public/_build/results?buildId=1093815&view=ms.vss-test-web.build-test-results-tab

@BruceForstall
Copy link
Member

@dotnet/crossgen-contrib @sdmaclea

@BruceForstall BruceForstall mentioned this issue Apr 17, 2021
Merged
@mangod9
Copy link
Member

mangod9 commented Apr 17, 2021

this looks to be an issue in the async layer somewhere and not specifically related to R2R. cc @kouvel, in case you have seen such intermittent failures before.

@jkotas
Copy link
Member

jkotas commented Apr 17, 2021

This is on OSX arm64. It looks like a dup of #51250 and other OSX arm64 stability issues to me.

@sdmaclea
Copy link
Contributor

sdmaclea commented Apr 17, 2021 •
edited
Loading

Feel free to disable the test. This may be fixed when the helix queues are upgraded to macOS 11.3 (blocked on Apple release of 11.3). See dotnet/core-eng#12776

@mangod9
Copy link
Member

mangod9 commented Apr 17, 2021

since this is intermittent we shouldn't disable the test just yet. osx-arm64 will be somewhat flaky for the next preview or two.

@sdmaclea
Copy link
Contributor

I have been disabling intermittent test failures to help keep CI green. It depends on how often this fails whether it is worth fixing. I had targeted disabling tests with >~1% failure rate.

@VincentBu
Copy link
Contributor

Failed again in runtime-coreclr jitstress 20210419.1

Failed test:

CoreCLR OSX arm64 Checked jitstress1 @ OSX.1100.ARM64.Open
 -readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh

Error message:

1 / 1 (100%, 1 failed): failed in 5598 msecs, exit code 134, expected 0: corerun /tmp/helix/working/AB22092D/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/AB22092D/w/BDF60A3E/e/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
!! Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
!! at System.SZArrayHelper.GetEnumeratorT in System.Private.CoreLib.dll:token 0x6000159+0x7
!! at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.GetStaticDependenciesImpl(DependencyNodeCore1 node) in ILCompiler.DependencyAnalysisFramework.dll:token 0x6000038+0x0
!! at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ProcessMarkStack() in ILCompiler.DependencyAnalysisFramework.dll:token 0x600003a+0x37
!! at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer2.ComputeMarkedNodes() in ILCompiler.DependencyAnalysisFramework.dll:token 0x600003b+0x19
!! at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile) in ILCompiler.ReadyToRun.dll:token 0x6000128+0x0
!! at ILCompiler.Program.Run(String[] args) in crossgen2.dll:token 0x6000154+0x93d
!! at ILCompiler.Program.Main(String[] args) in crossgen2.dll:token 0x600015a+0x5
!! task_for_pid(9697) FAILED 5 (os/kern) failure

Return code: 1
Raw output file: /private/tmp/helix/working/AB22092D/w/BDF60A3E/e/readytorun/coreroot_determinism/Reports/readytorun.coreroot_determinism/coreroot_determinism/coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/AB22092D/p/corerun coreroot_determinism.dll ''
================================== Compiling with seed 1 ==================================
Locating output folder /private/tmp/helix/working/AB22092D/w/BDF60A3E/e/readytorun/coreroot_determinism/coreroot_determinism/seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 61 msecs
Using dotnet: /private/tmp/helix/working/AB22092D/p/corerun
1 / 1 (0%, 0 failed): launching: /private/tmp/helix/working/AB22092D/p/corerun /tmp/helix/working/AB22092D/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/AB22092D/w/BDF60A3E/e/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Compilation failed. /tmp/helix/working/AB22092D/p/corerun /tmp/helix/working/AB22092D/p/R2RTest/R2RTest.dll compile-directory -cr /tmp/helix/working/AB22092D/p -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed1 failed with exit code 1
Expected: 100
Actual: 1
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/tmp/helix/working/AB22092D/p
> /private/tmp/helix/working/AB22092D/w/BDF60A3E/e/readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
Expected: True
Actual: False

Stack trace
   at readytorun_coreroot_determinism._coreroot_determinism_coreroot_determinism_._coreroot_determinism_coreroot_determinism_sh()

@VincentBu
Copy link
Contributor

Historical failures of this test:

Queued OS Arch Pipeline
2021-04-20T06:43:19.907Z osx.1100.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2021-04-17T06:40:19.71Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
2021-04-15T06:43:20.839Z osx.1100.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2021-04-05T05:47:19.044Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
More failures
Queued OS Arch Pipeline
2021-04-05T05:31:35.442Z ubuntu.1804.armarch.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-05T05:17:47.219Z ubuntu.1804.armarch.open arm runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-05T01:08:30.795Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:55:48.686Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:52:00.514Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:42:00.629Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:41:37.53Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:37:39.414Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:36:34.803Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-05T00:34:39.935Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-05T00:25:17.563Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-05T00:16:38.32Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:13:37.405Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-05T00:12:24.16Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:09:48.253Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T23:55:36.978Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T23:50:04.715Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T23:48:58.526Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T23:47:35.873Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:24:04.748Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:24:04.102Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T23:24:00.529Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:23:59.652Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:23:58.98Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T23:23:43.076Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T23:23:42.383Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:23:41.664Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:23:36.147Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:23:35.314Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:23:34.403Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T15:49:20.494Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:49:19.541Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:47:24.154Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:47:23.478Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:46:46.957Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:46:46.254Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:43:19.452Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:43:17.653Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T10:54:04.9Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T10:34:30.964Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2021-04-04T10:01:12.105Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-04-04T09:47:52.052Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T09:45:52.635Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2021-04-04T09:39:30.482Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-04-04T09:19:22.753Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-04-04T09:05:32.486Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-04-04T08:23:49.086Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T08:23:48.093Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T08:23:47.47Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-04T08:19:07.38Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-04T08:19:06.44Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T08:19:05.69Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T05:38:21.939Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T05:14:15.332Z ubuntu.1804.armarch.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T05:07:20.614Z ubuntu.1804.armarch.open arm runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T01:27:31.004Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:11:50.013Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:10:08.617Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T01:08:31.307Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:04:48.6Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T01:01:22.868Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:55:37.132Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:50:52.077Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:49:04.149Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:47:02.582Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:33:27.978Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:33:14.924Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T00:31:33.211Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:12:58.117Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:05:56.014Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:04:47.177Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:03:37.325Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:48:44.904Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:48:43.049Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:48:42.346Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:42:19.576Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:42:18.772Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:42:16.543Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:37:37.878Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:37:37.189Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:37:36.411Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:37:25.113Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:37:24.024Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T15:49:56.388Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:49:55.684Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:47:25.637Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:47:24.895Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:46:53.566Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:46:52.592Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:44:51.272Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:44:50.433Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc

@mangod9
Copy link
Member

mangod9 commented Jul 6, 2021

Closing this since this is a dupe of #51250

@mangod9 mangod9 closed this as completed Jul 6, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Aug 5, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-ReadyToRun-coreclr os-macos-bigsur (macOS11) untriaged New issue has not been triaged by the area owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jkotas @jeffschwMSFT @BruceForstall @sdmaclea @VincentBu @mangod9