Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance-ci centos tests failing in BDN #2496

Closed
LoopedBard3 opened this issue Jun 22, 2022 · 11 comments
Closed

Performance-ci centos tests failing in BDN #2496

LoopedBard3 opened this issue Jun 22, 2022 · 11 comments
Assignees
@LoopedBard3
Labels
pipeline blocker issue blocks perf pipeline stays green. This be considered high priority. WRI Issues and PR's that may be especially useful to WRI or should be viewed during WRI

Comments

@LoopedBard3
Copy link
Member

CentOS runs on the performance-ci are failing due to a type initializer exception. This can be seen in the pipeline here https://dev.azure.com/dnceng/public/_build/results?buildId=1838682&view=logs&jobId=c691a557-b8b7-5576-3a35-6adbad78c2b5, the full error is as follows:

[2022/06/22 12:09:18][INFO] System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property.
[2022/06/22 12:09:18][INFO]  ---> EETypeRva:0x030E3B60(System.Reflection.MissingRuntimeArtifactException): This object cannot be invoked because no code was generated for it: 'BenchmarkDotNet.Jobs.EnvironmentMode.PlatformCharacteristic'.
[2022/06/22 12:09:18][INFO]    at System.Reflection.Runtime.FieldInfos.RuntimeFieldInfo.get_FieldAccessor() + 0x11e
[2022/06/22 12:09:18][INFO]    at System.Reflection.Runtime.FieldInfos.RuntimeFieldInfo.GetValue(Object) + 0x10
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicHelper.<>c.<GetThisTypeCharacteristicsCore>b__8_1(FieldInfo) + 0x19
[2022/06/22 12:09:18][INFO]    at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() + 0x80
[2022/06/22 12:09:18][INFO]    at System.Linq.Enumerable.ConcatIterator`1.MoveNext() + 0x61
[2022/06/22 12:09:18][INFO]    at System.Collections.Generic.HashSet`1.UnionWith(IEnumerable`1) + 0x3f
[2022/06/22 12:09:18][INFO]    at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1, IEqualityComparer`1) + 0xfa
[2022/06/22 12:09:18][INFO]    at System.Linq.Enumerable.DistinctIterator`1.ToArray() + 0x36
[2022/06/22 12:09:18][INFO]    at System.Linq.Buffer`1..ctor(IEnumerable`1) + 0x34
[2022/06/22 12:09:18][INFO]    at System.Linq.OrderedEnumerable`1.ToArray() + 0x38
[2022/06/22 12:09:18][INFO]    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0x8d
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicHelper.FillAllCharacteristicsCore(Type, List`1, HashSet`1) + 0x28
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicHelper.GetAllCharacteristicsCore(Type) + 0x47
[2022/06/22 12:09:18][INFO]    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0x8d
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicObject.GetCharacteristicsToApply() + 0x2c
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Characteristics.CharacteristicObject.GetCharacteristicsToApply(CharacteristicObject) + 0x1c
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Jobs.Job..cctor() + 0x204
[2022/06/22 12:09:18][INFO]    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xb9
[2022/06/22 12:09:18][INFO]    --- End of inner exception stack trace ---
[2022/06/22 12:09:18][INFO]    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0x153
[2022/06/22 12:09:18][INFO]    at System.Runtime.CompilerServices.ClassConstructorRunner.CheckStaticClassConstructionReturnGCStaticBase(StaticClassConstructionContext*, Object) + 0x9
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Jobs.Job..ctor(String) + 0x57
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost, String) + 0x1b4
[2022/06/22 12:09:18][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[]) + 0x256
@LoopedBard3 LoopedBard3 added pipeline blocker issue blocks perf pipeline stays green. This be considered high priority. WRI Issues and PR's that may be especially useful to WRI or should be viewed during WRI labels Jun 22, 2022
@LoopedBard3 LoopedBard3 mentioned this issue Jun 22, 2022
Closed
@LoopedBard3
Copy link
Member Author

LoopedBard3 commented Jul 8, 2022 •
edited
Loading

@adamsitnik @naricc Have y'all seen an error like this before with BDN? Here is a log with the error: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-performance-refs-heads-main-367830c2c6914864b5/Partition0/1/console.f1d1238a.log?helixlogtype=result
Thanks!

@LoopedBard3 LoopedBard3 mentioned this issue Jul 8, 2022
Merged
@adamsitnik
Copy link
Member

Hi @LoopedBard3 !

@MichalStrehovsky has recently fixed this issue with dotnet/BenchmarkDotNet#2020

I've just fixed a recent breaking change: dotnet/BenchmarkDotNet#2045

To fix these builds we need is to upload 0.13.1.1818 to our internal feed and update BDN version in perf repo. Could you please do that?

BTW BenchmarkDotNet has recently updated all it's dependencies like TraceEvent, ClrMd and Iced. Some of them might not be in our internal feed(s) yet and you might need to upload them as well.

@LoopedBard3
Copy link
Member Author

Taking a look!

@LoopedBard3
Copy link
Member Author

It looks like there is a new 0.13.1.18.19 version, any reason not to take that one?

@LoopedBard3 LoopedBard3 mentioned this issue Jul 19, 2022
Merged
@MichalStrehovsky
Copy link
Member

0.13.1.1820 should have my fix. Can we try with that one?

@adamsitnik
Copy link
Member

@MichalStrehovsky I've verified that your fix has solved this problem in #2534, but a new one popped out: dotnet/roslyn#62864

@LoopedBard3 could you please update to 0.13.1.1820? It's going to help and then as soon as dotnet/roslyn#62864 gets fixed the CI should become green again

@MichalStrehovsky
Copy link
Member

@MichalStrehovsky I've verified that your fix has solved this problem in #2534, but a new one popped out: dotnet/roslyn#62864

If it's blocking, downgrading the Roslyn used to build this repo to a build from 2 weeks ago would work around.

@LoopedBard3
Copy link
Member Author

Updated to 0.13.1.1823 in the #2532 PR.

@LoopedBard3
Copy link
Member Author

Looks like the BDN changes are running properly, but there are some tests that are unsupported:

[2022/07/22 15:50:58][INFO] // Found 1 benchmarks:
[2022/07/22 15:50:58][INFO] //   MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // **************************
[2022/07/22 15:50:58][INFO] // Benchmark: MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] // *** Execute ***
[2022/07/22 15:50:58][INFO] // Launch: 1 / 1
[2022/07/22 15:50:58][INFO] // Execute: /home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/91d5ac24-fe26-461f-afdd-5e7aeda1a89d/bin/Release/net7.0/linux-x64/publish/91d5ac24-fe26-461f-afdd-5e7aeda1a89d --benchmarkName "BenchmarksGame.MandelBrot_7.Bench(size: 4000, lineLength: 500, checksum: \"C7-E6-66-43-66-73-F8-A8-D3-B4-D7-97-2F-FC-A1-D3\")" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0" --benchmarkId 9 in
[2022/07/22 15:50:58][INFO] Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied
[2022/07/22 15:50:58][INFO] // BeforeAnythingElse
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // Benchmark Process Environment Information:
[2022/07/22 15:50:58][INFO] // Runtime=.NET 7.0.0-rc.1.22372.1, X64 NativeAOT
[2022/07/22 15:50:58][INFO] // GC=Non-concurrent Workstation
[2022/07/22 15:50:58][INFO] // Job: Job-UDTYPC(PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // BeforeActualRun
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] System.PlatformNotSupportedException: Operation is not supported on this platform.
[2022/07/22 15:50:58][INFO]    at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowPlatformNotSupportedException() + 0x20
[2022/07/22 15:50:58][INFO]    at BenchmarksGame.MandelBrot_7.DoBench(Int32, Int32) + 0x8a
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_9.WorkloadActionUnroll(Int64) + 0x4d
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData) + 0x1d3
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.EngineStage.RunIteration(IterationMode, IterationStage, Int32, Int64, Int32) + 0x87
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.EngineActualStage.RunSpecific(Int64, IterationMode, Int32, Int32) + 0x79
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Engines.Engine.Run() + 0x291
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_9.Run(IHost, String) + 0x6af
[2022/07/22 15:50:58][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[]) + 0x2ee
[2022/07/22 15:50:58][INFO] // AfterAll
[2022/07/22 15:50:58][INFO] No Workload Results were obtained from the run.
[2022/07/22 15:50:58][INFO] // Benchmark Process 9718 has exited with code 255.
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // ** Remained 852 (98.8%) benchmark(s) to run. Estimated finish 2022-07-22 16:13 (0h 22m from now) **
[2022/07/22 15:50:58][INFO] // ***** BenchmarkRunner: Finish  *****
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Export *
[2022/07/22 15:50:58][INFO]   home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/BenchmarkDotNet.Artifacts/results/BenchmarksGame.MandelBrot_7-report-github.md
[2022/07/22 15:50:58][INFO]   home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/BenchmarkDotNet.Artifacts/results/BenchmarksGame.MandelBrot_7-report-full.json
[2022/07/22 15:50:58][INFO]   home/helixbot/work/A626091F/w/BCD109E7/e/artifacts/bin/MicroBenchmarks/Release/net7.0/BenchmarkDotNet.Artifacts/results/BenchmarksGame.MandelBrot_7-perf-lab-report.json
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Detailed results *
[2022/07/22 15:50:58][INFO] MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] Runtime = .NET 7.0.0-rc.1.22372.1, X64 NativeAOT; GC = Non-concurrent Workstation
[2022/07/22 15:50:58][INFO] There are not any results runs
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Summary *
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] BenchmarkDotNet=v0.13.1.1823-nightly, OS=centos 7
[2022/07/22 15:50:58][INFO] AMD EPYC 7452, 1 CPU, 2 logical cores and 1 physical core
[2022/07/22 15:50:58][INFO] .NET SDK=7.0.100-rc.1.22371.5
[2022/07/22 15:50:58][INFO]   [Host]     : .NET 7.0.0 (7.0.22.36704), X64 RyuJIT
[2022/07/22 15:50:58][INFO]   Job-CLDRFN : .NET 7.0.0-rc.1.22372.1, X64 NativeAOT
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] PowerPlanMode=00000000-0000-0000-0000-000000000000  Runtime=NativeAOT 7.0  Toolchain=ILCompiler 7.0.0-*
[2022/07/22 15:50:58][INFO] InvocationCount=1  IterationCount=1  IterationTime=250.0000 ms
[2022/07/22 15:50:58][INFO] MaxIterationCount=20  MinIterationCount=15  RunStrategy=ColdStart
[2022/07/22 15:50:58][INFO] UnrollFactor=1  WarmupCount=0
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] |       Method | size | lineLength |                             checksum | Mean | Error | Median | Min | Max |
[2022/07/22 15:50:58][INFO] |------------- |----- |----------- |------------------------------------- |-----:|------:|-------:|----:|----:|
[2022/07/22 15:50:58][INFO] | MandelBrot_7 | 4000 |        500 | C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47] |   NA |    NA |     NA |  NA |  NA |
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] Benchmarks with issues:
[2022/07/22 15:50:58][INFO]   MandelBrot_7.MandelBrot_7: Job-CLDRFN(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=NativeAOT 7.0, Toolchain=ILCompiler 7.0.0-*, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [size=4000, lineLength=500, checksum=C7-E6-66-43-6(...)7-2F-FC-A1-D3 [47]]
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Legends *
[2022/07/22 15:50:58][INFO]   size       : Value of the 'size' parameter
[2022/07/22 15:50:58][INFO]   lineLength : Value of the 'lineLength' parameter
[2022/07/22 15:50:58][INFO]   checksum   : Value of the 'checksum' parameter
[2022/07/22 15:50:58][INFO]   Mean       : Arithmetic mean of all measurements
[2022/07/22 15:50:58][INFO]   Error      : Half of 99.9% confidence interval
[2022/07/22 15:50:58][INFO]   Median     : Value separating the higher half of all measurements (50th percentile)
[2022/07/22 15:50:58][INFO]   Min        : Minimum
[2022/07/22 15:50:58][INFO]   Max        : Maximum
[2022/07/22 15:50:58][INFO]   1 ns       : 1 Nanosecond (0.000000001 sec)
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // * Diagnostic Output - MemoryDiagnoser *
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] 
[2022/07/22 15:50:58][INFO] // ***** BenchmarkRunner: End *****
[2022/07/22 15:50:58][INFO] Run time: 00:00:00 (0.04 sec), executed benchmarks: 1

If y'all see something immediately obvious as to why this error was hit (pertaining to what isn't supported, not that something is not supported), please respond.

@LoopedBard3 LoopedBard3 self-assigned this Jul 22, 2022
@LoopedBard3 LoopedBard3 mentioned this issue Jul 22, 2022
Merged
@MichalStrehovsky
Copy link
Member

If y'all see something immediately obvious as to why this error was hit

It's the issue Adam wrote about above

@LoopedBard3
Copy link
Member Author

It appears that the performance-ci tests for centos are now succeeding on main after merging #2532. Closing issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LoopedBard3 LoopedBard3

Labels
pipeline blocker issue blocks perf pipeline stays green. This be considered high priority. WRI Issues and PR's that may be especially useful to WRI or should be viewed during WRI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@adamsitnik @MichalStrehovsky @LoopedBard3