Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 18 Regressions on 7/28/2023 9:42:28 PM #20306

Open
performanceautofiler bot opened this issue Aug 1, 2023 · 3 comments
Open

[Perf] Windows/x64: 18 Regressions on 7/28/2023 9:42:28 PM #20306

performanceautofiler bot opened this issue Aug 1, 2023 · 3 comments
Labels
arch-x64 branch-refs/heads/main kind-micro Look Again os-windows perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Aug 1, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 1da23b146496d95f017439a48f62e78ec15b0289
Compare 60c49c6156e20183e19b87dc7ba8e147674a9f85
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.91 ns 16.95 ns 1.55 0.11 False
10.91 ns 16.93 ns 1.55 0.09 False
14.20 ns 16.65 ns 1.17 0.14 False
14.20 ns 16.54 ns 1.16 0.11 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'

Payloads

Baseline
Compare

System.Text.Perf_Ascii.ToUpperInPlace_Bytes(Size: 128)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.945390757000887 > 11.466947246222322.
IsChangePoint: Marked as a change because one of 6/5/2023 10:43:22 AM, 6/9/2023 2:21:24 AM, 7/1/2023 6:36:20 PM, 7/4/2023 5:42:46 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -17.400310834690554 (T) = (0 -16.938327827161864) / Math.Sqrt((3.1446976062206784 / (32)) + (4.515776295810331E-05 / (8))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (8) - 2, .025) and -0.4750190555227033 = (11.483463731361368 - 16.938327827161864) / 11.483463731361368 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Ascii.ToLowerInPlace_Bytes(Size: 128)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.93264212378833 > 11.469720154781747.
IsChangePoint: Marked as a change because one of 6/5/2023 10:43:22 AM, 6/8/2023 12:15:18 PM, 7/1/2023 6:36:20 PM, 7/4/2023 5:42:46 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -26.071054912183286 (T) = (0 -16.935070328190182) / Math.Sqrt((1.4713638661739326 / (30)) + (1.679228413554539E-05 / (8))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (8) - 2, .025) and -0.517316787108403 = (11.161196180043499 - 16.935070328190182) / 11.161196180043499 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Ascii.ToLower_Bytes_Chars(Size: 128)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.651865503386166 > 14.912934233197227.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/14/2023 2:26:40 PM, 7/28/2023 2:45:53 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -27.202120180911493 (T) = (0 -16.640526250180592) / Math.Sqrt((0.18884898562693675 / (32)) + (0.0034181048944267687 / (10))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (10) - 2, .025) and -0.14832292118584853 = (14.491155704700448 - 16.640526250180592) / 14.491155704700448 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Ascii.ToUpper_Bytes_Chars(Size: 128)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.53869617071014 > 14.910169461578253.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 7/14/2023 2:26:40 PM, 7/28/2023 2:45:53 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -53.50427174640943 (T) = (0 -16.635476646596025) / Math.Sqrt((0.03931862552165771 / (30)) + (0.0039032733867398147 / (10))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (10) - 2, .025) and -0.15293365485770946 = (14.428823875948968 - 16.635476646596025) / 14.428823875948968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 1da23b146496d95f017439a48f62e78ec15b0289
Compare 60c49c6156e20183e19b87dc7ba8e147674a9f85
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.06 μs 1.20 μs 1.14 0.02 False
430.76 ns 464.40 ns 1.08 0.10 False
1.04 μs 1.15 μs 1.11 0.02 False
1.08 μs 1.19 μs 1.10 0.03 False
475.72 ns 512.89 ns 1.08 0.07 False
470.52 ns 512.56 ns 1.09 0.07 False
1.05 μs 1.15 μs 1.09 0.02 False
469.84 ns 516.46 ns 1.10 0.08 False
1.07 μs 1.19 μs 1.12 0.03 False
1.10 μs 1.23 μs 1.11 0.01 False
1.74 μs 1.86 μs 1.07 0.02 False
75.25 ns 94.87 ns 1.26 0.14 False
1.03 μs 1.13 μs 1.09 0.03 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2033458442652154 > 1.0913220517782147.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -15.39883513932232 (T) = (0 -1165.7624038407716) / Math.Sqrt((173.38500082790478 / (34)) + (518.5609719690433 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.1241677024492517 = (1037.000441571926 - 1165.7624038407716) / 1037.000441571926 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedClassHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 464.4024663805764 > 463.01272814206584.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 6/2/2023 6:53:36 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -12.955348776057459 (T) = (0 -475.12741264280453) / Math.Sqrt((45.129432289396675 / (30)) + (62.38506443276502 / (8))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (8) - 2, .025) and -0.09070812336848467 = (435.61371045394475 - 475.12741264280453) / 435.61371045394475 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1480937760970877 > 1.091773074982817.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -33.523913574800794 (T) = (0 -1154.5618440576136) / Math.Sqrt((178.79718759358343 / (34)) + (62.64067415391154 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.11737837883592961 = (1033.277416071377 - 1154.5618440576136) / 1033.277416071377 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1886302245082816 > 1.1331919335256004.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -31.681420601244753 (T) = (0 -1195.6894419741193) / Math.Sqrt((217.49917016024926 / (34)) + (57.46150057783705 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.10820623371717317 = (1078.9412706726146 - 1195.6894419741193) / 1078.9412706726146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideMissInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 512.8928723718305 > 499.7785030160689.
IsChangePoint: Marked as a change because one of 4/28/2023 4:10:17 AM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -8.373435036695229 (T) = (0 -516.1869485246174) / Math.Sqrt((121.02304060969952 / (32)) + (177.7481674621225 / (8))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (8) - 2, .025) and -0.09017442904358537 = (473.4902367664873 - 516.1869485246174) / 473.4902367664873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 512.5642135231152 > 492.8083987425571.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 6/16/2023 8:17:59 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -12.062315454309987 (T) = (0 -506.5960195534238) / Math.Sqrt((139.67482281839293 / (29)) + (46.37094071963958 / (8))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (8) - 2, .025) and -0.0840907710411972 = (467.3003710444578 - 506.5960195534238) / 467.3003710444578 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodBaseHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.145794140086631 > 1.092653023650099.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -42.98397118516 (T) = (0 -1154.454232270694) / Math.Sqrt((104.0476047109114 / (32)) + (35.5393598435794 / (8))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (8) - 2, .025) and -0.11517148190452386 = (1035.2257486885173 - 1154.454232270694) / 1035.2257486885173 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodOverrideHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 516.4559877315426 > 491.35162925303916.
IsChangePoint: Marked as a change because one of 5/19/2023 3:32:16 PM, 6/8/2023 8:20:44 PM, 6/29/2023 1:56:51 PM, 7/28/2023 9:24:36 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -15.376493840092877 (T) = (0 -505.07797049901706) / Math.Sqrt((110.28749046002089 / (35)) + (30.51881610300974 / (9))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (9) - 2, .025) and -0.08444277250855625 = (465.74884659949356 - 505.07797049901706) / 465.74884659949356 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideMissInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1948879934110415 > 1.131872795878832.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/9/2023 7:03:36 PM, 7/28/2023 9:24:36 AM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -31.97201103656371 (T) = (0 -1200.64641080014) / Math.Sqrt((208.07017240165084 / (33)) + (76.30101260911684 / (9))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (9) - 2, .025) and -0.11406351714975492 = (1077.7180944511142 - 1200.64641080014) / 1077.7180944511142 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2273293870308795 > 1.1792834267881571.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/8/2023 8:02:22 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -23.86113855385404 (T) = (0 -1248.8534093444675) / Math.Sqrt((311.8711372095623 / (36)) + (200.8607217966237 / (8))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (8) - 2, .025) and -0.12490034390710987 = (1110.1902636164482 - 1248.8534093444675) / 1110.1902636164482 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesMethodOverrideHitInherit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.8647952779438135 > 1.8347327861515332.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -27.734351260856663 (T) = (0 -1888.9375074880422) / Math.Sqrt((409.2879488590866 / (34)) + (163.41387745680362 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.09129516652745642 = (1730.913473665163 - 1888.9375074880422) / 1730.913473665163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.IsDefinedMethodBaseMiss

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 94.87007207979183 > 86.04620853582298.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -15.470753665636535 (T) = (0 -94.1100538660566) / Math.Sqrt((16.598485164035257 / (36)) + (5.23911958417669 / (8))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (8) - 2, .025) and -0.2101554902192212 = (77.76691063807714 - 94.1100538660566) / 77.76691063807714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Reflection.Attributes.GetCustomAttributesClassHit

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.126386049351166 > 1048.2970671284065.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/9/2023 2:21:24 AM, 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -25.16240065384954 (T) = (0 -1117.9365648787443) / Math.Sqrt((140.5013212658495 / (35)) + (153.08651616783808 / (8))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (35) + (8) - 2, .025) and -0.12144814978806084 = (996.8687050667655 - 1117.9365648787443) / 996.8687050667655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 branch-refs/heads/main kind-micro os-windows perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr untriaged labels Aug 1, 2023
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Aug 1, 2023 •
edited
Loading

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 1da23b146496d95f017439a48f62e78ec15b0289
Compare 60c49c6156e20183e19b87dc7ba8e147674a9f85
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
136.15 ns 159.47 ns 1.17 0.39 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark*'

Payloads

Baseline
Compare

Microsoft.Extensions.DependencyInjection.ActivatorUtilitiesBenchmark.CreateInstance_0

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 159.47390086057328 > 148.14123098889073.
IsChangePoint: Marked as a change because one of 7/28/2023 3:39:20 PM, 7/31/2023 7:57:50 PM falls between 7/23/2023 4:13:29 AM and 7/31/2023 7:57:50 PM.
IsRegressionStdDev: Marked as regression because -11.199163156427783 (T) = (0 -158.95787242096355) / Math.Sqrt((31.85387088514806 / (34)) + (16.402658535408044 / (8))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (34) + (8) - 2, .025) and -0.13865248927558024 = (139.60174321675072 - 158.95787242096355) / 139.60174321675072 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@cincuranet
Copy link

Look again for System.Reflection.Attributes, maybe PGO needs to catch up with dotnet/runtime#89641.

@cincuranet
Copy link

Same on Intel: #20264

This was referenced Aug 3, 2023
Closed
Closed
Closed
This was referenced Sep 5, 2023
Closed
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro Look Again os-windows perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cincuranet