Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Changes at 4/8/2022 8:59:46 PM #4601

Closed
performanceautofiler bot opened this issue Apr 12, 2022 · 1 comment
Closed

[Perf] Changes at 4/8/2022 8:59:46 PM #4601

performanceautofiler bot opened this issue Apr 12, 2022 · 1 comment

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 12, 2022

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Improvements in Exceptions.Handling

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ThrowAndCatchDeep - Duration of single invocation 176.17 μs 45.00 μs 0.26 0.06 True
ThrowAndCatch_ManyCatchBlocks - Duration of single invocation 172.07 μs 40.76 μs 0.24 0.07 True
ThrowAndCatchDeepRecursive - Duration of single invocation 176.14 μs 45.88 μs 0.26 0.08 True
ThrowAndCatchDeepRecursive - Duration of single invocation 180.18 μs 50.64 μs 0.28 0.07 True
ThrowAndCatch_ManyCatchBlocks - Duration of single invocation 175.19 μs 46.62 μs 0.27 0.07 True
ThrowAndCatch - Duration of single invocation 176.59 μs 44.14 μs 0.25 0.08 True
ThrowAndCatch - Duration of single invocation 169.75 μs 39.69 μs 0.23 0.07 True
ThrowAndCatchDeep - Duration of single invocation 181.33 μs 49.16 μs 0.27 0.05 True

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Exceptions.Handling*'

Payloads

Baseline
Compare

Histogram

Exceptions.Handling.ThrowAndCatchDeep(kind: ReflectionSoftware)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 45.00201218242398 < 167.56654611423215.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 181.71905238741783 (T) = (0 -45302.784438907736) / Math.Sqrt((135365421.14605597 / (299)) + (164263.05430900396 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7316163731302842 = (168798.61475640436 - 45302.784438907736) / 168798.61475640436 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionSoftware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 40.75888961038961 < 162.7593277455108.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 186.27611589178517 (T) = (0 -40949.90916787513) / Math.Sqrt((129032235.72867544 / (299)) + (95247.9124522452 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7504068735405165 = (164066.65419338996 - 40949.90916787513) / 164066.65419338996 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatchDeepRecursive(kind: ReflectionSoftware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 45.88157205697529 < 167.3379790496255.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 177.08311743050015 (T) = (0 -45820.291414688094) / Math.Sqrt((139552405.64028078 / (299)) + (275404.5622915354 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7284986074659188 = (168766.32192203705 - 45820.291414688094) / 168766.32192203705 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatchDeepRecursive(kind: ReflectionHardware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 50.64326180875576 < 171.79386906492243.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 182.66404813827114 (T) = (0 -50349.71833109894) / Math.Sqrt((128623978.21226904 / (299)) + (543250.6099182707 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7111105080342964 = (174287.1226935328 - 50349.71833109894) / 174287.1226935328 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatch_ManyCatchBlocks(kind: ReflectionHardware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 46.61854910714286 < 166.48812137432378.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 176.3355728490302 (T) = (0 -45308.203626591116) / Math.Sqrt((140402145.6287587 / (299)) + (362034.51905214385 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7314328980215884 = (168703.47593888527 - 45308.203626591116) / 168703.47593888527 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatch(kind: ReflectionHardware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 44.137319563152886 < 165.71153252314815.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 177.86625136396475 (T) = (0 -44399.11078894102) / Math.Sqrt((138489038.8120928 / (299)) + (336827.0567271175 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7355155567423235 = (167870.40569219706 - 44399.11078894102) / 167870.40569219706 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatch(kind: ReflectionSoftware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 39.68799276672694 < 161.35102865454292.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 181.26931293844484 (T) = (0 -40031.93331345326) / Math.Sqrt((133666025.46544965 / (299)) + (282376.0318494197 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7549153218617601 = (163339.19206027748 - 40031.93331345326) / 163339.19206027748 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### Exceptions.Handling.ThrowAndCatchDeep(kind: ReflectionHardware)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 49.158474842767305 < 171.7302649020041.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 183.2728930355685 (T) = (0 -49699.126073673346) / Math.Sqrt((134345567.77333865 / (299)) + (147453.26184145102 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.7138197449773777 = (173663.71439477772 - 49699.126073673346) / 173663.71439477772 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x86
OS Windows 10.0.18362
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Improvements in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Select - Duration of single invocation 729.43 ns 641.46 ns 0.88 0.08 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 732.37 ns 650.77 ns 0.89 0.16 False
AnyWithPredicate_LastElementMatches - Duration of single invocation 731.66 ns 649.81 ns 0.89 0.17 False
Concat_TenTimes - Duration of single invocation 12.61 μs 10.23 μs 0.81 0.20 False
FirstWithPredicate_LastElementMatches - Duration of single invocation 731.98 ns 652.16 ns 0.89 0.20 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Select(input: Array)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 641.4593634116193 < 678.8512043837875.
IsChangePoint: Marked as a change because one of 3/4/2022 1:10:06 AM, 3/15/2022 12:53:39 AM, 3/31/2022 5:34:47 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 20.671853293224608 (T) = (0 -641.4619602686571) / Math.Sqrt((1125.8492985654605 / (299)) + (7.873641587042035 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.06197180059847001 = (683.8408063615948 - 641.4619602686571) / 683.8408063615948 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 650.7694493250781 < 696.0555992829588.
IsChangePoint: Marked as a change because one of 2/6/2022 6:59:35 PM, 2/7/2022 5:30:00 PM, 4/1/2022 10:26:17 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 21.613808503332937 (T) = (0 -650.9138546585806) / Math.Sqrt((827.0951089964549 / (299)) + (0.023305022798667724 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.05234803630150757 = (686.8701586584557 - 650.9138546585806) / 686.8701586584557 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.AnyWithPredicate_LastElementMatches(input: Array)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 649.8067254394027 < 693.8183560933578.
IsChangePoint: Marked as a change because one of 3/4/2022 7:13:54 PM, 4/1/2022 10:26:17 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 22.895433269354925 (T) = (0 -649.6631854700091) / Math.Sqrt((977.0912427978858 / (299)) + (0.0981315488032512 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.059939140297068816 = (691.0863044285338 - 649.6631854700091) / 691.0863044285338 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.Concat_TenTimes(input: IEnumerable)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.229241551459294 < 10.8503941311929.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 3/7/2022 10:45:01 PM, 3/16/2022 3:05:50 AM, 4/3/2022 1:52:36 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 10.192257333636018 (T) = (0 -9925.26632477396) / Math.Sqrt((1407941.4174104433 / (299)) + (68347.72434830003 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.08651484000504708 = (10865.273744380038 - 9925.26632477396) / 10865.273744380038 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Linq.Tests.Perf_Enumerable.FirstWithPredicate_LastElementMatches(input: IEnumerable)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 652.1623614182976 < 658.7140378053357.
IsChangePoint: Marked as a change because one of 2/4/2022 6:00:25 PM, 2/5/2022 11:25:50 PM, 2/23/2022 5:12:02 AM, 4/3/2022 1:52:36 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 22.160869915640614 (T) = (0 -652.0191212743962) / Math.Sqrt((767.2969869398006 / (299)) + (0.5762675957392862 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.0519398608721238 = (687.7402544043154 - 652.0191212743962) / 687.7402544043154 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Improvements in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SliceSingleSegment - Duration of single invocation 9.96 ns 0.02 ns 0.00 0.10 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.ReadOnlySequenceTests<Char>.SliceSingleSegment


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.022783212861415755 < 9.406018640945186.
IsChangePoint: Marked as a change because one of 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 8.952373246002152 (T) = (0 -1.7020684136090043) / Math.Sqrt((1.6241565909378564 / (299)) + (14.642267405288328 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8263780973891701 = (9.80330469839486 - 1.7020684136090043) / 9.80330469839486 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Improvements in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NewDelegateClassEmptyInstanceFn - Duration of single invocation 538.01 μs 469.92 μs 0.87 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.NewDelegateClassEmptyInstanceFn


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 469.921743697479 < 509.33121874999995.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 2/25/2022 1:44:54 AM, 3/4/2022 3:52:45 PM, 3/17/2022 2:30:39 AM, 4/2/2022 3:40:16 AM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 13.761254479424146 (T) = (0 -465725.81677899085) / Math.Sqrt((756157835.878909 / (299)) + (20878083.266743504 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.05370324535896887 = (492156.20205276896 - 465725.81677899085) / 492156.20205276896 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline dee2bb50f12e5d6ef16eee7ed944b0fb7ef27053
Compare 285b8259af30e255980b1ebfb2ddd60ca291a83a
Diff Diff

Improvements in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FooObjIsDescendant - Duration of single invocation 486.39 μs 406.68 μs 0.84 0.19 False
FooObjIsFoo - Duration of single invocation 696.20 μs 532.24 μs 0.76 0.18 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.FooObjIsDescendant


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 406.6779829545454 < 449.9927023590686.
IsChangePoint: Marked as a change because one of 2/25/2022 1:44:54 AM, 3/4/2022 7:00:20 AM, 3/7/2022 10:45:01 PM, 3/16/2022 5:02:20 PM, 3/21/2022 1:28:14 PM, 3/31/2022 5:34:47 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 18.374786340199297 (T) = (0 -409285.22598498955) / Math.Sqrt((2114176897.7362835 / (299)) + (51230450.349095725 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.12386699269945668 = (467149.64802666183 - 409285.22598498955) / 467149.64802666183 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.FooObjIsFoo

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 532.2384226190476 < 566.5915962202381.
IsChangePoint: Marked as a change because one of 3/10/2022 1:12:51 AM, 3/17/2022 2:30:39 AM, 3/22/2022 2:38:09 PM, 4/3/2022 1:52:36 PM, 4/8/2022 7:02:49 PM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsImprovementStdDev: Marked as improvement because 8.381128425665576 (T) = (0 -473999.642943701) / Math.Sqrt((8778116778.931799 / (299)) + (2323505482.0583086 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.18204847724912254 = (579496.0089438779 - 473999.642943701) / 579496.0089438779 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant