Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

[performanceautofiler]

Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare	22870c2e31ae3c028cece52babd3cae57b560713
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	41.86 ms	45.88 ms	1.10	0.00	True
Count - Duration of single invocation	82.61 ms	149.66 ms	1.81	0.02	True
Count - Duration of single invocation	41.46 ms	45.89 ms	1.11	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.878706666666666 > 43.48936900000001.
IsChangePoint: Marked as a change because one of 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsRegressionStdDev: Marked as regression because -57.555669970713076 (T) = (0 -45707237.61464842) / Math.Sqrt((120156442500.45284 / (23)) + (31756925953.486813 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.10965227881703218 = (41190595.00637043 - 45707237.61464842) / 41190595.00637043 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 149.66452666666666 > 86.776050375.
IsChangePoint: Marked as a change because one of 4/6/2022 2:58:43 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsRegressionStdDev: Marked as regression because -260.1647840681483 (T) = (0 -148773791.14330116) / Math.Sqrt((1423360941669.864 / (24)) + (189911038895.10126 / (33))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (33) - 2, .025) and -0.8052199515347898 = (82413110.38957572 - 148773791.14330116) / 82413110.38957572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.890645 > 43.55782025.
IsChangePoint: Marked as a change because one of 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsRegressionStdDev: Marked as regression because -68.24184720333166 (T) = (0 -45646513.477878325) / Math.Sqrt((54640030103.08665 / (23)) + (50199729662.30552 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.10227845696972163 = (41411054.7015183 - 45646513.477878325) / 41411054.7015183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[DrewScoggins]

LIkely #67365 or #67576

[DrewScoggins]

More regressions on x86 from this change. dotnet/perf-autofiling-issues#4589

[stephentoub]

@DrewScoggins, does this only show up as a regression on 32-bit? If yes, it's almost certainly https://github.com/dotnet/runtime/pull/67365/files#diff-f57fafe69c92d1c6acae34ecb5078ce2993286d9d9d2f7a639a4ab90178d4b67R4834-R4874.

[DrewScoggins]

Yes, this is only on 32bit

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	0c5e7daf58e5d79cb7a2a8767d1f0e6ec4de4dd4
Compare	22870c2e31ae3c028cece52babd3cae57b560713
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	41.86 ms	45.88 ms	1.10	0.00	True
Count - Duration of single invocation	82.61 ms	149.66 ms	1.81	0.02	True
Count - Duration of single invocation	41.46 ms	45.89 ms	1.11	0.01	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: NonBacktracking)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.878706666666666 > 43.48936900000001.
IsChangePoint: Marked as a change because one of 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsRegressionStdDev: Marked as regression because -57.555669970713076 (T) = (0 -45707237.61464842) / Math.Sqrt((120156442500.45284 / (23)) + (31756925953.486813 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.10965227881703218 = (41190595.00637043 - 45707237.61464842) / 41190595.00637043 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "(?i)Tom|Sawyer|Huckleberry|Finn", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 149.66452666666666 > 86.776050375.
IsChangePoint: Marked as a change because one of 4/6/2022 2:58:43 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsRegressionStdDev: Marked as regression because -260.1647840681483 (T) = (0 -148773791.14330116) / Math.Sqrt((1423360941669.864 / (24)) + (189911038895.10126 / (33))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (33) - 2, .025) and -0.8052199515347898 = (82413110.38957572 - 148773791.14330116) / 82413110.38957572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "\p{Sm}", Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.890645 > 43.55782025.
IsChangePoint: Marked as a change because one of 4/6/2022 12:51:22 AM, 4/12/2022 5:02:33 PM falls between 4/4/2022 4:14:58 AM and 4/12/2022 5:02:33 PM.
IsRegressionStdDev: Marked as regression because -68.24184720333166 (T) = (0 -45646513.477878325) / Math.Sqrt((54640030103.08665 / (23)) + (50199729662.30552 / (34))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (34) - 2, .025) and -0.10227845696972163 = (41411054.7015183 - 45646513.477878325) / 41411054.7015183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Text.RegularExpressions, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.18362, Regression, CoreClr, x86
Milestone:	-