[Perf -26%] System.Tests.Perf_String (4)

[DrewScoggins]

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
ToLower	19.36 ns	22.83 ns	1.18		False
IndexerCheckBoundCheckHoist	32.35 ns	47.41 ns	1.47		False
ToUpper	19.69 ns	22.59 ns	1.15		False
Replace_Char	12.59 ns	15.80 ns	1.25		False

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Tests.Perf_String*';

Histogram

System.Tests.Perf_String.ToLower(s: "TEST")

[17.383 ; 18.296) | @@
[18.296 ; 19.756) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[19.756 ; 21.029) | @@@@@
[21.029 ; 22.525) | 
[22.525 ; 23.985) | @@@@@@@@@@@@@@@@@
[23.985 ; 25.194) | @
[25.194 ; 25.850) | 
[25.850 ; 27.310) | @

System.Tests.Perf_String.IndexerCheckBoundCheckHoist

[30.905 ; 35.815) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[35.815 ; 40.085) | 
[40.085 ; 45.827) | @@@@
[45.827 ; 50.738) | @@@@@@@@@@@@@@@@@@@@@@@@@

System.Tests.Perf_String.ToUpper(s: "TeSt")

[17.170 ; 18.265) | @@@
[18.265 ; 19.772) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[19.772 ; 21.112) | @@@@
[21.112 ; 22.508) | 
[22.508 ; 24.015) | @@@@@@@@@@@@@@@@@
[24.015 ; 25.273) | @
[25.273 ; 26.343) | 
[26.343 ; 27.850) | @

System.Tests.Perf_String.Replace_Char(text: "Hello", oldChar: 'l', newChar: '!')

[11.532 ; 13.074) | @@@@@@@@@@@@@@@@
[13.074 ; 14.615) | 
[14.615 ; 16.631) | @@@@@@@@@@@@@@@@@
[16.631 ; 18.420) | @@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[DrewScoggins]

@adamsitnik

[danmoseley]

@DrewScoggins is there some way to close the gap in the graph? Are we missing all perf data from Dec-April or were just these tests broken?

Ideally we would see which build it regressed in.

[DrewScoggins]

The tests are not broken. We are also not missing any data. In general, for comparisons that span many months we try not to put every point in the graph. I will regenerate the graphs that do not show the actual move and update the issues accordingly.

[danmoseley]

Thanks @DrewScoggins. Looks like you did this. Can you clarify the Legend on the graph? The purple and orange lines have build numbers, but I think I would expect the build number to be changing every day. My mental picture is there's a single line, with a datapoint every day, is that wrong?

[danmoseley]

In the case of scenarios like this that are likely affected by the ICU change, a superimposed Linux measurement would likely be interesting - for example, we might see that Windows now matches Linux. (Whether we're OK with that is a separate issue)

[DrewScoggins]

Yeah, I should probably change the legend as it doesn't make sense here outside the context of the full report. You are correct in thinking that these are about one build per day. Also yeah, I can take a look at building out graphs that have the Windows vs Linux comparison so we can see how we look compared to that as well. I will post those in the relevant issues.

[danmoseley]

One other feedback, it would be good in these issue to clarify what the diff link represents. In the graphs above, it seems clear that there are 3 distinct regressions across these scenarios -- Dec, early Jan, and early March. The single diff link provided above covers 7 months , and it's out of the dotnet/installer repo. It's so huge that I get a Unicorn trying to view it. What might be ideal is 3 diff links, each of ~1 week or less out of the dotnet/runtime repo.

cc @adamsitnik @tannergooding since I know you guys are collectively tuning these issues.