Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ASM] Join with char separator bug fix #4793

Merged
merged 2 commits into from
Nov 2, 2023

Conversation

daniel-romano-DD
Copy link
Contributor

@daniel-romano-DD daniel-romano-DD commented Nov 1, 2023

Summary of changes

Join with char separators overloads had a bug which made them return the object ToString, and if they where a container the results were wrong.

Reason for change

A client detected a misbehaving app caused by this Join function result change.

Implementation details

Redirected the Join char overloads to the corresponding separator.ToString() versions

Test coverage

Added instrumentation tests

Other details

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 1, 2023

Datadog Report

Branch report: dani/asm/join_bug_fix
Commit report: 3c7a3f6

❄️ dd-trace-dotnet: 0 Failed, 1 New Flaky, 296350 Passed, 1310 Skipped, 39m 19.48s Wall Time

New Flaky Tests (1)

  • DetectContinousProfilerState - Datadog.Trace.Tools.dd_dotnet.ArtifactTests.Checks.ProcessBasicChecksTests - Last Failure

    Expand for error
     Timeout when waiting for the target process to start
    

@andrewlock
Copy link
Member

andrewlock commented Nov 1, 2023

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4793) - mean (73ms)  : 64, 82
     .   : milestone, 73,
    master - mean (71ms)  : 62, 81
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4793) - mean (1,006ms)  : 988, 1023
     .   : milestone, 1006,
    master - mean (999ms)  : 982, 1017
     .   : milestone, 999,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4793) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (106ms)  : 103, 108
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4793) - mean (689ms)  : 669, 708
     .   : milestone, 689,
    master - mean (688ms)  : 669, 707
     .   : milestone, 688,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4793) - mean (90ms)  : 88, 93
     .   : milestone, 90,
    master - mean (90ms)  : 87, 94
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4793) - mean (654ms)  : 631, 677
     .   : milestone, 654,
    master - mean (656ms)  : 630, 682
     .   : milestone, 656,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4793) - mean (188ms)  : 185, 191
     .   : milestone, 188,
    master - mean (188ms)  : 186, 191
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4793) - mean (1,147ms)  : 1126, 1168
     .   : milestone, 1147,
    master - mean (1,141ms)  : 1115, 1166
     .   : milestone, 1141,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4793) - mean (273ms)  : 267, 278
     .   : milestone, 273,
    master - mean (271ms)  : 266, 277
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4793) - mean (1,100ms)  : 1079, 1121
     .   : milestone, 1100,
    master - mean (1,098ms)  : 1079, 1116
     .   : milestone, 1098,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4793) - mean (263ms)  : 260, 266
     .   : milestone, 263,
    master - mean (262ms)  : 257, 267
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4793) - mean (1,065ms)  : 1041, 1090
     .   : milestone, 1065,
    master - mean (1,067ms)  : 1044, 1090
     .   : milestone, 1067,

@andrewlock
Copy link
Member

andrewlock commented Nov 1, 2023

Benchmarks Report 🐌

Benchmarks for #4793 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.136
  • 1 benchmarks are slower, with geometric mean 1.194
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.13μs 44ns 237ns 0.0285 0.0122 0 7.29 KB
master StartStopWithChild netcoreapp3.1 9.88μs 53.5ns 293ns 0.0248 0.00993 0 7.39 KB
master StartStopWithChild net472 15.6μs 46.1ns 179ns 1.28 0.315 0.1 7.65 KB
#4793 StartStopWithChild net6.0 8.28μs 45.3ns 264ns 0.0255 0.00849 0 7.29 KB
#4793 StartStopWithChild netcoreapp3.1 10μs 53.4ns 282ns 0.0236 0.00945 0 7.38 KB
#4793 StartStopWithChild net472 15.4μs 53.6ns 208ns 1.28 0.322 0.107 7.67 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 462μs 143ns 514ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 615μs 157ns 587ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 798μs 200ns 722ns 0.401 0 0 3.3 KB
#4793 WriteAndFlushEnrichedTraces net6.0 476μs 322ns 1.25μs 0 0 0 2.7 KB
#4793 WriteAndFlushEnrichedTraces netcoreapp3.1 632μs 417ns 1.61μs 0 0 0 2.7 KB
#4793 WriteAndFlushEnrichedTraces net472 802μs 528ns 2.05μs 0.401 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 42.7μs 65.4ns 253ns 0.0213 0 0 2.03 KB
master AllCycleSimpleBody netcoreapp3.1 45μs 30.5ns 118ns 0.0224 0 0 2.01 KB
master AllCycleSimpleBody net472 46.2μs 92.6ns 359ns 0.321 0 0 2.08 KB
master AllCycleMoreComplexBody net6.0 229μs 162ns 627ns 0.115 0 0 8.63 KB
master AllCycleMoreComplexBody netcoreapp3.1 235μs 140ns 542ns 0.117 0 0 8.52 KB
master AllCycleMoreComplexBody net472 241μs 79.5ns 308ns 1.32 0 0 8.7 KB
master ObjectExtractorSimpleBody net6.0 124ns 0.0902ns 0.338ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 179ns 0.118ns 0.443ns 0.00375 0 0 272 B
master ObjectExtractorSimpleBody net472 146ns 0.133ns 0.514ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.02μs 1.48ns 5.73ns 0.0543 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.11μs 1.44ns 5.57ns 0.0515 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.15μs 3.33ns 12.9ns 0.618 0.00622 0 3.89 KB
#4793 AllCycleSimpleBody net6.0 42.9μs 16.8ns 65.2ns 0.0215 0 0 2.03 KB
#4793 AllCycleSimpleBody netcoreapp3.1 45.1μs 69.9ns 271ns 0.0221 0 0 2.01 KB
#4793 AllCycleSimpleBody net472 47.4μs 20.6ns 79.8ns 0.328 0 0 2.08 KB
#4793 AllCycleMoreComplexBody net6.0 229μs 51.1ns 191ns 0 0 0 8.63 KB
#4793 AllCycleMoreComplexBody netcoreapp3.1 233μs 182ns 706ns 0.116 0 0 8.52 KB
#4793 AllCycleMoreComplexBody net472 238μs 69.3ns 259ns 1.3 0 0 8.7 KB
#4793 ObjectExtractorSimpleBody net6.0 126ns 0.0887ns 0.332ns 0.00393 0 0 280 B
#4793 ObjectExtractorSimpleBody netcoreapp3.1 173ns 0.0919ns 0.331ns 0.00375 0 0 272 B
#4793 ObjectExtractorSimpleBody net472 147ns 0.213ns 0.824ns 0.0446 0 0 281 B
#4793 ObjectExtractorMoreComplexBody net6.0 3.12μs 1.08ns 4.17ns 0.0547 0 0 3.88 KB
#4793 ObjectExtractorMoreComplexBody netcoreapp3.1 4.16μs 1.01ns 3.77ns 0.052 0 0 3.78 KB
#4793 ObjectExtractorMoreComplexBody net472 4.09μs 4.14ns 14.9ns 0.618 0.00618 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 12.6μs 4.01ns 15ns 0.133 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 19.6μs 6.31ns 23.6ns 0.128 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) net472 28.4μs 10.3ns 39.7ns 1.49 0 0 9.48 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 77.2μs 46ns 172ns 0.188 0 0 15.77 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 87.2μs 212ns 792ns 0.22 0 0 15.72 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 97.5μs 231ns 894ns 2.51 0 0 16.04 KB
master RunWaf(args=NestedMap (100)) net6.0 24.3μs 9.32ns 34.9ns 0.268 0 0 19.66 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 38.4μs 13.9ns 50.2ns 0.269 0 0 20.42 KB
master RunWaf(args=NestedMap (100)) net472 55.1μs 16.7ns 62.6ns 3.26 0.0274 0 20.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 91.1μs 32.1ns 124ns 0.363 0 0 26.01 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 110μs 163ns 586ns 0.325 0 0 26.72 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 131μs 86.7ns 336ns 4.26 0.0655 0 27.19 KB
master RunWaf(args=NestedMap (20)) net6.0 24.1μs 8.28ns 32.1ns 0.266 0 0 19.4 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 38.1μs 21.6ns 80.7ns 0.267 0 0 19.84 KB
master RunWaf(args=NestedMap (20)) net472 56.7μs 24ns 92.8ns 3.16 0.0565 0 20.04 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 93.6μs 74.9ns 290ns 0.328 0 0 25.74 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 107μs 116ns 434ns 0.323 0 0 26.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 132μs 173ns 672ns 4.18 0.0652 0 26.6 KB
#4793 RunWaf(args=NestedMap (10)) net6.0 12.7μs 4.21ns 16.3ns 0.133 0 0 9.42 KB
#4793 RunWaf(args=NestedMap (10)) netcoreapp3.1 19.7μs 29.3ns 113ns 0.127 0 0 9.42 KB
#4793 RunWaf(args=NestedMap (10)) net472 28.1μs 17.4ns 67.4ns 1.5 0 0 9.48 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 75.6μs 52.3ns 181ns 0.191 0 0 15.77 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 86.2μs 49.4ns 178ns 0.171 0 0 15.72 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [22]) net472 98.8μs 125ns 486ns 2.52 0 0 16.04 KB
#4793 RunWaf(args=NestedMap (100)) net6.0 24.5μs 8.86ns 34.3ns 0.271 0 0 19.66 KB
#4793 RunWaf(args=NestedMap (100)) netcoreapp3.1 38μs 13.1ns 50.6ns 0.266 0 0 20.42 KB
#4793 RunWaf(args=NestedMap (100)) net472 57.3μs 27.7ns 107ns 3.26 0.0286 0 20.63 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 90.3μs 24.6ns 92.1ns 0.361 0 0 26.01 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 109μs 140ns 542ns 0.329 0 0 26.72 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [23]) net472 132μs 161ns 624ns 4.31 0.0652 0 27.19 KB
#4793 RunWaf(args=NestedMap (20)) net6.0 23.4μs 16.7ns 60.2ns 0.27 0 0 19.4 KB
#4793 RunWaf(args=NestedMap (20)) netcoreapp3.1 37.8μs 10.8ns 40.5ns 0.265 0 0 19.84 KB
#4793 RunWaf(args=NestedMap (20)) net472 56.5μs 35.8ns 139ns 3.17 0.0562 0 20.04 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 91.2μs 45.2ns 169ns 0.319 0 0 25.74 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 110μs 209ns 755ns 0.333 0 0 26.14 KB
#4793 RunWafWithAttack(args=Neste(...)tack) [22]) net472 131μs 122ns 473ns 4.17 0.0641 0 26.6 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 168μs 108ns 419ns 0.251 0 0 18.04 KB
master SendRequest netcoreapp3.1 191μs 300ns 1.16μs 0.191 0 0 20.2 KB
master SendRequest net472 0.000491ns 0.000234ns 0.000844ns 0 0 0 0 b
#4793 SendRequest net6.0 168μs 106ns 410ns 0.251 0 0 18.04 KB
#4793 SendRequest netcoreapp3.1 191μs 275ns 1.07μs 0.192 0 0 20.2 KB
#4793 SendRequest net472 7.57E‑05ns 5.12E‑05ns 0.000198ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #4793

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.44 KB 41.79 KB 353 B 0.85%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 540μs 601ns 2.25μs 0.541 0 0 41.44 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 642μs 552ns 2.14μs 0.324 0 0 41.78 KB
master WriteAndFlushEnrichedTraces net472 844μs 2.75μs 10.6μs 8.22 2.47 0.411 53.22 KB
#4793 WriteAndFlushEnrichedTraces net6.0 558μs 1.25μs 4.7μs 0.548 0 0 41.79 KB
#4793 WriteAndFlushEnrichedTraces netcoreapp3.1 664μs 1.57μs 6.1μs 0.331 0 0 41.75 KB
#4793 WriteAndFlushEnrichedTraces net472 842μs 3.94μs 15.7μs 8.28 2.48 0.414 53.23 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.02μs 0.385ns 1.49ns 0.0107 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.32μs 0.587ns 2.19ns 0.0103 0 0 768 B
master ExecuteNonQuery net472 1.68μs 3.83ns 14.9ns 0.115 0 0 730 B
#4793 ExecuteNonQuery net6.0 1.11μs 0.509ns 1.97ns 0.0105 0 0 768 B
#4793 ExecuteNonQuery netcoreapp3.1 1.36μs 0.556ns 2.15ns 0.01 0 0 768 B
#4793 ExecuteNonQuery net472 1.62μs 0.575ns 2.15ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4793

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.131 1,147.28 1,014.46

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.15μs 0.346ns 1.34ns 0.013 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.4μs 0.817ns 3.06ns 0.0126 0 0 936 B
master CallElasticsearch net472 2.36μs 0.76ns 2.74ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.14μs 0.629ns 2.35ns 0.0126 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.42μs 0.893ns 3.46ns 0.0135 0 0 984 B
master CallElasticsearchAsync net472 2.61μs 0.663ns 2.57ns 0.16 0 0 1.01 KB
#4793 CallElasticsearch net6.0 1.01μs 0.402ns 1.5ns 0.0132 0 0 936 B
#4793 CallElasticsearch netcoreapp3.1 1.35μs 0.42ns 1.57ns 0.0129 0 0 936 B
#4793 CallElasticsearch net472 2.33μs 0.631ns 2.45ns 0.151 0 0 955 B
#4793 CallElasticsearchAsync net6.0 1.18μs 0.236ns 0.881ns 0.0124 0 0 912 B
#4793 CallElasticsearchAsync netcoreapp3.1 1.51μs 1.36ns 5.1ns 0.0132 0 0 984 B
#4793 CallElasticsearchAsync net472 2.43μs 2.35ns 9.12ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.22μs 0.817ns 3.16ns 0.0129 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.4μs 0.525ns 1.97ns 0.0121 0 0 912 B
master ExecuteAsync net472 1.66μs 0.909ns 3.52ns 0.139 0 0 875 B
#4793 ExecuteAsync net6.0 1.2μs 0.97ns 3.76ns 0.0128 0 0 912 B
#4793 ExecuteAsync netcoreapp3.1 1.49μs 0.559ns 2.09ns 0.0127 0 0 912 B
#4793 ExecuteAsync net472 1.59μs 0.272ns 0.941ns 0.139 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.68μs 1.15ns 4.44ns 0.0257 0 0 1.9 KB
master SendAsync netcoreapp3.1 4.47μs 0.855ns 3.08ns 0.0334 0 0 2.43 KB
master SendAsync net472 7.14μs 11.7ns 45.4ns 0.472 0 0 2.99 KB
#4793 SendAsync net6.0 3.73μs 0.968ns 3.49ns 0.0263 0 0 1.9 KB
#4793 SendAsync netcoreapp3.1 4.49μs 2.15ns 8.34ns 0.0313 0 0 2.43 KB
#4793 SendAsync net472 7.02μs 3.65ns 13.7ns 0.473 0 0 2.99 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.4μs 0.988ns 3.83ns 0.0219 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 1.85μs 1.12ns 4.34ns 0.0216 0 0 1.57 KB
master EnrichedLog net472 2.26μs 1.38ns 4.99ns 0.237 0 0 1.49 KB
#4793 EnrichedLog net6.0 1.39μs 0.672ns 2.51ns 0.0224 0 0 1.57 KB
#4793 EnrichedLog netcoreapp3.1 1.99μs 0.83ns 3.21ns 0.0209 0 0 1.57 KB
#4793 EnrichedLog net472 2.17μs 1.13ns 4.21ns 0.236 0 0 1.49 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 115μs 224ns 869ns 0.0571 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 119μs 88.8ns 344ns 0 0 0 4.21 KB
master EnrichedLog net472 150μs 101ns 390ns 0.67 0.223 0 4.38 KB
#4793 EnrichedLog net6.0 113μs 123ns 478ns 0.0559 0 0 4.21 KB
#4793 EnrichedLog netcoreapp3.1 118μs 87.9ns 329ns 0.0588 0 0 4.21 KB
#4793 EnrichedLog net472 149μs 70.1ns 243ns 0.672 0.224 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.87μs 1.26ns 4.89ns 0.0304 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 3.8μs 0.625ns 2.34ns 0.0284 0 0 2.13 KB
master EnrichedLog net472 4.61μs 1.68ns 6.04ns 0.305 0 0 1.93 KB
#4793 EnrichedLog net6.0 2.99μs 1.1ns 4.24ns 0.0299 0 0 2.13 KB
#4793 EnrichedLog netcoreapp3.1 4.01μs 1.18ns 4.42ns 0.0282 0 0 2.13 KB
#4793 EnrichedLog net472 4.55μs 2.56ns 9.58ns 0.306 0 0 1.93 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.28μs 0.848ns 3.29ns 0.0154 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.62μs 0.825ns 3.19ns 0.0146 0 0 1.1 KB
master SendReceive net472 1.95μs 3.13ns 12.1ns 0.177 0.000962 0 1.12 KB
#4793 SendReceive net6.0 1.2μs 0.524ns 2.03ns 0.0156 0 0 1.1 KB
#4793 SendReceive netcoreapp3.1 1.62μs 0.545ns 2.04ns 0.0153 0 0 1.1 KB
#4793 SendReceive net472 2.09μs 1.53ns 5.92ns 0.176 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.5μs 0.851ns 3.19ns 0.0212 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.78μs 1.43ns 5.55ns 0.0208 0 0 1.58 KB
master EnrichedLog net472 4.16μs 1.26ns 4.72ns 0.311 0 0 1.96 KB
#4793 EnrichedLog net6.0 2.66μs 1.2ns 4.66ns 0.0212 0 0 1.53 KB
#4793 EnrichedLog netcoreapp3.1 3.52μs 3.71ns 14.4ns 0.0211 0 0 1.58 KB
#4793 EnrichedLog net472 4.12μs 2.83ns 10.6ns 0.311 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4793

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.194 389.40 464.85

Faster 🎉 in #4793

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.141 600.99 526.65

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 389ns 0.13ns 0.488ns 0.0076 0 0 536 B
master StartFinishSpan netcoreapp3.1 601ns 0.153ns 0.592ns 0.00728 0 0 536 B
master StartFinishSpan net472 678ns 0.18ns 0.673ns 0.0854 0 0 538 B
master StartFinishScope net6.0 501ns 0.155ns 0.598ns 0.00908 0 0 656 B
master StartFinishScope netcoreapp3.1 643ns 0.228ns 0.791ns 0.00881 0 0 656 B
master StartFinishScope net472 845ns 0.207ns 0.801ns 0.0981 0 0 618 B
#4793 StartFinishSpan net6.0 465ns 0.107ns 0.414ns 0.0074 0 0 536 B
#4793 StartFinishSpan netcoreapp3.1 527ns 0.669ns 2.59ns 0.00708 0 0 536 B
#4793 StartFinishSpan net472 676ns 0.218ns 0.846ns 0.0853 0 0 538 B
#4793 StartFinishScope net6.0 480ns 0.166ns 0.597ns 0.00917 0 0 656 B
#4793 StartFinishScope netcoreapp3.1 667ns 0.112ns 0.418ns 0.00901 0 0 656 B
#4793 StartFinishScope net472 932ns 0.486ns 1.88ns 0.0979 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 612ns 0.185ns 0.717ns 0.00932 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 791ns 0.287ns 1.07ns 0.00876 0 0 656 B
master RunOnMethodBegin net472 955ns 0.226ns 0.876ns 0.0978 0 0 618 B
#4793 RunOnMethodBegin net6.0 596ns 0.386ns 1.5ns 0.00937 0 0 656 B
#4793 RunOnMethodBegin netcoreapp3.1 785ns 0.281ns 1.09ns 0.00904 0 0 656 B
#4793 RunOnMethodBegin net472 935ns 0.249ns 0.93ns 0.0978 0 0 618 B

@andrewlock
Copy link
Member

andrewlock commented Nov 1, 2023

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

Loading
gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4793) (10.740M)   : 0, 10739800
    master (10.293M)   : 0, 10292812
    benchmarks/2.38.0 (11.867M)   : 0, 11866956
    benchmarks/2.9.0 (10.958M)   : 0, 10957550

    section Automatic
    This PR (4793) (7.675M)   : 0, 7675433
    master (7.626M)   : 0, 7626259
    benchmarks/2.38.0 (8.176M)   : 0, 8175636
    benchmarks/2.9.0 (7.908M)   : 0, 7907971

    section Trace stats
    This PR (4793) (8.140M)   : 0, 8139630
    master (7.981M)   : 0, 7980764
    benchmarks/2.38.0 (8.450M)   : 0, 8450370

    section Manual
    This PR (4793) (10.062M)   : 0, 10062003
    master (9.696M)   : 0, 9696455
    benchmarks/2.38.0 (10.334M)   : 0, 10334368

    section Manual + Automatic
    This PR (4793) (7.439M)   : 0, 7438766
    master (7.342M)   : 0, 7342321
    benchmarks/2.38.0 (7.750M)   : 0, 7750484

    section Version Conflict
    This PR (4793) (6.985M)   : 0, 6984906
    master (6.511M)   : 0, 6510524
    benchmarks/2.38.0 (7.137M)   : 0, 7136691

Loading
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4793) (9.629M)   : 0, 9628712
    master (9.373M)   : 0, 9372820
    benchmarks/2.38.0 (9.548M)   : 0, 9548121
    benchmarks/2.9.0 (9.589M)   : 0, 9589306

    section Automatic
    This PR (4793) (6.666M)   : 0, 6666157
    master (6.736M)   : 0, 6736494
    benchmarks/2.38.0 (6.747M)   : 0, 6747227

    section Trace stats
    This PR (4793) (6.868M)   : 0, 6867551
    master (6.838M)   : 0, 6838459
    benchmarks/2.38.0 (6.815M)   : 0, 6814846

    section Manual
    This PR (4793) (8.508M)   : 0, 8507776
    master (8.359M)   : 0, 8359027
    benchmarks/2.38.0 (8.263M)   : 0, 8263131

    section Manual + Automatic
    This PR (4793) (6.291M)   : 0, 6291175
    master (6.341M)   : 0, 6341109
    benchmarks/2.38.0 (6.275M)   : 0, 6275411

    section Version Conflict
    This PR (4793) (5.850M)   : 0, 5850289
    master (5.798M)   : 0, 5797802
    benchmarks/2.38.0 (5.670M)   : 0, 5669744

Loading
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4793) (9.257M)   : 0, 9257078
    master (9.505M)   : 0, 9504792
    benchmarks/2.38.0 (9.007M)   : 0, 9007066
    benchmarks/2.9.0 (9.595M)   : 0, 9594704

    section Automatic
    This PR (4793) (6.619M)   : 0, 6619361
    master (6.739M)   : 0, 6739142
    benchmarks/2.38.0 (6.423M)   : 0, 6422756
    benchmarks/2.9.0 (7.049M)   : 0, 7048800

    section Trace stats
    This PR (4793) (6.984M)   : 0, 6983614
    master (7.014M)   : 0, 7013989
    benchmarks/2.38.0 (6.721M)   : 0, 6720838

    section Manual
    This PR (4793) (8.192M)   : 0, 8191971
    master (8.381M)   : 0, 8380765
    benchmarks/2.38.0 (7.977M)   : 0, 7977494

    section Manual + Automatic
    This PR (4793) (6.336M)   : 0, 6335754
    master (6.515M)   : 0, 6515344
    benchmarks/2.38.0 (6.349M)   : 0, 6349066

    section Version Conflict
    This PR (4793) (5.908M)   : 0, 5908031
    master (6.043M)   : 0, 6043023
    benchmarks/2.38.0 (5.682M)   : 0, 5681654

Loading
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4793) (7.772M)   : 0, 7771787
    master (7.912M)   : 0, 7912308
    benchmarks/2.38.0 (7.578M)   : 0, 7577544
    benchmarks/2.9.0 (7.477M)   : 0, 7476547

    section No attack
    This PR (4793) (2.253M)   : 0, 2252826
    master (2.269M)   : 0, 2269345
    benchmarks/2.38.0 (2.184M)   : 0, 2184279
    benchmarks/2.9.0 (3.151M)   : 0, 3151044

    section Attack
    This PR (4793) (1.792M)   : 0, 1791747
    master (1.782M)   : 0, 1782477
    benchmarks/2.38.0 (1.711M)   : 0, 1710542
    benchmarks/2.9.0 (2.361M)   : 0, 2361022

    section Blocking
    This PR (4793) (3.606M)   : 0, 3605526
    master (3.615M)   : 0, 3614991
    benchmarks/2.38.0 (3.486M)   : 0, 3486124

    section IAST default
    This PR (4793) (7.075M)   : 0, 7075058
    master (7.079M)   : 0, 7078508

    section IAST full
    This PR (4793) (6.364M)   : 0, 6363765
    master (6.347M)   : 0, 6346955

    section Base vuln
    This PR (4793) (0.964M)   : 0, 964010
    master (0.957M)   : 0, 956568

    section IAST vuln
    This PR (4793) (0.968M)   : 0, 968299
    master (0.931M)   : 0, 930647

@daniel-romano-DD daniel-romano-DD marked this pull request as ready for review November 1, 2023 19:11
@daniel-romano-DD daniel-romano-DD requested a review from a team as a code owner November 1, 2023 19:11
@daniel-romano-DD daniel-romano-DD merged commit 00d32ab into master Nov 2, 2023
@daniel-romano-DD daniel-romano-DD deleted the dani/asm/join_bug_fix branch November 2, 2023 09:15
@github-actions github-actions bot added this to the vNext milestone Nov 2, 2023
@@ -79,6 +79,7 @@ static const WSTRING _fixedAssemblyExcludeFilters[] = {
WStr("testhost.*"),
WStr("Oracle.ManagedDataAccess"),
WStr("DelegateDecompiler*"),
WStr("FluentValidation*"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really needed, but convenient, for performance reasons

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants