Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TESTS]Added BenchmarkDotNet debug helper #4717

Merged
merged 5 commits into from
Nov 3, 2023
Merged

[TESTS]Added BenchmarkDotNet debug helper #4717

merged 5 commits into from
Nov 3, 2023

Conversation

daniel-romano-DD
Copy link
Contributor

Summary of changes

Implemented helper functions to launch, by using reflection, a benchmark in debug with the debugger attached.

Reason for change

BenchmarkDotNet would not allow (or make easy) to debug a benchmark.

Implementation details

Implemented reflection relying functions to discover benchmark envelope and launch it mimicking what BenchmarkDotNet would do.

@daniel-romano-DD daniel-romano-DD added the area:tests unit tests, integration tests label Oct 11, 2023
@daniel-romano-DD daniel-romano-DD changed the title Added BenchmarkDotNet debug helper [TESTS]Added BenchmarkDotNet debug helper Oct 11, 2023
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Oct 11, 2023 •
edited
Loading

Datadog Report

Branch report: dani/testing/benchmark_debug
Commit report: a1c1550

❌ dd-trace-dotnet: 1 Failed (0 Known Flaky), 0 New Flaky, 307324 Passed, 1138 Skipped, 30m 52.43s Wall Time

❌ Failed Tests (1)

  • CheckGitMetataFromEnvironmentVariablesIfTracerFeatureIsDisabled - Datadog.Profiler.IntegrationTests.DebugInfo.GitMetadataTest - Details

    Expand for error 
     Exit code of "catchsegv" should be 0 instead of 139 (= 0x8B)
 Expected: True
 Actual:   False

@andrewlock
Copy link
Member

andrewlock commented Oct 11, 2023 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4717) - mean (71ms)  : 64, 79
     .   : milestone, 71,
    master - mean (71ms)  : 63, 78
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4717) - mean (1,002ms)  : 981, 1023
     .   : milestone, 1002,
    master - mean (998ms)  : 979, 1017
     .   : milestone, 998,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4717) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (106ms)  : 103, 109
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4717) - mean (689ms)  : 666, 712
     .   : milestone, 689,
    master - mean (686ms)  : 669, 703
     .   : milestone, 686,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4717) - mean (90ms)  : 87, 93
     .   : milestone, 90,
    master - mean (90ms)  : 87, 94
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4717) - mean (656ms)  : 635, 678
     .   : milestone, 656,
    master - mean (659ms)  : 634, 683
     .   : milestone, 659,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4717) - mean (187ms)  : 184, 191
     .   : milestone, 187,
    master - mean (187ms)  : 184, 189
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (4717) - mean (1,131ms)  : 1110, 1152
     .   : milestone, 1131,
    master - mean (1,127ms)  : 1107, 1148
     .   : milestone, 1127,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4717) - mean (271ms)  : 267, 275
     .   : milestone, 271,
    master - mean (271ms)  : 266, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4717) - mean (1,100ms)  : 1075, 1125
     .   : milestone, 1100,
    master - mean (1,098ms)  : 1077, 1119
     .   : milestone, 1098,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4717) - mean (262ms)  : 258, 266
     .   : milestone, 262,
    master - mean (261ms)  : 258, 264
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4717) - mean (1,062ms)  : 1031, 1092
     .   : milestone, 1062,
    master - mean (1,064ms)  : 1041, 1088
     .   : milestone, 1064,
Loading

@daniel-romano-DD daniel-romano-DD marked this pull request as ready for review October 11, 2023 15:26
@daniel-romano-DD daniel-romano-DD requested a review from a team as a code owner October 11, 2023 15:26
tonyredondo
tonyredondo approved these changes Oct 13, 2023
View reviewed changes
tracer/test/benchmarks/Benchmarks.Trace/Program.cs Outdated
@@ -67,6 +78,52 @@ private static void Main(string[] args)

Console.WriteLine("Running tests...");
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);

return 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we check if BenchmarkSwitcher.Run changes the Environment.ExitCode? Maybe to play safe we can do return Environment.ExitCode; here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1b1c7ab

anna-git
anna-git reviewed Nov 3, 2023
View reviewed changes
tracer/test/benchmarks/Benchmarks.Trace/Program.cs Outdated
@@ -8,13 +8,24 @@
using BenchmarkDotNet.Filters;
using Benchmarks.Trace.DatadogProfiler;
using Benchmarks.Trace.Jetbrains;
using Benchmarks.Trace.Iast;
Copy link
Contributor

@anna-git anna-git Nov 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

namespace doesnt exist in the project, did you mean to add some iast benchmarks?

Copy link
Contributor Author

@daniel-romano-DD daniel-romano-DD Nov 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right. This is cherrypicked from the IAST micro benchmark PR. Fixed in 84c58fe

@daniel-romano-DD
Update Program.cs
84c58fe 
Removed unnecessary using clause
anna-git
anna-git reviewed Nov 3, 2023
View reviewed changes
tracer/test/benchmarks/Benchmarks.Trace/Program.cs Outdated
}

// Retrieve the Benchmark methods
var benchmarkMethods = typeof(T).GetMethods().Where(m => Attribute.GetCustomAttribute(m, typeof(BenchmarkDotNet.Attributes.BenchmarkAttribute)) != null).ToList();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchmarkMethods is unused?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 204350f

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 3, 2023 •
edited
Loading

Datadog Report

Branch report: dani/testing/benchmark_debug
Commit report: 1b1c7ab

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 296735 Passed, 940 Skipped, 36m 25.87s Wall Time

@andrewlock
Copy link
Member

Benchmarks Report 🐌

Benchmarks for #4717 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.164
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.31μs 45.7ns 289ns 0.0172 0.00858 0 7.29 KB
master StartStopWithChild netcoreapp3.1 10.1μs 56.9ns 377ns 0.034 0.0146 0 7.39 KB
master StartStopWithChild net472 15.5μs 46.8ns 181ns 1.3 0.319 0.109 7.66 KB
#4717 StartStopWithChild net6.0 8.31μs 38.3ns 184ns 0.0202 0.00809 0 7.29 KB
#4717 StartStopWithChild netcoreapp3.1 10.1μs 55.2ns 322ns 0.0288 0.0144 0 7.38 KB
#4717 StartStopWithChild net472 15.6μs 59.9ns 232ns 1.29 0.326 0.101 7.66 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 489μs 229ns 858ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 632μs 189ns 708ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 819μs 476ns 1.84μs 0.408 0 0 3.3 KB
#4717 WriteAndFlushEnrichedTraces net6.0 471μs 282ns 1.09μs 0 0 0 2.7 KB
#4717 WriteAndFlushEnrichedTraces netcoreapp3.1 643μs 330ns 1.28μs 0 0 0 2.7 KB
#4717 WriteAndFlushEnrichedTraces net472 808μs 419ns 1.57μs 0.401 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 42.9μs 47.4ns 184ns 0.0218 0 0 2.03 KB
master AllCycleSimpleBody netcoreapp3.1 45.3μs 127ns 492ns 0.0221 0 0 2.01 KB
master AllCycleSimpleBody net472 47.3μs 132ns 513ns 0.326 0 0 2.08 KB
master AllCycleMoreComplexBody net6.0 228μs 111ns 428ns 0 0 0 8.63 KB
master AllCycleMoreComplexBody netcoreapp3.1 235μs 62.6ns 243ns 0 0 0 8.52 KB
master AllCycleMoreComplexBody net472 239μs 72.3ns 280ns 1.31 0 0 8.7 KB
master ObjectExtractorSimpleBody net6.0 121ns 0.124ns 0.464ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 179ns 0.229ns 0.855ns 0.00377 0 0 272 B
master ObjectExtractorSimpleBody net472 144ns 0.132ns 0.51ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.98μs 0.822ns 3.08ns 0.0535 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.06μs 1.25ns 4.51ns 0.0509 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.1μs 2.55ns 9.87ns 0.618 0.00616 0 3.89 KB
#4717 AllCycleSimpleBody net6.0 43.4μs 54.6ns 212ns 0.0218 0 0 2.03 KB
#4717 AllCycleSimpleBody netcoreapp3.1 45μs 38.2ns 143ns 0.0227 0 0 2.01 KB
#4717 AllCycleSimpleBody net472 46.9μs 128ns 497ns 0.322 0 0 2.08 KB
#4717 AllCycleMoreComplexBody net6.0 227μs 63.7ns 239ns 0 0 0 8.63 KB
#4717 AllCycleMoreComplexBody netcoreapp3.1 233μs 95.9ns 372ns 0 0 0 8.52 KB
#4717 AllCycleMoreComplexBody net472 240μs 98.8ns 383ns 1.31 0 0 8.7 KB
#4717 ObjectExtractorSimpleBody net6.0 118ns 0.0884ns 0.342ns 0.00393 0 0 280 B
#4717 ObjectExtractorSimpleBody netcoreapp3.1 171ns 0.0788ns 0.295ns 0.00367 0 0 272 B
#4717 ObjectExtractorSimpleBody net472 146ns 0.132ns 0.51ns 0.0446 0 0 281 B
#4717 ObjectExtractorMoreComplexBody net6.0 3.14μs 1.51ns 5.64ns 0.0542 0 0 3.88 KB
#4717 ObjectExtractorMoreComplexBody netcoreapp3.1 4.11μs 1.79ns 6.95ns 0.0511 0 0 3.78 KB
#4717 ObjectExtractorMoreComplexBody net472 4.19μs 3.54ns 12.8ns 0.618 0.0062 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 12.6μs 4.95ns 18.5ns 0.132 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 19.6μs 9.61ns 34.6ns 0.127 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) net472 27.9μs 7.2ns 26ns 1.5 0 0 9.48 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 75.8μs 110ns 426ns 0.191 0 0 15.77 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 85μs 231ns 894ns 0.168 0 0 15.72 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 97.2μs 60.1ns 225ns 2.52 0 0 16.04 KB
master RunWaf(args=NestedMap (100)) net6.0 24.6μs 5.45ns 19.7ns 0.271 0 0 19.66 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 38.4μs 26.6ns 99.7ns 0.27 0 0 20.42 KB
master RunWaf(args=NestedMap (100)) net472 54.2μs 22ns 85ns 3.28 0.0542 0 20.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 90.3μs 30.6ns 110ns 0.362 0 0 26.01 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 109μs 406ns 1.47μs 0.32 0 0 26.72 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 133μs 177ns 687ns 4.27 0.0657 0 27.19 KB
master RunWaf(args=NestedMap (20)) net6.0 24.2μs 10.6ns 39.5ns 0.267 0 0 19.4 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 37.8μs 26.2ns 101ns 0.263 0 0 19.84 KB
master RunWaf(args=NestedMap (20)) net472 54.4μs 12.5ns 43.4ns 3.17 0.0542 0 20.04 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 89.9μs 37.4ns 145ns 0.362 0 0 25.74 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 109μs 111ns 415ns 0.33 0 0 26.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 133μs 86.8ns 336ns 4.2 0.0657 0 26.6 KB
#4717 RunWaf(args=NestedMap (10)) net6.0 12.5μs 4.43ns 16.6ns 0.131 0 0 9.42 KB
#4717 RunWaf(args=NestedMap (10)) netcoreapp3.1 19.6μs 9.26ns 34.6ns 0.128 0 0 9.42 KB
#4717 RunWaf(args=NestedMap (10)) net472 28μs 13.3ns 49.6ns 1.5 0 0 9.48 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 75.5μs 26.1ns 101ns 0.189 0 0 15.77 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 85.2μs 201ns 779ns 0.215 0 0 15.72 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [22]) net472 97.5μs 222ns 859ns 2.53 0 0 16.04 KB
#4717 RunWaf(args=NestedMap (100)) net6.0 24.1μs 55ns 198ns 0.276 0 0 19.66 KB
#4717 RunWaf(args=NestedMap (100)) netcoreapp3.1 38.2μs 12.6ns 48.8ns 0.267 0 0 20.42 KB
#4717 RunWaf(args=NestedMap (100)) net472 54.4μs 16ns 59.7ns 3.26 0.0272 0 20.63 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 91.3μs 56.3ns 218ns 0.363 0 0 26.01 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 109μs 234ns 906ns 0.322 0 0 26.72 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [23]) net472 131μs 129ns 499ns 4.27 0.0657 0 27.19 KB
#4717 RunWaf(args=NestedMap (20)) net6.0 23.7μs 8.17ns 31.7ns 0.271 0 0 19.4 KB
#4717 RunWaf(args=NestedMap (20)) netcoreapp3.1 37.5μs 20.1ns 77.8ns 0.262 0 0 19.84 KB
#4717 RunWaf(args=NestedMap (20)) net472 53.9μs 23.5ns 90.8ns 3.17 0.0538 0 20.04 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 90.8μs 54.5ns 211ns 0.362 0 0 25.74 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 109μs 188ns 727ns 0.33 0 0 26.14 KB
#4717 RunWafWithAttack(args=Neste(...)tack) [22]) net472 132μs 71.3ns 267ns 4.21 0.0648 0 26.6 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 170μs 169ns 654ns 0.17 0 0 18.04 KB
master SendRequest netcoreapp3.1 191μs 328ns 1.27μs 0.192 0 0 20.2 KB
master SendRequest net472 0.000356ns 0.00017ns 0.00066ns 0 0 0 0 b
#4717 SendRequest net6.0 166μs 167ns 625ns 0.249 0 0 18.04 KB
#4717 SendRequest netcoreapp3.1 189μs 244ns 943ns 0.187 0 0 20.2 KB
#4717 SendRequest net472 0.000333ns 0.000201ns 0.000753ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 559μs 791ns 2.96μs 0.543 0 0 41.75 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 651μs 1.71μs 6.63μs 0.331 0 0 41.73 KB
master WriteAndFlushEnrichedTraces net472 827μs 3.83μs 15.3μs 8.13 2.57 0.428 53.23 KB
#4717 WriteAndFlushEnrichedTraces net6.0 561μs 308ns 1.15μs 0.543 0 0 41.65 KB
#4717 WriteAndFlushEnrichedTraces netcoreapp3.1 657μs 380ns 1.42μs 0.329 0 0 41.67 KB
#4717 WriteAndFlushEnrichedTraces net472 823μs 3.33μs 12.9μs 8.06 2.42 0.403 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4717

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0 1.155 1,065.17 922.60

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.06μs 0.754ns 2.92ns 0.0106 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.36μs 0.455ns 1.76ns 0.0103 0 0 768 B
master ExecuteNonQuery net472 1.68μs 1.09ns 4.2ns 0.116 0 0 730 B
#4717 ExecuteNonQuery net6.0 922ns 0.274ns 1.06ns 0.0106 0 0 768 B
#4717 ExecuteNonQuery netcoreapp3.1 1.28μs 1.36ns 5.27ns 0.0104 0 0 768 B
#4717 ExecuteNonQuery net472 1.57μs 0.563ns 2.03ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.11μs 0.558ns 2.16ns 0.0132 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.42μs 0.669ns 2.41ns 0.0126 0 0 936 B
master CallElasticsearch net472 2.37μs 0.979ns 3.79ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.27μs 0.478ns 1.79ns 0.0127 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.45μs 0.501ns 1.74ns 0.0132 0 0 984 B
master CallElasticsearchAsync net472 2.47μs 0.978ns 3.66ns 0.16 0 0 1.01 KB
#4717 CallElasticsearch net6.0 1.21μs 0.463ns 1.73ns 0.0127 0 0 936 B
#4717 CallElasticsearch netcoreapp3.1 1.43μs 0.54ns 2.09ns 0.0129 0 0 936 B
#4717 CallElasticsearch net472 2.34μs 0.528ns 1.97ns 0.151 0 0 955 B
#4717 CallElasticsearchAsync net6.0 1.22μs 0.461ns 1.72ns 0.0129 0 0 912 B
#4717 CallElasticsearchAsync netcoreapp3.1 1.47μs 0.44ns 1.71ns 0.0133 0 0 984 B
#4717 CallElasticsearchAsync net472 2.44μs 1.56ns 6.06ns 0.161 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.27μs 1.19ns 4.46ns 0.0127 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.44μs 0.74ns 2.67ns 0.0121 0 0 912 B
master ExecuteAsync net472 1.64μs 0.939ns 3.63ns 0.139 0 0 875 B
#4717 ExecuteAsync net6.0 1.15μs 0.629ns 2.44ns 0.0128 0 0 912 B
#4717 ExecuteAsync netcoreapp3.1 1.47μs 0.75ns 2.81ns 0.0124 0 0 912 B
#4717 ExecuteAsync net472 1.69μs 1.12ns 4.34ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.7μs 0.911ns 3.41ns 0.0259 0 0 1.9 KB
master SendAsync netcoreapp3.1 4.52μs 1.23ns 4.62ns 0.0318 0 0 2.43 KB
master SendAsync net472 7.12μs 2.34ns 9.07ns 0.473 0 0 2.99 KB
#4717 SendAsync net6.0 3.8μs 1.38ns 4.77ns 0.0266 0 0 1.9 KB
#4717 SendAsync netcoreapp3.1 4.43μs 1.42ns 5.33ns 0.0333 0 0 2.43 KB
#4717 SendAsync net472 7.04μs 3.74ns 14.5ns 0.475 0 0 2.99 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.36μs 0.601ns 2.25ns 0.0222 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 1.95μs 1.69ns 6.33ns 0.0211 0 0 1.57 KB
master EnrichedLog net472 2.34μs 1.53ns 5.94ns 0.236 0 0 1.49 KB
#4717 EnrichedLog net6.0 1.38μs 0.752ns 2.91ns 0.0221 0 0 1.57 KB
#4717 EnrichedLog netcoreapp3.1 2.08μs 1.05ns 4.08ns 0.0207 0 0 1.57 KB
#4717 EnrichedLog net472 2.26μs 1.11ns 4.16ns 0.236 0 0 1.49 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 96ns 359ns 0 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 117μs 133ns 513ns 0.0587 0 0 4.21 KB
master EnrichedLog net472 149μs 142ns 550ns 0.671 0.224 0 4.38 KB
#4717 EnrichedLog net6.0 112μs 83.1ns 322ns 0 0 0 4.21 KB
#4717 EnrichedLog netcoreapp3.1 119μs 307ns 1.19μs 0 0 0 4.21 KB
#4717 EnrichedLog net472 147μs 70.8ns 265ns 0.654 0.218 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.71μs 1.02ns 3.82ns 0.03 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 3.91μs 0.861ns 3.22ns 0.0274 0 0 2.13 KB
master EnrichedLog net472 4.5μs 1.24ns 4.64ns 0.306 0 0 1.93 KB
#4717 EnrichedLog net6.0 2.78μs 0.793ns 2.97ns 0.0293 0 0 2.13 KB
#4717 EnrichedLog netcoreapp3.1 3.87μs 1.25ns 4.85ns 0.0288 0 0 2.13 KB
#4717 EnrichedLog net472 4.58μs 3.34ns 12.5ns 0.306 0 0 1.93 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.33μs 0.538ns 2.08ns 0.0153 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.67μs 1.06ns 4.1ns 0.0151 0 0 1.1 KB
master SendReceive net472 2.02μs 1.77ns 6.84ns 0.177 0 0 1.12 KB
#4717 SendReceive net6.0 1.3μs 0.967ns 3.49ns 0.0156 0 0 1.1 KB
#4717 SendReceive netcoreapp3.1 1.64μs 2.56ns 9.92ns 0.0147 0 0 1.1 KB
#4717 SendReceive net472 2.04μs 1.99ns 7.71ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.55μs 5.25ns 20.3ns 0.0215 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.54μs 1.49ns 5.76ns 0.0213 0 0 1.58 KB
master EnrichedLog net472 4.11μs 1.76ns 6.81ns 0.311 0 0 1.96 KB
#4717 EnrichedLog net6.0 2.69μs 0.814ns 2.82ns 0.0214 0 0 1.53 KB
#4717 EnrichedLog netcoreapp3.1 3.57μs 1.56ns 6.03ns 0.0216 0 0 1.58 KB
#4717 EnrichedLog net472 4.05μs 6.76ns 26.2ns 0.311 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4717

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.174 459.76 391.71

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 460ns 0.183ns 0.709ns 0.0076 0 0 536 B
master StartFinishSpan netcoreapp3.1 576ns 0.209ns 0.753ns 0.0072 0 0 536 B
master StartFinishSpan net472 661ns 0.351ns 1.36ns 0.0852 0 0 538 B
master StartFinishScope net6.0 557ns 0.18ns 0.696ns 0.00922 0 0 656 B
master StartFinishScope netcoreapp3.1 728ns 0.232ns 0.899ns 0.00875 0 0 656 B
master StartFinishScope net472 935ns 0.312ns 1.21ns 0.0978 0 0 618 B
#4717 StartFinishSpan net6.0 392ns 0.196ns 0.733ns 0.00747 0 0 536 B
#4717 StartFinishSpan netcoreapp3.1 529ns 0.319ns 1.11ns 0.00745 0 0 536 B
#4717 StartFinishSpan net472 696ns 0.163ns 0.632ns 0.0854 0 0 538 B
#4717 StartFinishScope net6.0 526ns 0.422ns 1.58ns 0.00922 0 0 656 B
#4717 StartFinishScope netcoreapp3.1 673ns 0.294ns 1.14ns 0.00887 0 0 656 B
#4717 StartFinishScope net472 853ns 0.192ns 0.72ns 0.0979 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 615ns 0.276ns 1.07ns 0.00922 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 792ns 0.445ns 1.72ns 0.00869 0 0 656 B
master RunOnMethodBegin net472 1.02μs 0.604ns 2.34ns 0.0978 0 0 618 B
#4717 RunOnMethodBegin net6.0 661ns 0.649ns 2.51ns 0.00926 0 0 656 B
#4717 RunOnMethodBegin netcoreapp3.1 830ns 0.444ns 1.72ns 0.00878 0 0 656 B
#4717 RunOnMethodBegin net472 994ns 0.244ns 0.945ns 0.0979 0 0 618 B

andrewlock
andrewlock approved these changes Nov 3, 2023
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not :D

@daniel-romano-DD daniel-romano-DD merged commit f1cf405 into master Nov 3, 2023
@daniel-romano-DD daniel-romano-DD deleted the dani/testing/benchmark_debug branch November 3, 2023 12:04
@github-actions github-actions bot added this to the vNext milestone Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock approved these changes

@anna-git anna-git anna-git approved these changes

@tonyredondo tonyredondo tonyredondo approved these changes

@robertpi robertpi robertpi approved these changes

Assignees
No one assigned
Labels
area:tests unit tests, integration tests
Projects
None yet
Milestone
2.41.0
Development

Successfully merging this pull request may close these issues.
5 participants
@daniel-romano-DD @andrewlock @robertpi @tonyredondo @anna-git