Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kevin/randomize tests 2 #6508

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Kevin/randomize tests 2 #6508

wants to merge 4 commits into from
+180 −560

Conversation

kevingosse
Copy link
Collaborator

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

@andrewlock
Copy link
Member

andrewlock commented Jan 7, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6508) - mean (69ms)  : 66, 71
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6508) - mean (984ms)  : 958, 1010
     .   : milestone, 984,
    master - mean (981ms)  : 954, 1008
     .   : milestone, 981,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6508) - mean (108ms)  : 106, 109
     .   : milestone, 108,
    master - mean (107ms)  : 105, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6508) - mean (678ms)  : 658, 697
     .   : milestone, 678,
    master - mean (678ms)  : 662, 695
     .   : milestone, 678,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6508) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 90, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6508) - mean (636ms)  : 621, 652
     .   : milestone, 636,
    master - mean (633ms)  : 616, 651
     .   : milestone, 633,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6508) - mean (194ms)  : 190, 198
     .   : milestone, 194,
    master - mean (195ms)  : 189, 202
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6508) - mean (1,101ms)  : 1060, 1142
     .   : milestone, 1101,
    master - mean (1,104ms)  : 1076, 1132
     .   : milestone, 1104,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6508) - mean (279ms)  : 273, 285
     .   : milestone, 279,
    master - mean (278ms)  : 273, 282
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6508) - mean (863ms)  : 840, 886
     .   : milestone, 863,
    master - mean (874ms)  : 847, 901
     .   : milestone, 874,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6508) - mean (268ms)  : 264, 272
     .   : milestone, 268,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6508) - mean (851ms)  : 817, 885
     .   : milestone, 851,
    master - mean (857ms)  : 829, 885
     .   : milestone, 857,
Loading

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jan 7, 2025 •
edited
Loading

Datadog Report

Branch report: kevin/randomize_tests_2
Commit report: aa88b9f
Test service: dd-trace-dotnet

❌ 24 Failed (0 Known Flaky), 591228 Passed, 3869 Skipped, 34h 4m 35.66s Total Time
⌛ 1 Performance Regression

❌ Failed Tests (24)

This report shows up to 5 failed tests.

  • CoberturaCodeCoverage - Datadog.Trace.Tools.Runner.IntegrationTests.CiRunCommandTests - Details

    Expand for error 
     Expected exitCode to be 0, but found 1 (difference of 1).
 Expected callbackInvoked to be true, but found False.
 Expected command to be "test.exe", but found <null>.
 Expected environmentVariables not to be <null>.
 Expected testSession not to be <null>.
 
 With output:
  [FAILURE]: Error: The specified home folder doesn't exist.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/net9.0/TestTracerHome/netstandard2.0/Datadog.Trace.MSBuild.dll' can't be found.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/net9.0/TestTracerHome/linux-x64/Datadog.Trace.ClrProfiler.Native.so' can't be found.
 ...

  • CoberturaCodeCoverage - Datadog.Trace.Tools.Runner.IntegrationTests.CiRunCommandTests - Details

    Expand for error 
     Expected callbackInvoked to be 0, but found 1 (difference of 1).
 Expected command to be true, but found False.
 Expected arguments to be "test.exe", but found <null>.
 Expected testSession not to be <null>.
 Expected testSession.Meta not to be <null>.
 
 With output:
  [FAILURE]: Error: The specified home folder doesn't exist.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/netcoreapp2.1/TestTracerHome/netstandard2.0/Datadog.Trace.MSBuild.dll' can't be found.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/netcoreapp2.1/TestTracerHome/linux-x64/Datadog.Trace.ClrProfiler.Native.so' can't be found.
 ...

  • CoberturaCodeCoverage - Datadog.Trace.Tools.Runner.IntegrationTests.CiRunCommandTests - Details

    Expand for error 
     Expected callbackInvoked to be 0, but found 1 (difference of 1).
 Expected command to be true, but found False.
 Expected arguments to be "test.exe", but found <null>.
 Expected testSession not to be <null>.
 Expected testSession.Meta not to be <null>.
 
 With output:
  [FAILURE]: Error: The specified home folder doesn't exist.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/netcoreapp3.0/TestTracerHome/netstandard2.0/Datadog.Trace.MSBuild.dll' can't be found.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/netcoreapp3.0/TestTracerHome/linux-x64/Datadog.Trace.ClrProfiler.Native.so' can't be found.
 ...

  • CoberturaCodeCoverage - Datadog.Trace.Tools.Runner.IntegrationTests.CiRunCommandTests - Details

    Expand for error 
     Expected exitCode to be 0, but found 1 (difference of 1).
 Expected callbackInvoked to be true, but found False.
 Expected command to be "test.exe", but found <null>.
 Expected environmentVariables not to be <null>.
 Expected testSession not to be <null>.
 
 With output:
  [FAILURE]: Error: The specified home folder doesn't exist.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/net6.0/TestTracerHome/netstandard2.0/Datadog.Trace.MSBuild.dll' can't be found.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/net6.0/TestTracerHome/linux-musl-x64/Datadog.Trace.ClrProfiler.Native.so' can't be found.
 ...

  • CoberturaCodeCoverage - Datadog.Trace.Tools.Runner.IntegrationTests.CiRunCommandTests - Details

    Expand for error 
     Expected exitCode to be 0, but found 1 (difference of 1).
 Expected callbackInvoked to be true, but found False.
 Expected command to be "test.exe", but found <null>.
 Expected environmentVariables not to be <null>.
 Expected testSession not to be <null>.
 
 With output:
  [FAILURE]: Error: The specified home folder doesn't exist.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/net5.0/TestTracerHome/netstandard2.0/Datadog.Trace.MSBuild.dll' can't be found.
  [FAILURE]: Error: The file '/project/tracer/test/Datadog.Trace.Tools.Runner.IntegrationTests/bin/Release/net5.0/TestTracerHome/linux-x64/Datadog.Trace.ClrProfiler.Native.so' can't be found.
 ...

⌛ Performance Regressions vs Default Branch (1)

  • ObjectExtractorSimpleBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 183ns (+41ns, +29%) - Details

@andrewlock
Copy link
Member

andrewlock commented Jan 7, 2025 •
edited
Loading

Benchmarks Report for appsec 🐌

Benchmarks for #6508 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.140
  • 2 benchmarks have fewer allocations
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6508

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.140 160.62 183.06

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 202μs 169ns 655ns 2.61 0 0 188.76 KB
master AllCycleSimpleBody netcoreapp3.1 299μs 200ns 773ns 2.68 0 0 196.09 KB
master AllCycleSimpleBody net472 273μs 118ns 441ns 35.8 2.05 0 225.79 KB
master AllCycleMoreComplexBody net6.0 209μs 123ns 459ns 2.72 0 0 192.26 KB
master AllCycleMoreComplexBody netcoreapp3.1 317μs 139ns 520ns 2.68 0 0 199.5 KB
master AllCycleMoreComplexBody net472 278μs 174ns 674ns 36.4 2.08 0 229.3 KB
master ObjectExtractorSimpleBody net6.0 161ns 0.13ns 0.486ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 195ns 0.139ns 0.52ns 0.00373 0 0 272 B
master ObjectExtractorSimpleBody net472 168ns 0.0874ns 0.338ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.91μs 1.13ns 4.07ns 0.0524 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.76μs 3.16ns 11.8ns 0.0488 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.63μs 2.34ns 8.75ns 0.603 0.00722 0 3.8 KB
#6508 AllCycleSimpleBody net6.0 205μs 92ns 344ns 2.67 0 0 188.76 KB
#6508 AllCycleSimpleBody netcoreapp3.1 297μs 130ns 485ns 2.68 0 0 196.09 KB
#6508 AllCycleSimpleBody net472 274μs 166ns 622ns 35.8 2.05 0 225.79 KB
#6508 AllCycleMoreComplexBody net6.0 211μs 113ns 424ns 2.72 0 0 192.27 KB
#6508 AllCycleMoreComplexBody netcoreapp3.1 320μs 134ns 518ns 2.71 0 0 199.51 KB
#6508 AllCycleMoreComplexBody net472 276μs 198ns 767ns 36.4 2.06 0 229.3 KB
#6508 ObjectExtractorSimpleBody net6.0 183ns 0.14ns 0.543ns 0.00397 0 0 280 B
#6508 ObjectExtractorSimpleBody netcoreapp3.1 200ns 0.139ns 0.5ns 0.00367 0 0 272 B
#6508 ObjectExtractorSimpleBody net472 164ns 0.111ns 0.43ns 0.0446 0 0 281 B
#6508 ObjectExtractorMoreComplexBody net6.0 2.87μs 1.42ns 5.51ns 0.0534 0 0 3.78 KB
#6508 ObjectExtractorMoreComplexBody netcoreapp3.1 3.79μs 2.29ns 8.85ns 0.0494 0 0 3.69 KB
#6508 ObjectExtractorMoreComplexBody net472 3.78μs 1.8ns 6.75ns 0.602 0.00632 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 36.7μs 26.7ns 104ns 0.457 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 55.2μs 33.8ns 131ns 0.412 0 0 32.4 KB
master EncodeArgs net472 67.9μs 47.8ns 179ns 5.15 0.0678 0 32.5 KB
master EncodeLegacyArgs net6.0 74.4μs 22.5ns 84.3ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 105μs 155ns 602ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 156μs 56.3ns 218ns 0.312 0 0 2.15 KB
#6508 EncodeArgs net6.0 39.1μs 10.3ns 37.3ns 0.45 0 0 32.4 KB
#6508 EncodeArgs netcoreapp3.1 53.6μs 28.7ns 107ns 0.428 0 0 32.4 KB
#6508 EncodeArgs net472 67.4μs 38.9ns 151ns 5.14 0.0664 0 32.5 KB
#6508 EncodeLegacyArgs net6.0 76.7μs 422ns 2.42μs 0.0364 0 0 2.14 KB
#6508 EncodeLegacyArgs netcoreapp3.1 105μs 124ns 482ns 0 0 0 2.14 KB
#6508 EncodeLegacyArgs net472 154μs 123ns 476ns 0.309 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 174μs 90.3ns 338ns 0 0 0 2.44 KB
master RunWafRealisticBenchmark netcoreapp3.1 187μs 225ns 812ns 0 0 0 2.39 KB
master RunWafRealisticBenchmark net472 199μs 119ns 462ns 0.3 0 0 2.46 KB
master RunWafRealisticBenchmarkWithAttack net6.0 114μs 65.1ns 252ns 0 0 0 1.47 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 124μs 279ns 1.08μs 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack net472 132μs 51.1ns 198ns 0.198 0 0 1.49 KB
#6508 RunWafRealisticBenchmark net6.0 176μs 95.3ns 369ns 0 0 0 2.44 KB
#6508 RunWafRealisticBenchmark netcoreapp3.1 188μs 225ns 871ns 0 0 0 2.39 KB
#6508 RunWafRealisticBenchmark net472 201μs 112ns 436ns 0.301 0 0 2.46 KB
#6508 RunWafRealisticBenchmarkWithAttack net6.0 118μs 115ns 447ns 0 0 0 1.47 KB
#6508 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 124μs 98.3ns 381ns 0 0 0 1.46 KB
#6508 RunWafRealisticBenchmarkWithAttack net472 133μs 56.7ns 220ns 0.229 0 0 1.49 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6508

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 276.42 KB 278.53 KB 2.11 KB 0.76%

Fewer allocations 🎉 in #6508

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 61.73 KB 59.85 KB -1.88 KB -3.05%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 262.87 KB 253 KB -9.87 KB -3.76%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 60μs 755ns 7.55μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 53.2μs 243ns 1.09μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.5μs 111ns 416ns 0 0 0 61.73 KB
master StringConcatAspectBenchmark net6.0 321μs 1.8μs 11.7μs 0 0 0 255.18 KB
master StringConcatAspectBenchmark netcoreapp3.1 340μs 1.87μs 11.4μs 0 0 0 262.87 KB
master StringConcatAspectBenchmark net472 269μs 5.16μs 49.2μs 0 0 0 276.42 KB
#6508 StringConcatBenchmark net6.0 61.7μs 733ns 7.22μs 0 0 0 43.44 KB
#6508 StringConcatBenchmark netcoreapp3.1 53.6μs 268ns 1.1μs 0 0 0 42.64 KB
#6508 StringConcatBenchmark net472 37.7μs 74ns 277ns 0 0 0 59.85 KB
#6508 StringConcatAspectBenchmark net6.0 309μs 1.5μs 8.63μs 0 0 0 254.02 KB
#6508 StringConcatAspectBenchmark netcoreapp3.1 348μs 1.42μs 7.08μs 0 0 0 253 KB
#6508 StringConcatAspectBenchmark net472 298μs 6.66μs 64.9μs 0 0 0 278.53 KB

@andrewlock
Copy link
Member

andrewlock commented Jan 7, 2025 •
edited
Loading

Benchmarks Report for tracer 🐌

Benchmarks for #6508 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.136
  • 1 benchmarks are slower, with geometric mean 1.117
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.83μs 42.8ns 239ns 0.0158 0.00789 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10μs 56ns 354ns 0.0147 0 0 5.81 KB
master StartStopWithChild net472 16.4μs 44.2ns 171ns 1.03 0.307 0.097 6.2 KB
#6508 StartStopWithChild net6.0 8.21μs 46.9ns 342ns 0.0204 0.00815 0 5.61 KB
#6508 StartStopWithChild netcoreapp3.1 10.1μs 50.2ns 213ns 0.0238 0.00951 0 5.8 KB
#6508 StartStopWithChild net472 16.4μs 78.7ns 305ns 1.03 0.303 0.0956 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 481μs 398ns 1.49μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 639μs 686ns 2.66μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 856μs 380ns 1.37μs 0.425 0 0 3.3 KB
#6508 WriteAndFlushEnrichedTraces net6.0 482μs 393ns 1.47μs 0 0 0 2.7 KB
#6508 WriteAndFlushEnrichedTraces netcoreapp3.1 654μs 402ns 1.51μs 0 0 0 2.7 KB
#6508 WriteAndFlushEnrichedTraces net472 849μs 870ns 3.37μs 0.422 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 151μs 852ns 7.13μs 0.145 0 0 14.47 KB
master SendRequest netcoreapp3.1 172μs 980ns 7.4μs 0.17 0 0 17.27 KB
master SendRequest net472 0.0398ns 0.00814ns 0.0315ns 0 0 0 0 b
#6508 SendRequest net6.0 151μs 866ns 7.35μs 0.142 0 0 14.47 KB
#6508 SendRequest netcoreapp3.1 172μs 951ns 7.84μs 0.174 0 0 17.27 KB
#6508 SendRequest net472 0.00352ns 0.00141ns 0.00545ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 547μs 2.09μs 7.54μs 0.556 0 0 41.65 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 687μs 3.99μs 34.5μs 0.332 0 0 41.69 KB
master WriteAndFlushEnrichedTraces net472 885μs 2.2μs 7.63μs 8.13 2.57 0.428 53.31 KB
#6508 WriteAndFlushEnrichedTraces net6.0 564μs 3.04μs 16.9μs 0.536 0 0 41.6 KB
#6508 WriteAndFlushEnrichedTraces netcoreapp3.1 676μs 3.81μs 26.2μs 0.321 0 0 41.77 KB
#6508 WriteAndFlushEnrichedTraces net472 870μs 3.38μs 13.1μs 8.56 2.57 0.428 53.27 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.27μs 0.677ns 2.53ns 0.0139 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.8μs 1.47ns 5.69ns 0.0135 0 0 1.02 KB
master ExecuteNonQuery net472 2.04μs 2.41ns 9.34ns 0.156 0.00102 0 987 B
#6508 ExecuteNonQuery net6.0 1.33μs 1.75ns 6.57ns 0.014 0 0 1.02 KB
#6508 ExecuteNonQuery netcoreapp3.1 1.74μs 1.07ns 4.02ns 0.0139 0 0 1.02 KB
#6508 ExecuteNonQuery net472 2.12μs 3.05ns 11.8ns 0.156 0.00105 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6508

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.117 1,185.68 1,324.17

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.19μs 0.339ns 1.17ns 0.0137 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.69μs 3.35ns 12.5ns 0.0134 0 0 976 B
master CallElasticsearch net472 2.54μs 2.27ns 8.79ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.32μs 1.39ns 5.38ns 0.0134 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.63μs 1.08ns 3.9ns 0.0138 0 0 1.02 KB
master CallElasticsearchAsync net472 2.64μs 2.03ns 7.87ns 0.167 0 0 1.05 KB
#6508 CallElasticsearch net6.0 1.32μs 0.48ns 1.73ns 0.0139 0 0 976 B
#6508 CallElasticsearch netcoreapp3.1 1.54μs 0.551ns 1.99ns 0.0129 0 0 976 B
#6508 CallElasticsearch net472 2.56μs 1.76ns 6.82ns 0.158 0 0 995 B
#6508 CallElasticsearchAsync net6.0 1.39μs 1.92ns 7.18ns 0.0132 0 0 952 B
#6508 CallElasticsearchAsync netcoreapp3.1 1.63μs 0.626ns 2.34ns 0.0138 0 0 1.02 KB
#6508 CallElasticsearchAsync net472 2.55μs 2.26ns 8.47ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.35μs 0.648ns 2.42ns 0.0136 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.67μs 0.646ns 2.42ns 0.0126 0 0 952 B
master ExecuteAsync net472 1.84μs 0.642ns 2.48ns 0.145 0 0 915 B
#6508 ExecuteAsync net6.0 1.38μs 0.627ns 2.34ns 0.0131 0 0 952 B
#6508 ExecuteAsync netcoreapp3.1 1.59μs 0.772ns 2.89ns 0.0127 0 0 952 B
#6508 ExecuteAsync net472 1.75μs 0.497ns 1.92ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.31μs 1.98ns 7.4ns 0.0323 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.44μs 2.08ns 8.07ns 0.0382 0 0 2.85 KB
master SendAsync net472 7.38μs 1.63ns 6.32ns 0.494 0 0 3.12 KB
#6508 SendAsync net6.0 4.4μs 2.76ns 10.3ns 0.0316 0 0 2.31 KB
#6508 SendAsync netcoreapp3.1 5.35μs 2.67ns 10.3ns 0.0376 0 0 2.85 KB
#6508 SendAsync net472 7.45μs 2.09ns 8.1ns 0.495 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6508

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1 1.115 2,362.06 2,118.24

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.53μs 1.07ns 3.99ns 0.0232 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.36μs 1.16ns 4.35ns 0.0225 0 0 1.64 KB
master EnrichedLog net472 2.68μs 1.87ns 7.01ns 0.25 0 0 1.57 KB
#6508 EnrichedLog net6.0 1.52μs 0.688ns 2.57ns 0.0227 0 0 1.64 KB
#6508 EnrichedLog netcoreapp3.1 2.12μs 1.33ns 4.98ns 0.0222 0 0 1.64 KB
#6508 EnrichedLog net472 2.66μs 1.32ns 5.1ns 0.25 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 117μs 312ns 1.21μs 0 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 122μs 86.8ns 313ns 0 0 0 4.28 KB
master EnrichedLog net472 150μs 148ns 574ns 0.675 0.225 0 4.46 KB
#6508 EnrichedLog net6.0 116μs 197ns 764ns 0.0585 0 0 4.28 KB
#6508 EnrichedLog netcoreapp3.1 121μs 144ns 558ns 0.0598 0 0 4.28 KB
#6508 EnrichedLog net472 150μs 122ns 473ns 0.678 0.226 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.04μs 0.665ns 2.49ns 0.0304 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.38μs 1.88ns 7.27ns 0.0284 0 0 2.2 KB
master EnrichedLog net472 4.95μs 1.27ns 4.91ns 0.319 0 0 2.02 KB
#6508 EnrichedLog net6.0 3.16μs 0.858ns 3.32ns 0.0301 0 0 2.2 KB
#6508 EnrichedLog netcoreapp3.1 4.2μs 1.57ns 6.09ns 0.0294 0 0 2.2 KB
#6508 EnrichedLog net472 4.85μs 0.948ns 3.67ns 0.321 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.39μs 0.803ns 3.11ns 0.016 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.76μs 2.26ns 8.75ns 0.0157 0 0 1.14 KB
master SendReceive net472 2.11μs 1.55ns 5.99ns 0.183 0 0 1.16 KB
#6508 SendReceive net6.0 1.41μs 0.855ns 3.31ns 0.0156 0 0 1.14 KB
#6508 SendReceive netcoreapp3.1 1.71μs 0.669ns 2.5ns 0.0147 0 0 1.14 KB
#6508 SendReceive net472 2.09μs 0.827ns 3.2ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.67μs 0.903ns 3.5ns 0.0214 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.98μs 1.21ns 4.37ns 0.0219 0 0 1.65 KB
master EnrichedLog net472 4.44μs 3.37ns 13ns 0.322 0 0 2.04 KB
#6508 EnrichedLog net6.0 2.77μs 0.837ns 3.24ns 0.0221 0 0 1.6 KB
#6508 EnrichedLog netcoreapp3.1 3.84μs 2.26ns 8.77ns 0.0212 0 0 1.65 KB
#6508 EnrichedLog net472 4.47μs 2.45ns 9.48ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6508

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.157 551.55 476.65

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 397ns 0.636ns 2.46ns 0.00798 0 0 576 B
master StartFinishSpan netcoreapp3.1 546ns 0.881ns 3.41ns 0.00786 0 0 576 B
master StartFinishSpan net472 731ns 2.25ns 8.7ns 0.0917 0 0 578 B
master StartFinishScope net6.0 551ns 1.01ns 3.9ns 0.00975 0 0 696 B
master StartFinishScope netcoreapp3.1 797ns 1.39ns 5.39ns 0.0094 0 0 696 B
master StartFinishScope net472 859ns 0.867ns 3.36ns 0.104 0 0 658 B
#6508 StartFinishSpan net6.0 397ns 0.654ns 2.53ns 0.00811 0 0 576 B
#6508 StartFinishSpan netcoreapp3.1 548ns 1.43ns 5.55ns 0.00794 0 0 576 B
#6508 StartFinishSpan net472 696ns 1.46ns 5.64ns 0.0917 0 0 578 B
#6508 StartFinishScope net6.0 476ns 0.744ns 2.68ns 0.00988 0 0 696 B
#6508 StartFinishScope netcoreapp3.1 777ns 1.12ns 4.33ns 0.00951 0 0 696 B
#6508 StartFinishScope net472 855ns 1.8ns 6.96ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 644ns 1.2ns 4.66ns 0.00959 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 985ns 3.19ns 12.4ns 0.00922 0 0 696 B
master RunOnMethodBegin net472 1.13μs 1.45ns 5.62ns 0.105 0 0 658 B
#6508 RunOnMethodBegin net6.0 654ns 1.18ns 4.58ns 0.00978 0 0 696 B
#6508 RunOnMethodBegin netcoreapp3.1 927ns 1.53ns 5.91ns 0.00919 0 0 696 B
#6508 RunOnMethodBegin net472 1.14μs 1.45ns 5.61ns 0.104 0 0 658 B

@kevingosse kevingosse force-pushed the kevin/randomize_tests_2 branch from 16a32e6 to ea37a8e Compare January 8, 2025 13:12
@kevingosse kevingosse force-pushed the kevin/randomize_tests_2 branch from ea37a8e to aa88b9f Compare January 8, 2025 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kevingosse @andrewlock