Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flaky rabbitmq datastreams test #4457

Merged
merged 3 commits into from
Jul 28, 2023
Merged

Fix flaky rabbitmq datastreams test #4457

merged 3 commits into from
Jul 28, 2023

Conversation

hokitam
Copy link
Contributor

@hokitam hokitam commented Jul 27, 2023 •
edited
Loading

Summary of changes

DataStreamsMonitoringRabbitMQTests were flaky due to the resulting mock StatsPoints not being sorted properly. This PR fixes the flakiness by sorting the points by both hashes and timestampType.

Reason for change

Fix tests flakiness.

Implementation details

Test coverage

Other details

@github-actions
Copy link
Contributor

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

7 occurrences of :

-    TimestampType: origin
+    TimestampType: current

7 occurrences of :

-    TimestampType: current
+    TimestampType: origin

@github-actions github-actions bot added the area:tests unit tests, integration tests label Jul 27, 2023
@hokitam hokitam marked this pull request as ready for review July 27, 2023 16:54
@hokitam hokitam requested a review from a team as a code owner July 27, 2023 16:54
@andrewlock
Copy link
Member

andrewlock commented Jul 27, 2023 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4457) - mean (3,000ms)  : 2919, 3081
     .   : milestone, 3000,
    master - mean (3,006ms)  : 2931, 3081
     .   : milestone, 3006,

    section CallTarget+Inlining+NGEN
    This PR (4457) - mean (3,828ms)  : 3749, 3908
     .   : milestone, 3828,
    master - mean (3,815ms)  : 3729, 3902
     .   : milestone, 3815,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4457) - mean (3,124ms)  : 3011, 3237
     .   : milestone, 3124,
    master - mean (3,118ms)  : 3012, 3225
     .   : milestone, 3118,

    section CallTarget+Inlining+NGEN
    This PR (4457) - mean (3,599ms)  : 3542, 3656
     .   : milestone, 3599,
    master - mean (3,623ms)  : 3547, 3699
     .   : milestone, 3623,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4457) - mean (3,111ms)  : 3019, 3204
     .   : milestone, 3111,
    master - mean (3,108ms)  : 3060, 3156
     .   : milestone, 3108,

    section CallTarget+Inlining+NGEN
    This PR (4457) - mean (3,577ms)  : 3533, 3622
     .   : milestone, 3577,
    master - mean (3,574ms)  : 3515, 3633
     .   : milestone, 3574,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4457) - mean (193ms)  : 187, 198
     .   : milestone, 193,
    master - mean (190ms)  : 188, 193
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (4457) - mean (1,127ms)  : 1091, 1163
     .   : milestone, 1127,
    master - mean (1,120ms)  : 1097, 1143
     .   : milestone, 1120,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4457) - mean (374ms)  : 367, 380
     .   : milestone, 374,
    master - mean (372ms)  : 366, 379
     .   : milestone, 372,

    section CallTarget+Inlining+NGEN
    This PR (4457) - mean (1,183ms)  : 1161, 1206
     .   : milestone, 1183,
    master - mean (1,166ms)  : 1136, 1196
     .   : milestone, 1166,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4457) - mean (365ms)  : 359, 371
     .   : milestone, 365,
    master - mean (361ms)  : 356, 366
     .   : milestone, 361,

    section CallTarget+Inlining+NGEN
    This PR (4457) - mean (1,140ms)  : 1113, 1168
     .   : milestone, 1140,
    master - mean (1,127ms)  : 1097, 1158
     .   : milestone, 1127,
Loading

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jul 27, 2023 •
edited
Loading

Datadog Report

Branch report: hokitam/rabbit_flaky
Commit report: 41eee5f

❄️ dd-trace-dotnet: 0 Failed, 1 New Flaky, 294341 Passed, 1091 Skipped, 28m 2.07s Wall Time

New Flaky Tests (1)

  • TelemetryControllerAddsAllAssembliesToCollector - Datadog.Trace.Tests.Telemetry.TelemetryControllerV2Tests - Last Failure

    Expand for error 
     Collection was modified; enumeration operation may not execute.

@andrewlock
Copy link
Member

andrewlock commented Jul 27, 2023 •
edited
Loading

Benchmarks Report 🐌

Benchmarks for #4457 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.187
  • 1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 457μs 101ns 376ns 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 625μs 271ns 1.05μs 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces net472 804μs 307ns 1.19μs 0.398 0 0 3.22 KB
#4457 WriteAndFlushEnrichedTraces net6.0 475μs 140ns 542ns 0 0 0 2.62 KB
#4457 WriteAndFlushEnrichedTraces netcoreapp3.1 636μs 125ns 469ns 0 0 0 2.63 KB
#4457 WriteAndFlushEnrichedTraces net472 788μs 458ns 1.77μs 0.393 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4457

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net472 1.207 147.35 177.85

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 38.9μs 70ns 262ns 0.0197 0 0 1.65 KB
master AllCycleSimpleBody netcoreapp3.1 40.8μs 141ns 545ns 0.02 0 0 1.63 KB
master AllCycleSimpleBody net472 42μs 18ns 67.4ns 0.256 0 0 1.69 KB
master AllCycleMoreComplexBody net6.0 221μs 79.9ns 299ns 0.111 0 0 9.22 KB
master AllCycleMoreComplexBody netcoreapp3.1 229μs 82.6ns 309ns 0.116 0 0 9.12 KB
master AllCycleMoreComplexBody net472 236μs 91.8ns 356ns 1.42 0 0 9.28 KB
master ObjectExtractorSimpleBody net6.0 118ns 0.0425ns 0.159ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 199ns 0.049ns 0.177ns 0.00363 0 0 272 B
master ObjectExtractorSimpleBody net472 147ns 0.116ns 0.449ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.01μs 0.545ns 2.04ns 0.0543 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.08μs 1.39ns 5.19ns 0.0511 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.73μs 8.33ns 32.3ns 0.618 0.00689 0 3.89 KB
#4457 AllCycleSimpleBody net6.0 39.3μs 87.3ns 338ns 0.0193 0 0 1.65 KB
#4457 AllCycleSimpleBody netcoreapp3.1 41μs 68.6ns 266ns 0.0205 0 0 1.63 KB
#4457 AllCycleSimpleBody net472 41.5μs 147ns 570ns 0.266 0 0 1.69 KB
#4457 AllCycleMoreComplexBody net6.0 223μs 663ns 2.57μs 0.111 0 0 9.22 KB
#4457 AllCycleMoreComplexBody netcoreapp3.1 231μs 310ns 1.2μs 0 0 0 9.12 KB
#4457 AllCycleMoreComplexBody net472 237μs 64ns 248ns 1.42 0 0 9.28 KB
#4457 ObjectExtractorSimpleBody net6.0 128ns 0.051ns 0.184ns 0.00395 0 0 280 B
#4457 ObjectExtractorSimpleBody netcoreapp3.1 180ns 0.0676ns 0.262ns 0.00362 0 0 272 B
#4457 ObjectExtractorSimpleBody net472 177ns 1.03ns 8.54ns 0.0446 0 0 281 B
#4457 ObjectExtractorMoreComplexBody net6.0 3.03μs 1.07ns 3.86ns 0.0545 0 0 3.88 KB
#4457 ObjectExtractorMoreComplexBody netcoreapp3.1 4.08μs 1.64ns 6.36ns 0.051 0 0 3.78 KB
#4457 ObjectExtractorMoreComplexBody net472 4.77μs 23.4ns 96.5ns 0.618 0.00711 0 3.89 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 169μs 119ns 446ns 0.169 0 0 18.08 KB
master SendRequest netcoreapp3.1 191μs 304ns 1.18μs 0.189 0 0 20.14 KB
master SendRequest net472 0.000889ns 0.000277ns 0.00107ns 0 0 0 0 b
#4457 SendRequest net6.0 170μs 99.3ns 385ns 0.17 0 0 18.08 KB
#4457 SendRequest netcoreapp3.1 188μs 212ns 794ns 0.187 0 0 20.14 KB
#4457 SendRequest net472 0.000777ns 0.000333ns 0.00129ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #4457

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.96 KB 41.45 KB -512 B -1.22%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 523μs 2.28μs 8.83μs 0.502 0 0 41.96 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 614μs 1.46μs 5.48μs 0.302 0 0 41.82 KB
master WriteAndFlushEnrichedTraces net472 794μs 3.26μs 12.6μs 8.06 2.42 0.403 53.22 KB
#4457 WriteAndFlushEnrichedTraces net6.0 536μs 1.24μs 4.79μs 0.506 0 0 41.45 KB
#4457 WriteAndFlushEnrichedTraces netcoreapp3.1 615μs 1.93μs 7.47μs 0.613 0 0 41.93 KB
#4457 WriteAndFlushEnrichedTraces net472 779μs 3.31μs 12.8μs 8.38 2.67 0.381 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 930ns 0.473ns 1.77ns 0.0107 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.22μs 0.227ns 0.851ns 0.0103 0 0 768 B
master ExecuteNonQuery net472 1.43μs 0.64ns 2.48ns 0.116 0.000719 0 730 B
#4457 ExecuteNonQuery net6.0 847ns 0.342ns 1.33ns 0.0107 0 0 768 B
#4457 ExecuteNonQuery netcoreapp3.1 1.24μs 0.63ns 2.44ns 0.0101 0 0 768 B
#4457 ExecuteNonQuery net472 1.47μs 0.416ns 1.56ns 0.116 0.000737 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.02μs 0.412ns 1.54ns 0.0138 0 0 992 B
master CallElasticsearch netcoreapp3.1 1.31μs 0.445ns 1.67ns 0.0132 0 0 992 B
master CallElasticsearch net472 2.12μs 0.781ns 3.02ns 0.159 0.00106 0 1 KB
master CallElasticsearchAsync net6.0 1.13μs 0.507ns 1.96ns 0.0134 0 0 968 B
master CallElasticsearchAsync netcoreapp3.1 1.44μs 0.683ns 2.64ns 0.0136 0 0 1.04 KB
master CallElasticsearchAsync net472 2.23μs 0.459ns 1.78ns 0.167 0.00112 0 1.06 KB
#4457 CallElasticsearch net6.0 1.07μs 0.294ns 1.06ns 0.014 0 0 992 B
#4457 CallElasticsearch netcoreapp3.1 1.25μs 0.481ns 1.8ns 0.0134 0 0 992 B
#4457 CallElasticsearch net472 2.11μs 0.691ns 2.59ns 0.158 0.00106 0 1 KB
#4457 CallElasticsearchAsync net6.0 1.13μs 0.475ns 1.84ns 0.0136 0 0 968 B
#4457 CallElasticsearchAsync netcoreapp3.1 1.35μs 0.534ns 1.92ns 0.0142 0 0 1.04 KB
#4457 CallElasticsearchAsync net472 2.19μs 0.747ns 2.89ns 0.167 0.00109 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.09μs 0.402ns 1.56ns 0.0125 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.34μs 0.408ns 1.41ns 0.0121 0 0 912 B
master ExecuteAsync net472 1.54μs 0.367ns 1.37ns 0.138 0.000769 0 875 B
#4457 ExecuteAsync net6.0 1.09μs 0.287ns 1.07ns 0.013 0 0 912 B
#4457 ExecuteAsync netcoreapp3.1 1.39μs 0.387ns 1.39ns 0.0125 0 0 912 B
#4457 ExecuteAsync net472 1.54μs 0.663ns 2.57ns 0.138 0.000769 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.57μs 0.903ns 3.5ns 0.0267 0 0 1.94 KB
master SendAsync netcoreapp3.1 4.42μs 1.11ns 4.16ns 0.033 0 0 2.48 KB
master SendAsync net472 6.93μs 10.7ns 41.5ns 0.482 0 0 3.05 KB
#4457 SendAsync net6.0 3.6μs 0.921ns 3.45ns 0.027 0 0 1.94 KB
#4457 SendAsync netcoreapp3.1 4.25μs 3.86ns 14.4ns 0.0336 0 0 2.48 KB
#4457 SendAsync net472 6.84μs 2.37ns 8.85ns 0.483 0 0 3.05 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.34μs 0.477ns 1.79ns 0.0228 0 0 1.62 KB
master EnrichedLog netcoreapp3.1 1.87μs 2.43ns 9.41ns 0.0223 0 0 1.62 KB
master EnrichedLog net472 2.22μs 1.09ns 4.21ns 0.244 0 0 1.54 KB
#4457 EnrichedLog net6.0 1.23μs 0.854ns 3.31ns 0.0228 0 0 1.62 KB
#4457 EnrichedLog netcoreapp3.1 1.79μs 0.576ns 2.15ns 0.0224 0 0 1.62 KB
#4457 EnrichedLog net472 2.25μs 2.96ns 11.5ns 0.244 0 0 1.54 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 170ns 636ns 0.056 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 117μs 188ns 729ns 0 0 0 4.21 KB
master EnrichedLog net472 145μs 71.5ns 268ns 0.651 0.217 0 4.38 KB
#4457 EnrichedLog net6.0 110μs 113ns 437ns 0.0574 0 0 4.21 KB
#4457 EnrichedLog netcoreapp3.1 115μs 126ns 486ns 0 0 0 4.21 KB
#4457 EnrichedLog net472 146μs 102ns 396ns 0.659 0.22 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.83μs 0.804ns 2.9ns 0.0298 0 0 2.18 KB
master EnrichedLog netcoreapp3.1 3.85μs 1.79ns 6.44ns 0.0291 0 0 2.18 KB
master EnrichedLog net472 4.66μs 3.52ns 13.6ns 0.316 0 0 1.99 KB
#4457 EnrichedLog net6.0 2.82μs 1.37ns 5.12ns 0.0299 0 0 2.18 KB
#4457 EnrichedLog netcoreapp3.1 3.74μs 1.44ns 5.37ns 0.0281 0 0 2.18 KB
#4457 EnrichedLog net472 4.69μs 2.21ns 8.29ns 0.316 0 0 1.99 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.18μs 0.4ns 1.44ns 0.0159 0 0 1.16 KB
master SendReceive netcoreapp3.1 1.49μs 0.818ns 3.17ns 0.0157 0 0 1.16 KB
master SendReceive net472 1.77μs 1.43ns 5.17ns 0.185 0.000883 0 1.16 KB
#4457 SendReceive net6.0 1.2μs 0.366ns 1.32ns 0.0161 0 0 1.16 KB
#4457 SendReceive netcoreapp3.1 1.45μs 1.78ns 6.89ns 0.0155 0 0 1.16 KB
#4457 SendReceive net472 1.82μs 1.06ns 4.09ns 0.184 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.5μs 0.537ns 2.08ns 0.0214 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.56μs 2.2ns 8.25ns 0.0213 0 0 1.58 KB
master EnrichedLog net472 3.93μs 0.947ns 3.54ns 0.31 0 0 1.96 KB
#4457 EnrichedLog net6.0 2.49μs 0.754ns 2.82ns 0.0209 0 0 1.53 KB
#4457 EnrichedLog netcoreapp3.1 3.49μs 1.32ns 4.93ns 0.0212 0 0 1.58 KB
#4457 EnrichedLog net472 3.94μs 1.05ns 4.08ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4457

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.168 619.79 723.71

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 397ns 0.172ns 0.644ns 0.00745 0 0 536 B
master StartFinishSpan netcoreapp3.1 502ns 0.258ns 0.966ns 0.00728 0 0 536 B
master StartFinishSpan net472 625ns 0.22ns 0.822ns 0.0852 0 0 538 B
master StartFinishScope net6.0 528ns 0.18ns 0.623ns 0.00918 0 0 656 B
master StartFinishScope netcoreapp3.1 620ns 0.152ns 0.527ns 0.00904 0 0 656 B
master StartFinishScope net472 784ns 0.3ns 1.16ns 0.0979 0 0 618 B
#4457 StartFinishSpan net6.0 427ns 0.202ns 0.756ns 0.00747 0 0 536 B
#4457 StartFinishSpan netcoreapp3.1 519ns 0.157ns 0.586ns 0.00728 0 0 536 B
#4457 StartFinishSpan net472 599ns 0.4ns 1.55ns 0.0853 0 0 538 B
#4457 StartFinishScope net6.0 508ns 0.15ns 0.581ns 0.00908 0 0 656 B
#4457 StartFinishScope netcoreapp3.1 724ns 0.168ns 0.628ns 0.00867 0 0 656 B
#4457 StartFinishScope net472 770ns 0.181ns 0.676ns 0.0979 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 646ns 0.201ns 0.753ns 0.00901 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 702ns 0.198ns 0.741ns 0.00881 0 0 656 B
master RunOnMethodBegin net472 926ns 0.434ns 1.68ns 0.0979 0 0 618 B
#4457 RunOnMethodBegin net6.0 590ns 0.195ns 0.754ns 0.00909 0 0 656 B
#4457 RunOnMethodBegin netcoreapp3.1 759ns 0.241ns 0.903ns 0.00891 0 0 656 B
#4457 RunOnMethodBegin net472 942ns 0.16ns 0.577ns 0.0978 0 0 618 B

@andrewlock
Copy link
Member

andrewlock commented Jul 27, 2023 •
edited
Loading

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4457) (10.971M)   : 0, 10971384
    master (10.877M)   : 0, 10877040
    benchmarks/2.34.0 (11.172M)   : 0, 11172386
    benchmarks/2.9.0 (11.226M)   : 0, 11225900

    section Automatic
    This PR (4457) (7.808M)   : 0, 7808255
    master (7.669M)   : 0, 7669221
    benchmarks/2.34.0 (7.859M)   : 0, 7858718
    benchmarks/2.9.0 (8.128M)   : 0, 8127670

    section Trace stats
    master (7.809M)   : 0, 7808796
    benchmarks/2.34.0 (7.905M)   : 0, 7904802

    section Manual
    This PR (4457) (9.872M)   : 0, 9872224
    master (9.823M)   : 0, 9822723
    benchmarks/2.34.0 (9.986M)   : 0, 9986137

    section Manual + Automatic
    This PR (4457) (7.458M)   : 0, 7458284
    master (7.406M)   : 0, 7405833
    benchmarks/2.34.0 (7.570M)   : 0, 7570487

    section Version Conflict
    master (6.770M)   : 0, 6769692
    benchmarks/2.34.0 (6.749M)   : 0, 6749094
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4457) (9.697M)   : 0, 9697311
    master (9.574M)   : 0, 9574337
    benchmarks/2.34.0 (9.689M)   : 0, 9689253
    benchmarks/2.9.0 (9.659M)   : 0, 9658728

    section Automatic
    This PR (4457) (6.780M)   : 0, 6779928
    master (6.813M)   : 0, 6812599
    benchmarks/2.34.0 (6.670M)   : 0, 6670396

    section Trace stats
    master (6.818M)   : 0, 6817725
    benchmarks/2.34.0 (6.743M)   : 0, 6743225

    section Manual
    This PR (4457) (8.465M)   : 0, 8465040
    master (8.437M)   : 0, 8437210
    benchmarks/2.34.0 (8.614M)   : 0, 8614349

    section Manual + Automatic
    This PR (4457) (6.312M)   : 0, 6311757
    master (6.566M)   : 0, 6566202
    benchmarks/2.34.0 (6.532M)   : 0, 6532340

    section Version Conflict
    master (5.892M)   : 0, 5892214
    benchmarks/2.34.0 (5.854M)   : 0, 5853936
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4457) (10.203M)   : 0, 10203006
    master (10.238M)   : 0, 10238267
    benchmarks/2.34.0 (10.187M)   : 0, 10187020
    benchmarks/2.9.0 (10.240M)   : 0, 10239951

    section Automatic
    This PR (4457) (7.492M)   : 0, 7491813
    master (7.591M)   : 0, 7591432
    benchmarks/2.34.0 (7.402M)   : 0, 7401872
    benchmarks/2.9.0 (7.435M)   : 0, 7435116

    section Trace stats
    master (7.326M)   : 0, 7326243
    benchmarks/2.34.0 (7.426M)   : 0, 7426314

    section Manual
    This PR (4457) (9.457M)   : 0, 9456585
    master (9.424M)   : 0, 9423734
    benchmarks/2.34.0 (9.173M)   : 0, 9173438

    section Manual + Automatic
    This PR (4457) (7.144M)   : 0, 7143738
    master (7.437M)   : 0, 7437421
    benchmarks/2.34.0 (7.218M)   : 0, 7218183

    section Version Conflict
    master (6.561M)   : 0, 6561453
    benchmarks/2.34.0 (6.597M)   : 0, 6597265
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4457) (7.344M)   : 0, 7344191
    master (7.353M)   : 0, 7353218
    benchmarks/2.34.0 (7.612M)   : 0, 7611651
    benchmarks/2.9.0 (7.840M)   : 0, 7839825

    section No attack
    This PR (4457) (2.132M)   : 0, 2131534
    master (2.138M)   : 0, 2137647
    benchmarks/2.34.0 (2.156M)   : 0, 2156076
    benchmarks/2.9.0 (3.181M)   : 0, 3180750

    section Attack
    This PR (4457) (1.806M)   : 0, 1806277
    master (1.816M)   : 0, 1816342
    benchmarks/2.9.0 (2.474M)   : 0, 2474287

    section Blocking
    This PR (4457) (3.692M)   : 0, 3692405
    master (3.656M)   : 0, 3655676
Loading

andrewlock
andrewlock approved these changes Jul 28, 2023
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@pierotibou pierotibou merged commit f43d98a into master Jul 28, 2023
@pierotibou pierotibou deleted the hokitam/rabbit_flaky branch July 28, 2023 08:49
@github-actions github-actions bot added this to the vNext milestone Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pierotibou pierotibou pierotibou approved these changes

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:data-streams-monitoring area:tests unit tests, integration tests
Projects
None yet
Milestone
2.35.0
Development

Successfully merging this pull request may close these issues.
3 participants
@hokitam @andrewlock @pierotibou