Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CIVisibility] ConfigureCiCommand fixes. #4558

Merged
merged 12 commits into from
Aug 29, 2023
Merged

[CIVisibility] ConfigureCiCommand fixes. #4558

merged 12 commits into from
Aug 29, 2023

Conversation

tonyredondo
Copy link
Member

@tonyredondo tonyredondo commented Aug 28, 2023 •
edited
Loading

Summary of changes

This PR fixes some issues detected when using dd-trace ci configure and also makes some small changes in the runner.

Changes:

  • Clean clrprofiler paths to absolute paths
  • Replace AnsiConsole with Console when settings the global environment variables for azure ci using logs commands.
  • Adds a mechanism to try to reduce the profiler filepath lengths due to a limitation in the azure logs commands length.

Reason for change

A customer tried to use dd-trace ci configure command and is failing, I was able to reproduce the issue in a sample repo.

@tonyredondo tonyredondo self-assigned this Aug 28, 2023
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Aug 28, 2023 •
edited
Loading

Datadog Report

Branch report: tony/civisibility-runner-absolute-paths
Commit report: eb51dd2

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 299656 Passed, 1115 Skipped, 22m 11.86s Wall Time

@tonyredondo tonyredondo changed the title [CIVisibility] Clean clrprofiler paths to absolute paths [CIVisibility] Fixes on the ConfigureCiCommand Aug 29, 2023
@tonyredondo tonyredondo changed the title [CIVisibility] Fixes on the ConfigureCiCommand [CIVisibility] ConfigureCiCommand fixes. Aug 29, 2023
@DataDog DataDog deleted a comment from andrewlock Aug 29, 2023
@DataDog DataDog deleted a comment from andrewlock Aug 29, 2023
@DataDog DataDog deleted a comment from andrewlock Aug 29, 2023
@andrewlock
Copy link
Member

andrewlock commented Aug 29, 2023 •
edited
Loading

Benchmarks Report 🐌

Benchmarks for #4558 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.203
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.15μs 44.8ns 272ns 0.0238 0.0119 0 7.3 KB
master StartStopWithChild netcoreapp3.1 10.3μs 54.2ns 266ns 0.0353 0.0151 0 7.4 KB
master StartStopWithChild net472 15.6μs 62.4ns 242ns 1.29 0.327 0.0989 7.67 KB
#4558 StartStopWithChild net6.0 8.24μs 46ns 302ns 0.0208 0.00833 0 7.3 KB
#4558 StartStopWithChild netcoreapp3.1 10.1μs 53.3ns 266ns 0.025 0.01 0 7.4 KB
#4558 StartStopWithChild net472 15.6μs 62.8ns 243ns 1.28 0.319 0.101 7.67 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 469μs 326ns 1.26μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 648μs 128ns 479ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 813μs 224ns 807ns 0.411 0 0 3.3 KB
#4558 WriteAndFlushEnrichedTraces net6.0 464μs 171ns 640ns 0 0 0 2.7 KB
#4558 WriteAndFlushEnrichedTraces netcoreapp3.1 641μs 258ns 964ns 0 0 0 2.7 KB
#4558 WriteAndFlushEnrichedTraces net472 801μs 394ns 1.47μs 0.396 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4558

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.185 200.51 169.26

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 39μs 12.7ns 49.2ns 0.0197 0 0 1.69 KB
master AllCycleSimpleBody netcoreapp3.1 41μs 44.4ns 160ns 0.0205 0 0 1.67 KB
master AllCycleSimpleBody net472 42.9μs 17ns 63.5ns 0.258 0 0 1.73 KB
master AllCycleMoreComplexBody net6.0 229μs 185ns 868ns 0.115 0 0 9.26 KB
master AllCycleMoreComplexBody netcoreapp3.1 239μs 139ns 537ns 0.119 0 0 9.16 KB
master AllCycleMoreComplexBody net472 249μs 286ns 1.07μs 1.38 0 0 9.33 KB
master ObjectExtractorSimpleBody net6.0 121ns 0.0436ns 0.163ns 0.00391 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 201ns 0.122ns 0.472ns 0.00374 0 0 272 B
master ObjectExtractorSimpleBody net472 149ns 0.236ns 0.914ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.02μs 1.31ns 4.9ns 0.0539 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.06μs 1.03ns 3.86ns 0.0508 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.13μs 2.79ns 10.4ns 0.618 0.00618 0 3.89 KB
#4558 AllCycleSimpleBody net6.0 39.4μs 11.6ns 43.4ns 0.0196 0 0 1.69 KB
#4558 AllCycleSimpleBody netcoreapp3.1 41.5μs 73.6ns 285ns 0.0208 0 0 1.67 KB
#4558 AllCycleSimpleBody net472 43.6μs 14.8ns 57.2ns 0.256 0 0 1.73 KB
#4558 AllCycleMoreComplexBody net6.0 229μs 64.6ns 242ns 0.114 0 0 9.26 KB
#4558 AllCycleMoreComplexBody netcoreapp3.1 240μs 683ns 2.46μs 0.12 0 0 9.16 KB
#4558 AllCycleMoreComplexBody net472 247μs 76.3ns 296ns 1.48 0 0 9.33 KB
#4558 ObjectExtractorSimpleBody net6.0 123ns 0.069ns 0.267ns 0.00392 0 0 280 B
#4558 ObjectExtractorSimpleBody netcoreapp3.1 170ns 0.218ns 0.844ns 0.00376 0 0 272 B
#4558 ObjectExtractorSimpleBody net472 149ns 0.111ns 0.417ns 0.0446 0 0 281 B
#4558 ObjectExtractorMoreComplexBody net6.0 2.99μs 0.759ns 2.94ns 0.0543 0 0 3.88 KB
#4558 ObjectExtractorMoreComplexBody netcoreapp3.1 4.07μs 1.28ns 4.79ns 0.0508 0 0 3.78 KB
#4558 ObjectExtractorMoreComplexBody net472 4.14μs 2.61ns 9.42ns 0.617 0.00619 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 124μs 1.36μs 13.2μs 0 0 0 17.62 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 133μs 745ns 6.75μs 0 0 0 16.79 KB
master RunWaf(args=NestedMap (10)) net472 129μs 681ns 3.41μs 0 0 0 24.58 KB
master RunWaf(args=NestedMap (100)) net6.0 242μs 1.6μs 15.5μs 0 0 0 37.95 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 258μs 1.34μs 6.28μs 0 0 0 37.78 KB
master RunWaf(args=NestedMap (100)) net472 257μs 1.22μs 5.02μs 0 0 0 40.96 KB
master RunWaf(args=NestedMap (1000)) net6.0 235μs 1.3μs 7.57μs 0 0 0 37.95 KB
master RunWaf(args=NestedMap (1000)) netcoreapp3.1 258μs 1.39μs 7.5μs 0 0 0 37.78 KB
master RunWaf(args=NestedMap (1000)) net472 258μs 1.32μs 6.45μs 0 0 0 40.96 KB
#4558 RunWaf(args=NestedMap (10)) net6.0 126μs 804ns 7.33μs 0 0 0 17.62 KB
#4558 RunWaf(args=NestedMap (10)) netcoreapp3.1 132μs 703ns 4.97μs 0 0 0 16.79 KB
#4558 RunWaf(args=NestedMap (10)) net472 132μs 682ns 3.13μs 0 0 0 24.58 KB
#4558 RunWaf(args=NestedMap (100)) net6.0 235μs 1.37μs 11.6μs 0 0 0 37.95 KB
#4558 RunWaf(args=NestedMap (100)) netcoreapp3.1 264μs 1.4μs 7.52μs 0 0 0 37.78 KB
#4558 RunWaf(args=NestedMap (100)) net472 259μs 1.41μs 8.23μs 0 0 0 40.96 KB
#4558 RunWaf(args=NestedMap (1000)) net6.0 232μs 1.25μs 6.87μs 0 0 0 37.95 KB
#4558 RunWaf(args=NestedMap (1000)) netcoreapp3.1 268μs 1.46μs 8.77μs 0 0 0 37.78 KB
#4558 RunWaf(args=NestedMap (1000)) net472 260μs 1.33μs 6.53μs 0 0 0 40.96 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 171μs 166ns 643ns 0.255 0 0 18.09 KB
master SendRequest netcoreapp3.1 191μs 366ns 1.42μs 0.189 0 0 20.25 KB
master SendRequest net472 0.000167ns 0.000149ns 0.000558ns 0 0 0 0 b
#4558 SendRequest net6.0 171μs 190ns 737ns 0.171 0 0 18.09 KB
#4558 SendRequest netcoreapp3.1 192μs 199ns 771ns 0.192 0 0 20.25 KB
#4558 SendRequest net472 3.22E‑05ns 2.37E‑05ns 8.86E‑05ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 527μs 786ns 3.04μs 0.53 0 0 41.84 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 617μs 1.6μs 6.2μs 0.306 0 0 41.57 KB
master WriteAndFlushEnrichedTraces net472 790μs 3.42μs 13.3μs 8.33 2.65 0.379 53.24 KB
#4558 WriteAndFlushEnrichedTraces net6.0 520μs 977ns 3.38μs 0.523 0 0 41.85 KB
#4558 WriteAndFlushEnrichedTraces netcoreapp3.1 624μs 1.47μs 5.68μs 0.316 0 0 41.7 KB
#4558 WriteAndFlushEnrichedTraces net472 787μs 2.52μs 9.77μs 8.1 2.31 0.386 53.23 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.08μs 0.387ns 1.39ns 0.0106 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.45μs 0.806ns 2.79ns 0.0102 0 0 768 B
master ExecuteNonQuery net472 1.74μs 4.37ns 16.9ns 0.115 0 0 730 B
#4558 ExecuteNonQuery net6.0 1.05μs 0.569ns 2.21ns 0.011 0 0 768 B
#4558 ExecuteNonQuery netcoreapp3.1 1.37μs 0.631ns 2.28ns 0.0102 0 0 768 B
#4558 ExecuteNonQuery net472 1.66μs 0.606ns 2.35ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.14μs 0.793ns 2.75ns 0.0143 0 0 992 B
master CallElasticsearch netcoreapp3.1 1.4μs 0.331ns 1.19ns 0.0134 0 0 992 B
master CallElasticsearch net472 2.53μs 0.546ns 2.04ns 0.158 0.00127 0 1 KB
master CallElasticsearchAsync net6.0 1.29μs 0.797ns 2.98ns 0.0134 0 0 968 B
master CallElasticsearchAsync netcoreapp3.1 1.68μs 0.759ns 2.74ns 0.0135 0 0 1.04 KB
master CallElasticsearchAsync net472 2.51μs 0.905ns 3.51ns 0.168 0.00125 0 1.06 KB
#4558 CallElasticsearch net6.0 1.19μs 0.382ns 1.48ns 0.0137 0 0 992 B
#4558 CallElasticsearch netcoreapp3.1 1.46μs 0.534ns 2ns 0.0133 0 0 992 B
#4558 CallElasticsearch net472 2.4μs 0.54ns 2.09ns 0.159 0 0 1 KB
#4558 CallElasticsearchAsync net6.0 1.33μs 0.491ns 1.84ns 0.0134 0 0 968 B
#4558 CallElasticsearchAsync netcoreapp3.1 1.62μs 0.817ns 3.16ns 0.0138 0 0 1.04 KB
#4558 CallElasticsearchAsync net472 2.58μs 0.591ns 2.29ns 0.168 0.00129 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.16μs 0.73ns 2.73ns 0.0127 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.4μs 0.956ns 3.58ns 0.012 0 0 912 B
master ExecuteAsync net472 1.65μs 0.612ns 2.37ns 0.138 0.000823 0 875 B
#4558 ExecuteAsync net6.0 1.23μs 0.769ns 2.98ns 0.0129 0 0 912 B
#4558 ExecuteAsync netcoreapp3.1 1.44μs 1.1ns 4.11ns 0.012 0 0 912 B
#4558 ExecuteAsync net472 1.68μs 0.485ns 1.75ns 0.139 0.000835 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.86μs 0.902ns 3.38ns 0.027 0 0 1.94 KB
master SendAsync netcoreapp3.1 4.46μs 1.77ns 6.61ns 0.0333 0 0 2.48 KB
master SendAsync net472 7.09μs 4.37ns 16.4ns 0.482 0 0 3.05 KB
#4558 SendAsync net6.0 3.92μs 1.69ns 6.53ns 0.0275 0 0 1.94 KB
#4558 SendAsync netcoreapp3.1 4.5μs 1.9ns 7.1ns 0.0337 0 0 2.48 KB
#4558 SendAsync net472 7.17μs 2.85ns 11ns 0.483 0 0 3.05 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.3μs 0.647ns 2.42ns 0.0229 0 0 1.62 KB
master EnrichedLog netcoreapp3.1 1.78μs 0.784ns 3.04ns 0.0223 0 0 1.62 KB
master EnrichedLog net472 2.22μs 0.941ns 3.26ns 0.245 0 0 1.54 KB
#4558 EnrichedLog net6.0 1.35μs 0.639ns 2.39ns 0.0223 0 0 1.62 KB
#4558 EnrichedLog netcoreapp3.1 1.88μs 4.26ns 16ns 0.0216 0 0 1.62 KB
#4558 EnrichedLog net472 2.39μs 1.86ns 6.98ns 0.245 0 0 1.54 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 112μs 108ns 420ns 0.0555 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 119μs 175ns 676ns 0.0587 0 0 4.21 KB
master EnrichedLog net472 146μs 114ns 440ns 0.657 0.219 0 4.38 KB
#4558 EnrichedLog net6.0 116μs 221ns 855ns 0.0572 0 0 4.21 KB
#4558 EnrichedLog netcoreapp3.1 119μs 221ns 856ns 0.0595 0 0 4.21 KB
#4558 EnrichedLog net472 148μs 243ns 942ns 0.666 0.222 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.94μs 0.607ns 2.19ns 0.0294 0 0 2.18 KB
master EnrichedLog netcoreapp3.1 3.86μs 1.66ns 6.22ns 0.0299 0 0 2.18 KB
master EnrichedLog net472 4.49μs 2.56ns 9.92ns 0.315 0 0 1.99 KB
#4558 EnrichedLog net6.0 3.14μs 1.81ns 7.02ns 0.0309 0 0 2.18 KB
#4558 EnrichedLog netcoreapp3.1 3.78μs 1.22ns 4.39ns 0.0285 0 0 2.18 KB
#4558 EnrichedLog net472 4.55μs 1.7ns 6.57ns 0.316 0 0 1.99 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.32μs 0.496ns 1.86ns 0.0159 0 0 1.16 KB
master SendReceive netcoreapp3.1 1.57μs 0.636ns 2.29ns 0.0156 0 0 1.16 KB
master SendReceive net472 2.09μs 4.74ns 18.4ns 0.184 0.00105 0 1.16 KB
#4558 SendReceive net6.0 1.29μs 0.629ns 2.18ns 0.0166 0 0 1.16 KB
#4558 SendReceive netcoreapp3.1 1.68μs 2.89ns 11.2ns 0.0161 0 0 1.16 KB
#4558 SendReceive net472 2.13μs 1.73ns 6.69ns 0.184 0.00107 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.7μs 1.72ns 6.44ns 0.0216 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.63μs 0.92ns 3.44ns 0.0199 0 0 1.58 KB
master EnrichedLog net472 4.08μs 1.48ns 5.33ns 0.311 0 0 1.96 KB
#4558 EnrichedLog net6.0 2.55μs 1.29ns 5ns 0.0204 0 0 1.53 KB
#4558 EnrichedLog netcoreapp3.1 3.74μs 1.09ns 4.07ns 0.0206 0 0 1.58 KB
#4558 EnrichedLog net472 3.98μs 1.4ns 5.42ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4558

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.221 577.73 473.31

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 453ns 0.105ns 0.378ns 0.00747 0 0 536 B
master StartFinishSpan netcoreapp3.1 604ns 1.01ns 3.9ns 0.00728 0 0 536 B
master StartFinishSpan net472 646ns 0.114ns 0.394ns 0.0853 0 0 538 B
master StartFinishScope net6.0 578ns 0.0804ns 0.29ns 0.00919 0 0 656 B
master StartFinishScope netcoreapp3.1 758ns 0.224ns 0.838ns 0.00866 0 0 656 B
master StartFinishScope net472 814ns 0.567ns 2.19ns 0.0979 0 0 618 B
#4558 StartFinishSpan net6.0 444ns 0.14ns 0.54ns 0.00759 0 0 536 B
#4558 StartFinishSpan netcoreapp3.1 562ns 0.553ns 2.14ns 0.00729 0 0 536 B
#4558 StartFinishSpan net472 676ns 0.295ns 1.14ns 0.0853 0 0 538 B
#4558 StartFinishScope net6.0 473ns 0.101ns 0.392ns 0.00923 0 0 656 B
#4558 StartFinishScope netcoreapp3.1 695ns 0.527ns 1.9ns 0.00873 0 0 656 B
#4558 StartFinishScope net472 851ns 0.368ns 1.43ns 0.0978 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 606ns 0.171ns 0.639ns 0.00927 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 749ns 0.287ns 0.995ns 0.00887 0 0 656 B
master RunOnMethodBegin net472 977ns 0.347ns 1.25ns 0.0979 0 0 618 B
#4558 RunOnMethodBegin net6.0 576ns 0.205ns 0.792ns 0.00913 0 0 656 B
#4558 RunOnMethodBegin netcoreapp3.1 801ns 0.271ns 1.05ns 0.00874 0 0 656 B
#4558 RunOnMethodBegin net472 995ns 0.344ns 1.33ns 0.0982 0 0 618 B

@andrewlock
Copy link
Member

andrewlock commented Aug 29, 2023 •
edited
Loading

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4558) (11.462M)   : 0, 11461688
    master (11.261M)   : 0, 11261050
    benchmarks/2.36.0 (11.382M)   : 0, 11382084
    benchmarks/2.9.0 (11.174M)   : 0, 11173638

    section Automatic
    This PR (4558) (7.831M)   : 0, 7831293
    master (7.924M)   : 0, 7923952
    benchmarks/2.36.0 (7.895M)   : 0, 7895036
    benchmarks/2.9.0 (8.231M)   : 0, 8230767

    section Trace stats
    master (8.007M)   : 0, 8006784
    benchmarks/2.36.0 (7.934M)   : 0, 7934498

    section Manual
    This PR (4558) (10.000M)   : 0, 9999858
    master (10.123M)   : 0, 10123186
    benchmarks/2.36.0 (10.135M)   : 0, 10135365

    section Manual + Automatic
    This PR (4558) (7.621M)   : 0, 7620973
    master (7.691M)   : 0, 7690615
    benchmarks/2.36.0 (7.687M)   : 0, 7686860

    section Version Conflict
    master (6.882M)   : 0, 6882307
    benchmarks/2.36.0 (6.950M)   : 0, 6950075
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4558) (9.653M)   : 0, 9652818
    master (9.754M)   : 0, 9754253
    benchmarks/2.36.0 (9.679M)   : 0, 9678578
    benchmarks/2.9.0 (9.598M)   : 0, 9597680

    section Automatic
    This PR (4558) (6.662M)   : 0, 6662055
    master (6.880M)   : 0, 6879538
    benchmarks/2.36.0 (6.765M)   : 0, 6764548

    section Trace stats
    master (6.752M)   : 0, 6752232
    benchmarks/2.36.0 (6.684M)   : 0, 6684198

    section Manual
    This PR (4558) (8.640M)   : 0, 8640191
    master (8.470M)   : 0, 8470393
    benchmarks/2.36.0 (8.556M)   : 0, 8555932

    section Manual + Automatic
    This PR (4558) (6.516M)   : 0, 6516093
    master (6.503M)   : 0, 6503023
    benchmarks/2.36.0 (6.546M)   : 0, 6545995

    section Version Conflict
    master (6.042M)   : 0, 6041579
    benchmarks/2.36.0 (5.717M)   : 0, 5717018
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4558) (10.378M)   : 0, 10378026
    master (9.973M)   : 0, 9972625
    benchmarks/2.36.0 (10.614M)   : 0, 10613686
    benchmarks/2.9.0 (10.703M)   : 0, 10703397

    section Automatic
    This PR (4558) (7.299M)   : 0, 7298693
    master (6.883M)   : 0, 6883014
    benchmarks/2.36.0 (7.670M)   : 0, 7670031
    benchmarks/2.9.0 (7.805M)   : 0, 7805318

    section Trace stats
    master (6.983M)   : 0, 6982500
    benchmarks/2.36.0 (7.600M)   : 0, 7600232

    section Manual
    This PR (4558) (9.131M)   : 0, 9131497
    master (8.998M)   : 0, 8997600
    benchmarks/2.36.0 (9.431M)   : 0, 9431272

    section Manual + Automatic
    This PR (4558) (7.258M)   : 0, 7258074
    master (6.851M)   : 0, 6851483
    benchmarks/2.36.0 (7.296M)   : 0, 7296323

    section Version Conflict
    master (6.171M)   : 0, 6170565
    benchmarks/2.36.0 (6.914M)   : 0, 6914150
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4558) (7.344M)   : 0, 7343926
    master (7.355M)   : 0, 7355078
    benchmarks/2.36.0 (7.399M)   : 0, 7398988
    benchmarks/2.9.0 (7.797M)   : 0, 7797019

    section No attack
    This PR (4558) (2.145M)   : 0, 2145449
    master (2.131M)   : 0, 2131433
    benchmarks/2.36.0 (2.133M)   : 0, 2133428
    benchmarks/2.9.0 (3.234M)   : 0, 3234355

    section Attack
    This PR (4558) (1.676M)   : 0, 1675866
    master (1.645M)   : 0, 1645396
    benchmarks/2.36.0 (1.660M)   : 0, 1660412
    benchmarks/2.9.0 (2.471M)   : 0, 2471012

    section Blocking
    This PR (4558) (3.040M)   : 0, 3039914
    master (3.146M)   : 0, 3145804
    benchmarks/2.36.0 (3.090M)   : 0, 3090340
Loading

@tonyredondo tonyredondo marked this pull request as ready for review August 29, 2023 16:29
@tonyredondo tonyredondo requested a review from a team as a code owner August 29, 2023 16:29
@andrewlock
Copy link
Member

andrewlock commented Aug 29, 2023 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4558) - mean (76ms)  : 60, 91
     .   : milestone, 76,
    master - mean (71ms)  : 61, 81
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4558) - mean (1,039ms)  : 1000, 1079
     .   : milestone, 1039,
    master - mean (1,034ms)  : 1004, 1065
     .   : milestone, 1034,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4558) - mean (105ms)  : 103, 106
     .   : milestone, 105,
    master - mean (106ms)  : 100, 112
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4558) - mean (724ms)  : 690, 759
     .   : milestone, 724,
    master - mean (734ms)  : 696, 772
     .   : milestone, 734,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4558) - mean (90ms)  : 85, 94
     .   : milestone, 90,
    master - mean (89ms)  : 84, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4558) - mean (686ms)  : 660, 712
     .   : milestone, 686,
    master - mean (689ms)  : 655, 722
     .   : milestone, 689,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4558) - mean (191ms)  : 185, 197
     .   : milestone, 191,
    master - mean (194ms)  : 190, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (4558) - mean (1,129ms)  : 1097, 1161
     .   : milestone, 1129,
    master - mean (1,135ms)  : 1097, 1172
     .   : milestone, 1135,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4558) - mean (272ms)  : 264, 279
     .   : milestone, 272,
    master - mean (274ms)  : 268, 281
     .   : milestone, 274,

    section CallTarget+Inlining+NGEN
    This PR (4558) - mean (1,092ms)  : 1057, 1127
     .   : milestone, 1092,
    master - mean (1,101ms)  : 1062, 1141
     .   : milestone, 1101,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4558) - mean (265ms)  : 258, 272
     .   : milestone, 265,
    master - mean (262ms)  : 253, 271
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (4558) - mean (1,049ms)  : 1007, 1090
     .   : milestone, 1049,
    master - mean (1,062ms)  : 1028, 1097
     .   : milestone, 1062,
Loading

robertpi
robertpi approved these changes Aug 29, 2023
View reviewed changes
Copy link
Member

@robertpi robertpi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does what is says on the tin.

@tonyredondo tonyredondo merged commit 78fdfe5 into master Aug 29, 2023
@tonyredondo tonyredondo deleted the tony/civisibility-runner-absolute-paths branch August 29, 2023 18:56
@github-actions github-actions bot added this to the vNext milestone Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lucaspimentel lucaspimentel lucaspimentel approved these changes

@robertpi robertpi robertpi approved these changes

Assignees

@tonyredondo tonyredondo

Labels
area:ci-visibility
Projects
None yet
Milestone
2.37.0
Development

Successfully merging this pull request may close these issues.
4 participants
@tonyredondo @andrewlock @robertpi @lucaspimentel