Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the Rel Env docker-image workflow #4448

Merged
merged 3 commits into from
Jul 28, 2023
Merged

Conversation

andrewlock
Copy link
Member

@andrewlock andrewlock commented Jul 26, 2023

Summary of changes

  • Remove the creating of the rel env release images out of the create_draft_release workflow
  • Create a new workflow to create the images after the release is published
  • Extract a GitHub action that removes a bit of the complexity from the workflows
  • Rename is_snapshot to is_release_version to make it more explicit about what's happening

Reason for change

The existing trigger (silently) failed in the create_draft_release workflow. The implications of this weren't clear at the time, partly as the naming wasn't clear that it was related to the rel-env (and therefor low impact for the release).

By moving to a separate workflow it moves it out of the critical path.

Implementation details

  • Extracted a GitHub action to avoid triggering a workflow from inside an automated GH Action (because you can't do that without extra PAT etc)
  • Created an "on release" trigger
  • Added some extra comments and renaming to make it ore obvious what things are

Test coverage

  • Ran a test of the "manual" workflow (which passed), which also tests the new action. But it's not easy to test the "release" workflow.

TODO

  • Update the release documentation to add a step checking the output of the workflow after the release is published, and if it fails, running the "manual" version

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jul 26, 2023
@andrewlock andrewlock requested a review from a team as a code owner July 26, 2023 12:28
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jul 26, 2023

Datadog Report

Branch report: andrew/ci/fix-rel-env-images
Commit report: 712e40e

❄️ dd-trace-dotnet: 0 Failed, 2 New Flaky, 294341 Passed, 1091 Skipped, 29m 30.25s Wall Time

New Flaky Tests (2)

  • SubmitDbmCommentedSpanspropagationDisabled - Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.NpgsqlDatabaseMonitoringTests - Last Failure

    Expand for error
     Expected exit code: 0, actual exit code: 139.
    
  • HandleProduceAndConsume - Datadog.Trace.ClrProfiler.IntegrationTests.DataStreamsMonitoringRabbitMQTests - Last Failure

    Expand for error
     Results do not match.
     Differences:
     Received: DataStreamsMonitoringRabbitMQTests.HandleProduceAndConsume.received.txt
     Verified: DataStreamsMonitoringRabbitMQTests.HandleProduceAndConsume.verified.txt
     Received Content:
     [
       {
         EdgeTags: [
           direction:in,
           topic:TopicQueue2,
     ...
    

@andrewlock
Copy link
Member Author

andrewlock commented Jul 26, 2023

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4448) - mean (3,011ms)  : 2914, 3107
     .   : milestone, 3011,
    master - mean (3,014ms)  : 2932, 3095
     .   : milestone, 3014,

    section CallTarget+Inlining+NGEN
    This PR (4448) - mean (3,826ms)  : 3752, 3899
     .   : milestone, 3826,
    master - mean (3,820ms)  : 3729, 3911
     .   : milestone, 3820,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4448) - mean (3,117ms)  : 2978, 3256
     .   : milestone, 3117,
    master - mean (3,141ms)  : 3046, 3236
     .   : milestone, 3141,

    section CallTarget+Inlining+NGEN
    This PR (4448) - mean (3,604ms)  : 3562, 3645
     .   : milestone, 3604,
    master - mean (3,602ms)  : 3527, 3678
     .   : milestone, 3602,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4448) - mean (3,096ms)  : 2980, 3212
     .   : milestone, 3096,
    master - mean (3,118ms)  : 3007, 3229
     .   : milestone, 3118,

    section CallTarget+Inlining+NGEN
    This PR (4448) - mean (3,581ms)  : 3524, 3637
     .   : milestone, 3581,
    master - mean (3,587ms)  : 3505, 3669
     .   : milestone, 3587,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4448) - mean (194ms)  : 191, 196
     .   : milestone, 194,
    master - mean (191ms)  : 185, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (4448) - mean (1,125ms)  : 1093, 1158
     .   : milestone, 1125,
    master - mean (1,115ms)  : 1087, 1143
     .   : milestone, 1115,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4448) - mean (374ms)  : 366, 381
     .   : milestone, 374,
    master - mean (373ms)  : 367, 379
     .   : milestone, 373,

    section CallTarget+Inlining+NGEN
    This PR (4448) - mean (1,180ms)  : 1146, 1214
     .   : milestone, 1180,
    master - mean (1,179ms)  : 1152, 1207
     .   : milestone, 1179,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4448) - mean (361ms)  : 357, 366
     .   : milestone, 361,
    master - mean (361ms)  : 355, 367
     .   : milestone, 361,

    section CallTarget+Inlining+NGEN
    This PR (4448) - mean (1,135ms)  : 1109, 1161
     .   : milestone, 1135,
    master - mean (1,135ms)  : 1108, 1162
     .   : milestone, 1135,

@andrewlock
Copy link
Member Author

andrewlock commented Jul 26, 2023

Benchmarks Report 🐌

Benchmarks for #4448 compared to master:

  • 7 benchmarks are faster, with geometric mean 1.149
  • 1 benchmarks are slower, with geometric mean 1.294
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 485μs 193ns 748ns 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 641μs 218ns 845ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 813μs 173ns 625ns 0.403 0 0 3.22 KB
#4448 WriteAndFlushEnrichedTraces net6.0 474μs 302ns 1.17μs 0 0 0 2.62 KB
#4448 WriteAndFlushEnrichedTraces netcoreapp3.1 640μs 427ns 1.65μs 0 0 0 2.62 KB
#4448 WriteAndFlushEnrichedTraces net472 798μs 472ns 1.83μs 0.396 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4448

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net472 1.294 147.08 190.36

Faster 🎉 in #4448

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.178 202.94 172.25
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472 1.162 4,729.36 4,070.73

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 39.4μs 67.6ns 262ns 0.0195 0 0 1.65 KB
master AllCycleSimpleBody netcoreapp3.1 40.9μs 104ns 387ns 0.0204 0 0 1.63 KB
master AllCycleSimpleBody net472 42.3μs 203ns 788ns 0.253 0 0 1.69 KB
master AllCycleMoreComplexBody net6.0 222μs 77ns 298ns 0.112 0 0 9.22 KB
master AllCycleMoreComplexBody netcoreapp3.1 229μs 262ns 1.01μs 0.115 0 0 9.12 KB
master AllCycleMoreComplexBody net472 239μs 53.9ns 202ns 1.43 0 0 9.28 KB
master ObjectExtractorSimpleBody net6.0 120ns 0.0281ns 0.105ns 0.00395 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 203ns 0.0702ns 0.263ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 147ns 0.0525ns 0.197ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.02μs 1.13ns 4.23ns 0.0542 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.09μs 1.33ns 5.17ns 0.0512 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.72μs 15.9ns 61.5ns 0.618 0.00705 0 3.89 KB
#4448 AllCycleSimpleBody net6.0 38.7μs 16.7ns 64.7ns 0.0195 0 0 1.65 KB
#4448 AllCycleSimpleBody netcoreapp3.1 41.5μs 101ns 391ns 0.0209 0 0 1.63 KB
#4448 AllCycleSimpleBody net472 42.4μs 59.8ns 231ns 0.267 0 0 1.69 KB
#4448 AllCycleMoreComplexBody net6.0 220μs 58.4ns 219ns 0.11 0 0 9.22 KB
#4448 AllCycleMoreComplexBody netcoreapp3.1 230μs 160ns 599ns 0.117 0 0 9.12 KB
#4448 AllCycleMoreComplexBody net472 238μs 65.8ns 246ns 1.42 0 0 9.28 KB
#4448 ObjectExtractorSimpleBody net6.0 120ns 0.0373ns 0.139ns 0.00397 0 0 280 B
#4448 ObjectExtractorSimpleBody netcoreapp3.1 172ns 0.133ns 0.496ns 0.00374 0 0 272 B
#4448 ObjectExtractorSimpleBody net472 186ns 1.06ns 7.71ns 0.0446 0 0 281 B
#4448 ObjectExtractorMoreComplexBody net6.0 3.04μs 3.59ns 12.9ns 0.0543 0 0 3.88 KB
#4448 ObjectExtractorMoreComplexBody netcoreapp3.1 4.06μs 1.36ns 5.27ns 0.0509 0 0 3.78 KB
#4448 ObjectExtractorMoreComplexBody net472 4.07μs 1.42ns 5.51ns 0.618 0.00612 0 3.89 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 168μs 74.4ns 279ns 0.253 0 0 18.08 KB
master SendRequest netcoreapp3.1 188μs 360ns 1.39μs 0.189 0 0 20.14 KB
master SendRequest net472 0.000356ns 0.000172ns 0.000642ns 0 0 0 0 b
#4448 SendRequest net6.0 171μs 117ns 454ns 0.171 0 0 18.08 KB
#4448 SendRequest netcoreapp3.1 186μs 255ns 988ns 0.275 0 0 20.14 KB
#4448 SendRequest net472 0.000198ns 7.9E‑05ns 0.000285ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #4448

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.46 KB 41.76 KB 301 B 0.73%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.65 KB 41.95 KB 301 B 0.72%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 519μs 334ns 1.29μs 0.527 0 0 41.46 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 635μs 1.19μs 4.62μs 0.317 0 0 41.65 KB
master WriteAndFlushEnrichedTraces net472 773μs 3.13μs 12.1μs 8.15 2.33 0.388 53.22 KB
#4448 WriteAndFlushEnrichedTraces net6.0 517μs 1.57μs 5.66μs 0.504 0 0 41.76 KB
#4448 WriteAndFlushEnrichedTraces netcoreapp3.1 631μs 1.07μs 4.16μs 0.326 0 0 41.95 KB
#4448 WriteAndFlushEnrichedTraces net472 777μs 3.21μs 12μs 8.44 2.68 0.383 53.23 KB
Benchmarks.Trace.DbCommandBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4448

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0 1.137 1,015.74 893.54

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.02μs 0.385ns 1.44ns 0.0105 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.22μs 1.69ns 6.33ns 0.0104 0 0 768 B
master ExecuteNonQuery net472 1.38μs 0.333ns 1.2ns 0.116 0 0 730 B
#4448 ExecuteNonQuery net6.0 894ns 0.287ns 1.11ns 0.0107 0 0 768 B
#4448 ExecuteNonQuery netcoreapp3.1 1.23μs 0.492ns 1.9ns 0.0106 0 0 768 B
#4448 ExecuteNonQuery net472 1.45μs 0.509ns 1.9ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.15μs 8.75ns 81.1ns 0.0139 0 0 992 B
master CallElasticsearch netcoreapp3.1 1.28μs 0.278ns 1.04ns 0.0135 0 0 992 B
master CallElasticsearch net472 2.1μs 1.26ns 4.56ns 0.158 0.00106 0 1 KB
master CallElasticsearchAsync net6.0 1.2μs 0.294ns 1.1ns 0.0133 0 0 968 B
master CallElasticsearchAsync netcoreapp3.1 1.4μs 0.558ns 2.01ns 0.014 0 0 1.04 KB
master CallElasticsearchAsync net472 2.28μs 0.652ns 2.44ns 0.167 0.00114 0 1.06 KB
#4448 CallElasticsearch net6.0 1.13μs 0.449ns 1.74ns 0.0141 0 0 992 B
#4448 CallElasticsearch netcoreapp3.1 1.25μs 0.734ns 2.75ns 0.0131 0 0 992 B
#4448 CallElasticsearch net472 2.09μs 0.643ns 2.49ns 0.159 0.00105 0 1 KB
#4448 CallElasticsearchAsync net6.0 1.1μs 0.555ns 2.08ns 0.0138 0 0 968 B
#4448 CallElasticsearchAsync netcoreapp3.1 1.43μs 0.855ns 3.2ns 0.0137 0 0 1.04 KB
#4448 CallElasticsearchAsync net472 2.09μs 0.755ns 2.92ns 0.168 0.00104 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.06μs 1.2ns 4.49ns 0.0126 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.41μs 0.457ns 1.65ns 0.0127 0 0 912 B
master ExecuteAsync net472 1.55μs 0.565ns 2.19ns 0.139 0.000775 0 875 B
#4448 ExecuteAsync net6.0 1.17μs 1.21ns 4.19ns 0.013 0 0 912 B
#4448 ExecuteAsync netcoreapp3.1 1.45μs 2.61ns 9.76ns 0.0122 0 0 912 B
#4448 ExecuteAsync net472 1.6μs 0.392ns 1.41ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.53μs 1.01ns 3.8ns 0.0266 0 0 1.94 KB
master SendAsync netcoreapp3.1 4.29μs 11.8ns 45.8ns 0.034 0 0 2.48 KB
master SendAsync net472 6.99μs 10.9ns 42.2ns 0.481 0 0 3.05 KB
#4448 SendAsync net6.0 3.73μs 7.94ns 29.7ns 0.0275 0 0 1.94 KB
#4448 SendAsync netcoreapp3.1 4.29μs 2.06ns 7.72ns 0.0323 0 0 2.48 KB
#4448 SendAsync net472 6.82μs 1.84ns 6.88ns 0.484 0 0 3.05 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.23μs 0.521ns 1.95ns 0.0229 0 0 1.62 KB
master EnrichedLog netcoreapp3.1 1.83μs 0.641ns 2.48ns 0.0219 0 0 1.62 KB
master EnrichedLog net472 2.28μs 1.56ns 6.02ns 0.245 0 0 1.54 KB
#4448 EnrichedLog net6.0 1.33μs 0.542ns 2.03ns 0.0226 0 0 1.62 KB
#4448 EnrichedLog netcoreapp3.1 1.85μs 0.907ns 3.39ns 0.0221 0 0 1.62 KB
#4448 EnrichedLog net472 2.26μs 1.6ns 6.19ns 0.244 0 0 1.54 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 87ns 326ns 0.057 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 182ns 704ns 0.0584 0 0 4.21 KB
master EnrichedLog net472 146μs 113ns 423ns 0.655 0.218 0 4.38 KB
#4448 EnrichedLog net6.0 112μs 168ns 650ns 0.056 0 0 4.21 KB
#4448 EnrichedLog netcoreapp3.1 117μs 113ns 409ns 0 0 0 4.21 KB
#4448 EnrichedLog net472 153μs 750ns 3.09μs 0.666 0.222 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.88μs 0.917ns 3.55ns 0.0302 0 0 2.18 KB
master EnrichedLog netcoreapp3.1 3.82μs 1.19ns 4.47ns 0.0287 0 0 2.18 KB
master EnrichedLog net472 4.4μs 3.13ns 12.1ns 0.315 0 0 1.99 KB
#4448 EnrichedLog net6.0 2.9μs 12.9ns 46.6ns 0.03 0 0 2.18 KB
#4448 EnrichedLog netcoreapp3.1 3.79μs 1.37ns 4.76ns 0.0287 0 0 2.18 KB
#4448 EnrichedLog net472 4.69μs 3.27ns 12.2ns 0.315 0 0 1.99 KB
Benchmarks.Trace.RedisBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4448

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0 1.118 1,250.97 1,119.26

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.25μs 0.559ns 2.09ns 0.0163 0 0 1.16 KB
master SendReceive netcoreapp3.1 1.51μs 0.679ns 2.54ns 0.0151 0 0 1.16 KB
master SendReceive net472 1.73μs 0.951ns 3.3ns 0.184 0 0 1.16 KB
#4448 SendReceive net6.0 1.12μs 1.98ns 8.62ns 0.0163 0 0 1.16 KB
#4448 SendReceive netcoreapp3.1 1.5μs 1.03ns 3.84ns 0.0157 0 0 1.16 KB
#4448 SendReceive net472 1.83μs 1.6ns 5.99ns 0.185 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.47μs 0.665ns 2.58ns 0.0209 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.47μs 1.4ns 5.44ns 0.021 0 0 1.58 KB
master EnrichedLog net472 4.04μs 1.73ns 6.7ns 0.311 0 0 1.96 KB
#4448 EnrichedLog net6.0 2.69μs 0.824ns 3.19ns 0.0215 0 0 1.53 KB
#4448 EnrichedLog netcoreapp3.1 3.57μs 1.23ns 4.75ns 0.0216 0 0 1.58 KB
#4448 EnrichedLog net472 4.02μs 1.17ns 4.54ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4448

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.167 457.23 391.64

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 457ns 0.192ns 0.719ns 0.00759 0 0 536 B
master StartFinishSpan netcoreapp3.1 568ns 0.464ns 1.8ns 0.00718 0 0 536 B
master StartFinishSpan net472 576ns 0.142ns 0.551ns 0.0852 0 0 538 B
master StartFinishScope net6.0 459ns 0.119ns 0.446ns 0.00915 0 0 656 B
master StartFinishScope netcoreapp3.1 653ns 0.116ns 0.417ns 0.00884 0 0 656 B
master StartFinishScope net472 787ns 1.62ns 6.06ns 0.0979 0 0 618 B
#4448 StartFinishSpan net6.0 392ns 0.162ns 0.628ns 0.00758 0 0 536 B
#4448 StartFinishSpan netcoreapp3.1 574ns 0.183ns 0.659ns 0.0072 0 0 536 B
#4448 StartFinishSpan net472 556ns 0.174ns 0.672ns 0.0852 0 0 538 B
#4448 StartFinishScope net6.0 450ns 0.18ns 0.674ns 0.00907 0 0 656 B
#4448 StartFinishScope netcoreapp3.1 651ns 0.587ns 2.2ns 0.00885 0 0 656 B
#4448 StartFinishScope net472 746ns 0.309ns 1.2ns 0.0981 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4448

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.151 601.40 522.65
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 1.130 781.18 691.06

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 602ns 0.208ns 0.749ns 0.00937 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 781ns 0.372ns 1.39ns 0.00862 0 0 656 B
master RunOnMethodBegin net472 925ns 0.179ns 0.694ns 0.0979 0 0 618 B
#4448 RunOnMethodBegin net6.0 523ns 0.154ns 0.575ns 0.00922 0 0 656 B
#4448 RunOnMethodBegin netcoreapp3.1 691ns 0.279ns 0.968ns 0.0087 0 0 656 B
#4448 RunOnMethodBegin net472 971ns 0.156ns 0.563ns 0.0979 0 0 618 B

@andrewlock andrewlock force-pushed the andrew/ci/fix-rel-env-images branch from f1c6c30 to 087c0af Compare July 26, 2023 13:23
@andrewlock
Copy link
Member Author

andrewlock commented Jul 26, 2023

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

Loading
gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4448) (11.133M)   : 0, 11132835
    master (11.149M)   : 0, 11149466
    benchmarks/2.34.0 (11.172M)   : 0, 11172386
    benchmarks/2.9.0 (11.226M)   : 0, 11225900

    section Automatic
    This PR (4448) (7.811M)   : 0, 7811282
    master (7.882M)   : 0, 7882084
    benchmarks/2.34.0 (7.859M)   : 0, 7858718
    benchmarks/2.9.0 (8.128M)   : 0, 8127670

    section Trace stats
    master (7.826M)   : 0, 7826063
    benchmarks/2.34.0 (7.905M)   : 0, 7904802

    section Manual
    This PR (4448) (9.929M)   : 0, 9928894
    master (9.820M)   : 0, 9819660
    benchmarks/2.34.0 (9.986M)   : 0, 9986137

    section Manual + Automatic
    This PR (4448) (7.622M)   : 0, 7621770
    master (7.435M)   : 0, 7435018
    benchmarks/2.34.0 (7.570M)   : 0, 7570487

    section Version Conflict
    master (6.641M)   : 0, 6640824
    benchmarks/2.34.0 (6.749M)   : 0, 6749094

Loading
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4448) (9.665M)   : 0, 9665094
    master (9.617M)   : 0, 9617498
    benchmarks/2.34.0 (9.689M)   : 0, 9689253
    benchmarks/2.9.0 (9.659M)   : 0, 9658728

    section Automatic
    This PR (4448) (6.699M)   : 0, 6699445
    master (6.703M)   : 0, 6702641
    benchmarks/2.34.0 (6.670M)   : 0, 6670396

    section Trace stats
    master (6.666M)   : 0, 6665747
    benchmarks/2.34.0 (6.743M)   : 0, 6743225

    section Manual
    This PR (4448) (8.616M)   : 0, 8616243
    master (8.521M)   : 0, 8520521
    benchmarks/2.34.0 (8.614M)   : 0, 8614349

    section Manual + Automatic
    This PR (4448) (6.667M)   : 0, 6667397
    master (6.551M)   : 0, 6551069
    benchmarks/2.34.0 (6.532M)   : 0, 6532340

    section Version Conflict
    master (5.875M)   : 0, 5875078
    benchmarks/2.34.0 (5.854M)   : 0, 5853936

Loading
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4448) (10.190M)   : 0, 10190247
    master (10.155M)   : 0, 10155240
    benchmarks/2.34.0 (10.187M)   : 0, 10187020
    benchmarks/2.9.0 (10.240M)   : 0, 10239951

    section Automatic
    This PR (4448) (7.639M)   : 0, 7638574
    master (7.588M)   : 0, 7588027
    benchmarks/2.34.0 (7.402M)   : 0, 7401872
    benchmarks/2.9.0 (7.435M)   : 0, 7435116

    section Trace stats
    master (7.344M)   : 0, 7343611
    benchmarks/2.34.0 (7.426M)   : 0, 7426314

    section Manual
    This PR (4448) (9.224M)   : 0, 9224216
    master (9.276M)   : 0, 9275658
    benchmarks/2.34.0 (9.173M)   : 0, 9173438

    section Manual + Automatic
    This PR (4448) (7.019M)   : 0, 7018744
    master (7.213M)   : 0, 7213113
    benchmarks/2.34.0 (7.218M)   : 0, 7218183

    section Version Conflict
    master (6.567M)   : 0, 6566571
    benchmarks/2.34.0 (6.597M)   : 0, 6597265

Loading
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4448) (7.476M)   : 0, 7476175
    master (7.422M)   : 0, 7422015
    benchmarks/2.34.0 (7.612M)   : 0, 7611651
    benchmarks/2.9.0 (7.840M)   : 0, 7839825

    section No attack
    This PR (4448) (2.135M)   : 0, 2134542
    master (2.167M)   : 0, 2167375
    benchmarks/2.34.0 (2.156M)   : 0, 2156076
    benchmarks/2.9.0 (3.181M)   : 0, 3180750

    section Attack
    This PR (4448) (1.828M)   : 0, 1828104
    master (1.868M)   : 0, 1868231
    benchmarks/2.9.0 (2.474M)   : 0, 2474287

    section Blocking
    This PR (4448) (3.716M)   : 0, 3715570
    master (3.849M)   : 0, 3849329

Copy link
Collaborator

@pierotibou pierotibou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one comment on the trigger on the main pipeline.
I assume we should add @robertomonteromiguel as a reviewer as he's the one who introduced this so that he's aware of the changes if future refactos are needed

.azure-pipelines/ultimate-pipeline.yml Show resolved Hide resolved
.azure-pipelines/ultimate-pipeline.yml Show resolved Hide resolved
Base automatically changed from andrew/ci/fix-create-release to master July 27, 2023 12:59
- Extract a GitHub action that removes a bit of the complexity from the workflows
- Rename `is_snapshot` to `is_release_version` to make it more explicit about what's happening
- Remove the creating of the rel env release images out of the create_draft_release workflow
- Create a new workflow to create the images after the release is merged
@andrewlock andrewlock force-pushed the andrew/ci/fix-rel-env-images branch 2 times, most recently from 8511b87 to d917c5e Compare July 27, 2023 14:44
@andrewlock andrewlock force-pushed the andrew/ci/fix-rel-env-images branch from d917c5e to 712e40e Compare July 27, 2023 14:54
@bouwkast bouwkast merged commit e67bf03 into master Jul 28, 2023
@bouwkast bouwkast deleted the andrew/ci/fix-rel-env-images branch July 28, 2023 16:01
@github-actions github-actions bot added this to the vNext milestone Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants