-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Test Package Versions Bump] Updating package versions #6279
Conversation
Datadog ReportBranch report: ✅ 0 Failed, 453020 Passed, 2735 Skipped, 19h 42m 10.88s Total Time |
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6279) - mean (71ms) : 64, 78
. : milestone, 71,
master - mean (71ms) : 65, 78
. : milestone, 71,
section CallTarget+Inlining+NGEN
This PR (6279) - mean (980ms) : 953, 1008
. : milestone, 980,
master - mean (981ms) : 960, 1003
. : milestone, 981,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6279) - mean (108ms) : 106, 111
. : milestone, 108,
master - mean (108ms) : 106, 110
. : milestone, 108,
section CallTarget+Inlining+NGEN
This PR (6279) - mean (680ms) : 667, 694
. : milestone, 680,
master - mean (683ms) : 665, 702
. : milestone, 683,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6279) - mean (92ms) : 89, 95
. : milestone, 92,
master - mean (92ms) : 89, 95
. : milestone, 92,
section CallTarget+Inlining+NGEN
This PR (6279) - mean (636ms) : 618, 653
. : milestone, 636,
master - mean (638ms) : 623, 653
. : milestone, 638,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6279) - mean (190ms) : 185, 196
. : milestone, 190,
master - mean (190ms) : 186, 194
. : milestone, 190,
section CallTarget+Inlining+NGEN
This PR (6279) - mean (1,103ms) : 1069, 1138
. : milestone, 1103,
master - mean (1,094ms) : 1072, 1115
. : milestone, 1094,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6279) - mean (276ms) : 272, 280
. : milestone, 276,
master - mean (276ms) : 272, 279
. : milestone, 276,
section CallTarget+Inlining+NGEN
This PR (6279) - mean (875ms) : 842, 907
. : milestone, 875,
master - mean (875ms) : 850, 900
. : milestone, 875,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6279) - mean (265ms) : 260, 270
. : milestone, 265,
master - mean (264ms) : 260, 268
. : milestone, 264,
section CallTarget+Inlining+NGEN
This PR (6279) - mean (854ms) : 816, 893
. : milestone, 854,
master - mean (854ms) : 825, 884
. : milestone, 854,
|
|
||
<!-- Integration: Microsoft.Extensions.Telemetry --> | ||
<!-- Assembly: Microsoft.Extensions.Telemetry --> | ||
<!-- Latest package https://www.nuget.org/packages/Microsoft.Extensions.Telemetry/8.10.0 --> | ||
<!-- Latest package https://www.nuget.org/packages/Microsoft.Extensions.Telemetry/9.0.0 --> | ||
<PackageReference Include="Microsoft.Extensions.Telemetry" Version="8.10.0" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one probably needs updating
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤔 It looks like we have already updated to support 9.x on that one, just not updated the json matrix I think
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated the matrix in this PR so 🤞
<!-- Latest package https://www.nuget.org/packages/System.Data.SqlClient/4.8.6 --> | ||
<PackageReference Include="System.Data.SqlClient" Version="4.8.6" /> | ||
<!-- Latest package https://www.nuget.org/packages/System.Data.SqlClient/4.9.0 --> | ||
<PackageReference Include="System.Data.SqlClient" Version="4.9.0" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seeing some strange failures here, haven't looked into them
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
18:16:34 [DBG] ---> System.PlatformNotSupportedException: System.Data.SqlClient is not supported on this platform.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like they just killed it for EOL runtimes to me - we'll need to tweak the json to remove it:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done on a commit to this branch
Throughput/Crank Report ⚡Throughput results for AspNetCoreSimpleController comparing the following branches/commits: Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red. Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards! gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6279) (11.266M) : 0, 11265933
master (11.216M) : 0, 11215877
benchmarks/2.9.0 (11.033M) : 0, 11032866
section Automatic
This PR (6279) (7.277M) : 0, 7277060
master (7.356M) : 0, 7355647
benchmarks/2.9.0 (7.786M) : 0, 7785853
section Trace stats
master (7.653M) : 0, 7652612
section Manual
master (11.177M) : 0, 11177132
section Manual + Automatic
This PR (6279) (6.810M) : 0, 6809930
master (6.722M) : 0, 6722247
section DD_TRACE_ENABLED=0
master (10.246M) : 0, 10246375
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6279) (9.481M) : 0, 9480646
master (9.495M) : 0, 9495372
benchmarks/2.9.0 (9.495M) : 0, 9494821
section Automatic
This PR (6279) (6.416M) : 0, 6415946
master (6.445M) : 0, 6445315
section Trace stats
master (6.617M) : 0, 6617151
section Manual
master (9.564M) : 0, 9564169
section Manual + Automatic
This PR (6279) (5.928M) : 0, 5927534
master (5.943M) : 0, 5943103
section DD_TRACE_ENABLED=0
master (8.993M) : 0, 8992941
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6279) (9.118M) : 0, 9117515
master (8.980M) : 0, 8980408
benchmarks/2.9.0 (10.020M) : 0, 10019592
section Automatic
This PR (6279) (5.984M) : 0, 5984075
master (5.865M) : 0, 5865070
benchmarks/2.9.0 (7.255M) : 0, 7255257
section Trace stats
master (6.446M) : 0, 6446474
section Manual
master (9.151M) : 0, 9151083
section Manual + Automatic
This PR (6279) (5.659M) : 0, 5658760
master (5.595M) : 0, 5594798
section DD_TRACE_ENABLED=0
master (8.375M) : 0, 8375216
|
316c2ad
to
9e10789
Compare
Benchmarks Report for tracer 🐌Benchmarks for #6279 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Slower
|
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0 | 1.115 | 1,346.32 | 1,500.60 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | SendReceive |
net6.0 | 1.35μs | 0.38ns | 1.42ns | 0.0162 | 0 | 0 | 1.14 KB |
master | SendReceive |
netcoreapp3.1 | 1.73μs | 0.869ns | 3.25ns | 0.0156 | 0 | 0 | 1.14 KB |
master | SendReceive |
net472 | 2.06μs | 1.19ns | 4.44ns | 0.183 | 0 | 0 | 1.16 KB |
#6279 | SendReceive |
net6.0 | 1.5μs | 0.77ns | 2.98ns | 0.0157 | 0 | 0 | 1.14 KB |
#6279 | SendReceive |
netcoreapp3.1 | 1.69μs | 0.909ns | 3.52ns | 0.0153 | 0 | 0 | 1.14 KB |
#6279 | SendReceive |
net472 | 2.08μs | 1.23ns | 4.45ns | 0.183 | 0 | 0 | 1.16 KB |
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | EnrichedLog |
net6.0 | 2.65μs | 1.63ns | 6.3ns | 0.0226 | 0 | 0 | 1.6 KB |
master | EnrichedLog |
netcoreapp3.1 | 3.96μs | 2.55ns | 9.89ns | 0.022 | 0 | 0 | 1.65 KB |
master | EnrichedLog |
net472 | 4.35μs | 3.38ns | 13.1ns | 0.322 | 0 | 0 | 2.04 KB |
#6279 | EnrichedLog |
net6.0 | 2.68μs | 2.43ns | 9.4ns | 0.0215 | 0 | 0 | 1.6 KB |
#6279 | EnrichedLog |
netcoreapp3.1 | 3.88μs | 11.4ns | 44ns | 0.0214 | 0 | 0 | 1.65 KB |
#6279 | EnrichedLog |
net472 | 4.5μs | 2.62ns | 10.1ns | 0.324 | 0 | 0 | 2.04 KB |
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | StartFinishSpan |
net6.0 | 399ns | 0.725ns | 2.81ns | 0.00807 | 0 | 0 | 576 B |
master | StartFinishSpan |
netcoreapp3.1 | 590ns | 1.42ns | 5.51ns | 0.00775 | 0 | 0 | 576 B |
master | StartFinishSpan |
net472 | 659ns | 0.643ns | 2.32ns | 0.0916 | 0 | 0 | 578 B |
master | StartFinishScope |
net6.0 | 478ns | 0.619ns | 2.4ns | 0.00968 | 0 | 0 | 696 B |
master | StartFinishScope |
netcoreapp3.1 | 705ns | 1.06ns | 4.1ns | 0.0095 | 0 | 0 | 696 B |
master | StartFinishScope |
net472 | 844ns | 1.45ns | 5.62ns | 0.105 | 0 | 0 | 658 B |
#6279 | StartFinishSpan |
net6.0 | 392ns | 0.573ns | 2.22ns | 0.00815 | 0 | 0 | 576 B |
#6279 | StartFinishSpan |
netcoreapp3.1 | 583ns | 0.811ns | 3.03ns | 0.00777 | 0 | 0 | 576 B |
#6279 | StartFinishSpan |
net472 | 720ns | 1.28ns | 4.95ns | 0.0917 | 0 | 0 | 578 B |
#6279 | StartFinishScope |
net6.0 | 496ns | 0.855ns | 3.31ns | 0.00975 | 0 | 0 | 696 B |
#6279 | StartFinishScope |
netcoreapp3.1 | 735ns | 1.18ns | 4.58ns | 0.00909 | 0 | 0 | 696 B |
#6279 | StartFinishScope |
net472 | 883ns | 1.56ns | 6.05ns | 0.104 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️
Slower ⚠️ in #6279
Benchmark
diff/base
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net472
1.126
1,056.41
1,189.55
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net472 | 1.126 | 1,056.41 | 1,189.55 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | RunOnMethodBegin |
net6.0 | 656ns | 0.758ns | 2.94ns | 0.00974 | 0 | 0 | 696 B |
master | RunOnMethodBegin |
netcoreapp3.1 | 903ns | 1.41ns | 5.45ns | 0.00941 | 0 | 0 | 696 B |
master | RunOnMethodBegin |
net472 | 1.06μs | 1.46ns | 5.66ns | 0.105 | 0 | 0 | 658 B |
#6279 | RunOnMethodBegin |
net6.0 | 640ns | 1.17ns | 4.55ns | 0.0097 | 0 | 0 | 696 B |
#6279 | RunOnMethodBegin |
netcoreapp3.1 | 890ns | 1.67ns | 6.47ns | 0.00925 | 0 | 0 | 696 B |
#6279 | RunOnMethodBegin |
net472 | 1.19μs | 2.17ns | 8.4ns | 0.104 | 0 | 0 | 658 B |
9e10789
to
b53fbb0
Compare
b53fbb0
to
7cbf98c
Compare
7cbf98c
to
5895064
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM now!
Updates the package versions for integration tests. --------- Co-authored-by: andrewlock <andrewlock@users.noreply.github.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>
Updates the package versions for integration tests.