Ensure analyzer tests run against all TFMs #6502

andrewlock · 2025-01-06T12:35:59Z

Summary of changes

Removes the restriction to only running on .NET 5

Reason for change

There's no good reason for it to be this, it's almost certainly copy pasta

Implementation details

Remove the restricting target

Test coverage

The CI should reveal all...

Other details

Revealed by

Fix build issues on aspects on net5.0 #6500

datadog-ddstaging · 2025-01-06T12:59:49Z

Datadog Report

Branch report: andrew/ci/run-against-all-tfms
Commit report: ce6192c
Test service: dd-trace-dotnet

✅ 0 Failed, 239125 Passed, 2074 Skipped, 18h 38m 46.06s Total Time

andrewlock · 2025-01-06T13:22:29Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6502) - mean (68ms)  : 66, 71
     .   : milestone, 68,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6502) - mean (977ms)  : 945, 1010
     .   : milestone, 977,
    master - mean (973ms)  : 948, 998
     .   : milestone, 973,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6502) - mean (107ms)  : 105, 110
     .   : milestone, 107,
    master - mean (107ms)  : 104, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6502) - mean (676ms)  : 661, 691
     .   : milestone, 676,
    master - mean (678ms)  : 664, 691
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6502) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6502) - mean (630ms)  : 615, 646
     .   : milestone, 630,
    master - mean (631ms)  : 616, 645
     .   : milestone, 631,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6502) - mean (194ms)  : 189, 199
     .   : milestone, 194,
    master - mean (195ms)  : 190, 200
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6502) - mean (1,105ms)  : 1075, 1136
     .   : milestone, 1105,
    master - mean (1,099ms)  : 1063, 1135
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6502) - mean (279ms)  : 274, 283
     .   : milestone, 279,
    master - mean (277ms)  : 273, 282
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6502) - mean (874ms)  : 848, 900
     .   : milestone, 874,
    master - mean (873ms)  : 842, 903
     .   : milestone, 873,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6502) - mean (267ms)  : 263, 271
     .   : milestone, 267,
    master - mean (267ms)  : 263, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6502) - mean (850ms)  : 821, 880
     .   : milestone, 850,
    master - mean (851ms)  : 814, 888
     .   : milestone, 851,

andrewlock · 2025-01-06T17:45:28Z

Benchmarks Report for tracer 🐌

Benchmarks for #6502 compared to master:

1 benchmarks are faster, with geometric mean 1.182
1 benchmarks are slower, with geometric mean 1.126
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.99μs	44.6ns	303ns	0.016	0.00399	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	55.8ns	325ns	0.0152	0.00507	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	68.2ns	264ns	1.04	0.307	0.0914	6.2 KB
#6502	`StartStopWithChild`	net6.0	8.03μs	46.4ns	391ns	0.0131	0.00436	0	5.62 KB
#6502	`StartStopWithChild`	netcoreapp3.1	9.95μs	55.1ns	335ns	0.0199	0.00499	0	5.81 KB
#6502	`StartStopWithChild`	net472	16.5μs	70.9ns	275ns	1.03	0.298	0.0827	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	503μs	624ns	2.41μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	660μs	515ns	2μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	858μs	757ns	2.93μs	0.431	3.3 KB
#6502	`WriteAndFlushEnrichedTraces`	net6.0	483μs	263ns	985ns	0	2.7 KB
#6502	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	664μs	841ns	3.26μs	0	2.7 KB
#6502	`WriteAndFlushEnrichedTraces`	net472	846μs	430ns	1.67μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	905ns	8.24μs	0.145	14.47 KB
master	`SendRequest`	netcoreapp3.1	177μs	1.02μs	7.96μs	0.163	17.27 KB
master	`SendRequest`	net472	0.00113ns	0.000522ns	0.00202ns	0	0 b
#6502	`SendRequest`	net6.0	162μs	1.37μs	13.6μs	0.166	14.47 KB
#6502	`SendRequest`	netcoreapp3.1	178μs	1.31μs	12.9μs	0.156	17.27 KB
#6502	`SendRequest`	net472	0.000974ns	0.000662ns	0.00248ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	571μs	3.23μs	24.4μs	0.543	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	3.78μs	27μs	0.391	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	net472	852μs	3.98μs	15.4μs	8.39	2.52	0.419	53.33 KB
#6502	`WriteAndFlushEnrichedTraces`	net6.0	573μs	3.01μs	16.2μs	0.532	0	0	41.53 KB
#6502	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	708μs	4.12μs	35.5μs	0.342	0	0	41.82 KB
#6502	`WriteAndFlushEnrichedTraces`	net472	839μs	4.23μs	19.4μs	8.08	2.55	0.425	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6502

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0	1.126	1,204.62	1,355.85

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.2μs	1.4ns	5.43ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.29ns	4.99ns	0.0132	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.1μs	3.38ns	13.1ns	0.157	0.00105	987 B
#6502	`ExecuteNonQuery`	net6.0	1.36μs	1.69ns	6.54ns	0.0142	0	1.02 KB
#6502	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.76ns	6.58ns	0.0136	0	1.02 KB
#6502	`ExecuteNonQuery`	net472	2.09μs	1.68ns	6.52ns	0.156	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.22μs	1.79ns	6.92ns	0.014	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	2.37ns	8.21ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.59μs	2.4ns	9.29ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.35μs	0.501ns	1.81ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	1.03ns	3.97ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	2.36ns	9.13ns	0.167	1.05 KB
#6502	`CallElasticsearch`	net6.0	1.3μs	0.424ns	1.59ns	0.0137	976 B
#6502	`CallElasticsearch`	netcoreapp3.1	1.53μs	0.223ns	0.803ns	0.013	976 B
#6502	`CallElasticsearch`	net472	2.51μs	1.82ns	6.55ns	0.158	995 B
#6502	`CallElasticsearchAsync`	net6.0	1.45μs	1.44ns	5.56ns	0.0137	952 B
#6502	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	1.05ns	3.92ns	0.0135	1.02 KB
#6502	`CallElasticsearchAsync`	net472	2.73μs	1.77ns	6.87ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6502

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0	1.182	1,364.65	1,154.15

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.772ns	2.89ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	1.19ns	4.44ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.77μs	0.331ns	1.24ns	0.145	915 B
#6502	`ExecuteAsync`	net6.0	1.15μs	0.497ns	1.86ns	0.0131	952 B
#6502	`ExecuteAsync`	netcoreapp3.1	1.64μs	0.696ns	2.61ns	0.0131	952 B
#6502	`ExecuteAsync`	net472	1.92μs	0.558ns	2.16ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.52μs	1.57ns	5.88ns	0.0316	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.2μs	2.32ns	9ns	0.0388	2.85 KB
master	`SendAsync`	net472	7.47μs	1.93ns	7.47ns	0.496	3.12 KB
#6502	`SendAsync`	net6.0	4.26μs	2.94ns	11ns	0.032	2.31 KB
#6502	`SendAsync`	netcoreapp3.1	5.24μs	2.44ns	9.44ns	0.0394	2.85 KB
#6502	`SendAsync`	net472	7.35μs	1.74ns	6.52ns	0.496	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.47μs	0.695ns	2.6ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.23μs	1.01ns	3.78ns	0.0221	1.64 KB
master	`EnrichedLog`	net472	2.82μs	1.59ns	5.93ns	0.249	1.57 KB
#6502	`EnrichedLog`	net6.0	1.5μs	0.495ns	1.85ns	0.0232	1.64 KB
#6502	`EnrichedLog`	netcoreapp3.1	2.36μs	0.711ns	2.75ns	0.0223	1.64 KB
#6502	`EnrichedLog`	net472	2.75μs	1.24ns	4.79ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	117μs	110ns	425ns	0.0582	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	250ns	968ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	106ns	398ns	0.683	0.228	4.46 KB
#6502	`EnrichedLog`	net6.0	117μs	118ns	457ns	0.058	0	4.28 KB
#6502	`EnrichedLog`	netcoreapp3.1	121μs	109ns	409ns	0	0	4.28 KB
#6502	`EnrichedLog`	net472	151μs	133ns	517ns	0.678	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.04μs	1.18ns	4.4ns	0.0299	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.26μs	2.26ns	8.77ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.9μs	2.9ns	10.8ns	0.319	2.02 KB
#6502	`EnrichedLog`	net6.0	2.84μs	1.03ns	3.85ns	0.0312	2.2 KB
#6502	`EnrichedLog`	netcoreapp3.1	4.25μs	3.59ns	13.4ns	0.0296	2.2 KB
#6502	`EnrichedLog`	net472	4.85μs	1.83ns	6.86ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.31μs	0.763ns	2.86ns	0.0164	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.82μs	0.67ns	2.42ns	0.0154	1.14 KB
master	`SendReceive`	net472	2.1μs	1.22ns	4.71ns	0.183	1.16 KB
#6502	`SendReceive`	net6.0	1.33μs	0.888ns	3.44ns	0.0159	1.14 KB
#6502	`SendReceive`	netcoreapp3.1	1.76μs	0.804ns	3.01ns	0.015	1.14 KB
#6502	`SendReceive`	net472	2.11μs	1.55ns	6.02ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	0.802ns	3ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.98μs	1.3ns	5.05ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	4.39μs	3.91ns	15.1ns	0.322	2.04 KB
#6502	`EnrichedLog`	net6.0	2.67μs	0.723ns	2.71ns	0.0226	1.6 KB
#6502	`EnrichedLog`	netcoreapp3.1	3.87μs	1.11ns	4.31ns	0.0212	1.65 KB
#6502	`EnrichedLog`	net472	4.48μs	3.73ns	14ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	409ns	0.257ns	0.995ns	0.00802	576 B
master	`StartFinishSpan`	netcoreapp3.1	564ns	1.45ns	5.61ns	0.00768	576 B
master	`StartFinishSpan`	net472	636ns	0.721ns	2.79ns	0.0917	578 B
master	`StartFinishScope`	net6.0	549ns	0.265ns	0.993ns	0.00987	696 B
master	`StartFinishScope`	netcoreapp3.1	681ns	0.888ns	3.44ns	0.00929	696 B
master	`StartFinishScope`	net472	901ns	1.03ns	3.97ns	0.104	658 B
#6502	`StartFinishSpan`	net6.0	455ns	0.29ns	1.12ns	0.00798	576 B
#6502	`StartFinishSpan`	netcoreapp3.1	624ns	0.292ns	1.05ns	0.00781	576 B
#6502	`StartFinishSpan`	net472	659ns	0.634ns	2.45ns	0.0915	578 B
#6502	`StartFinishScope`	net6.0	555ns	0.271ns	1.01ns	0.00985	696 B
#6502	`StartFinishScope`	netcoreapp3.1	681ns	1.25ns	4.85ns	0.00957	696 B
#6502	`StartFinishScope`	net472	935ns	1.36ns	5.25ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	656ns	0.399ns	1.54ns	0.00976	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	932ns	0.652ns	2.53ns	0.00932	696 B
master	`RunOnMethodBegin`	net472	1.11μs	0.355ns	1.33ns	0.104	658 B
#6502	`RunOnMethodBegin`	net6.0	654ns	0.469ns	1.82ns	0.00983	696 B
#6502	`RunOnMethodBegin`	netcoreapp3.1	923ns	0.444ns	1.66ns	0.0095	696 B
#6502	`RunOnMethodBegin`	net472	1.14μs	0.225ns	0.811ns	0.105	658 B

Ensure analyzer tests run against all TFMs

ce6192c

andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests labels Jan 6, 2025

andrewlock requested a review from a team as a code owner January 6, 2025 12:36

robertpi approved these changes Jan 6, 2025

View reviewed changes

e-n-0 approved these changes Jan 6, 2025

View reviewed changes

kevingosse approved these changes Jan 6, 2025

View reviewed changes

lucaspimentel approved these changes Jan 6, 2025

View reviewed changes

andrewlock merged commit aa9cacd into master Jan 7, 2025
100 of 102 checks passed

andrewlock deleted the andrew/ci/run-against-all-tfms branch January 7, 2025 12:06

github-actions bot added this to the vNext-v3 milestone Jan 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure analyzer tests run against all TFMs #6502

Ensure analyzer tests run against all TFMs #6502

andrewlock commented Jan 6, 2025

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025

andrewlock commented Jan 6, 2025

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6502

Raw results

Raw results

Faster 🎉 in #6502

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Ensure analyzer tests run against all TFMs #6502

Ensure analyzer tests run against all TFMs #6502

Conversation

andrewlock commented Jan 6, 2025

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Jan 6, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 6, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 6, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6502

Raw results

Raw results

Faster 🎉 in #6502

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading