Convert integration tests to async #5018

kevingosse · 2024-01-05T16:10:52Z

Summary of changes

Convert the integration tests to async.

Reason for change

This is mainly done to avoid a sync-over-async introduced by the MemoryDumpHelper, but this should also allow us to convert other parts to async and reduce the pressure on the threadpool.

Implementation details

:eyetwitch:

Test coverage

Pretty much all the integration tests are impacted, so uh great coverage I guess.

datadog-ddstaging · 2024-01-05T16:32:52Z

Datadog Report

Branch report: kevin/async_all_the_things
Commit report: 9d478da
Test service: dd-trace-dotnet

✅ 0 Failed, 309916 Passed, 1448 Skipped, 39m 24.14s Wall Time

andrewlock · 2024-01-05T16:44:04Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5018) - mean (74ms)  : 55, 93
     .   : milestone, 74,
    master - mean (70ms)  : 62, 79
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (5018) - mean (1,018ms)  : 994, 1043
     .   : milestone, 1018,
    master - mean (1,019ms)  : 1000, 1038
     .   : milestone, 1019,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5018) - mean (106ms)  : 101, 110
     .   : milestone, 106,
    master - mean (105ms)  : 99, 110
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (5018) - mean (730ms)  : 714, 746
     .   : milestone, 730,
    master - mean (728ms)  : 712, 745
     .   : milestone, 728,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5018) - mean (89ms)  : 86, 93
     .   : milestone, 89,
    master - mean (90ms)  : 85, 94
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (5018) - mean (686ms)  : 664, 707
     .   : milestone, 686,
    master - mean (688ms)  : 668, 707
     .   : milestone, 688,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5018) - mean (188ms)  : 185, 191
     .   : milestone, 188,
    master - mean (188ms)  : 186, 190
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5018) - mean (1,130ms)  : 1114, 1146
     .   : milestone, 1130,
    master - mean (1,131ms)  : 1112, 1151
     .   : milestone, 1131,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5018) - mean (271ms)  : 266, 275
     .   : milestone, 271,
    master - mean (271ms)  : 266, 276
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (5018) - mean (1,085ms)  : 1065, 1105
     .   : milestone, 1085,
    master - mean (1,089ms)  : 1068, 1110
     .   : milestone, 1089,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5018) - mean (262ms)  : 258, 266
     .   : milestone, 262,
    master - mean (261ms)  : 258, 264
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (5018) - mean (1,055ms)  : 1032, 1077
     .   : milestone, 1055,
    master - mean (1,054ms)  : 1029, 1079
     .   : milestone, 1054,

andrewlock · 2024-01-05T17:35:13Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5018) (11.648M)   : 0, 11648303
    master (11.505M)   : 0, 11505045
    benchmarks/2.9.0 (11.038M)   : 0, 11037516

    section Automatic
    This PR (5018) (8.066M)   : 0, 8065563
    master (7.968M)   : 0, 7968334
    benchmarks/2.9.0 (8.016M)   : 0, 8015737

    section Trace stats
    This PR (5018) (8.357M)   : 0, 8356506
    master (8.238M)   : 0, 8237649

    section Manual
    This PR (5018) (10.245M)   : 0, 10244518
    master (10.096M)   : 0, 10095847

    section Manual + Automatic
    This PR (5018) (7.640M)   : 0, 7640219
    master (7.531M)   : 0, 7531312

    section Version Conflict
    This PR (5018) (6.947M)   : 0, 6947036
    master (6.771M)   : 0, 6770852

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5018) (9.371M)   : 0, 9370764
    master (9.512M)   : 0, 9512452
    benchmarks/2.9.0 (9.426M)   : 0, 9425527

    section Automatic
    This PR (5018) (6.511M)   : 0, 6510598
    master (6.574M)   : 0, 6574379

    section Trace stats
    This PR (5018) (6.743M)   : 0, 6743154
    master (6.778M)   : 0, 6777549

    section Manual
    This PR (5018) (8.142M)   : 0, 8142347
    master (8.382M)   : 0, 8382154

    section Manual + Automatic
    This PR (5018) (6.188M)   : 0, 6188446
    master (6.156M)   : 0, 6155846

    section Version Conflict
    This PR (5018) (5.663M)   : 0, 5662638
    master (5.759M)   : 0, 5758738

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5018) (10.340M)   : 0, 10339569
    master (10.122M)   : 0, 10121899
    benchmarks/2.9.0 (10.287M)   : 0, 10286693

    section Automatic
    This PR (5018) (7.158M)   : 0, 7158005
    master (7.104M)   : 0, 7104196
    benchmarks/2.9.0 (7.371M)   : 0, 7371038

    section Trace stats
    This PR (5018) (7.456M)   : 0, 7456301
    master (7.514M)   : 0, 7514168

    section Manual
    This PR (5018) (8.840M)   : 0, 8840435
    master (8.797M)   : 0, 8796561

    section Manual + Automatic
    This PR (5018) (6.900M)   : 0, 6900247
    master (6.822M)   : 0, 6822263

    section Version Conflict
    This PR (5018) (6.299M)   : 0, 6298874
    master (6.111M)   : 0, 6111488

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5018) (7.483M)   : 0, 7483227
    master (7.450M)   : 0, 7449569
    benchmarks/2.9.0 (7.874M)   : 0, 7873913

    section No attack
    This PR (5018) (1.741M)   : 0, 1740874
    master (1.775M)   : 0, 1775125
    benchmarks/2.9.0 (3.279M)   : 0, 3279165

    section Attack
    This PR (5018) (1.399M)   : 0, 1399433
    master (1.434M)   : 0, 1433867
    benchmarks/2.9.0 (2.566M)   : 0, 2566156

    section Blocking
    This PR (5018) (3.205M)   : 0, 3204588
    master (3.211M)   : 0, 3210516

    section IAST default
    This PR (5018) (6.560M)   : 0, 6559549
    master (6.468M)   : 0, 6468152

    section IAST full
    This PR (5018) (5.834M)   : 0, 5834360
    master (5.717M)   : 0, 5717130

    section Base vuln
    This PR (5018) (0.940M)   : 0, 940117
    master (0.954M)   : 0, 954483

    section IAST vuln
    This PR (5018) (0.885M)   : 0, 885081
    master (0.871M)   : 0, 870805

andrewlock · 2024-01-05T18:20:09Z

Benchmarks Report 🐌

Benchmarks for #5018 compared to master:

3 benchmarks are faster, with geometric mean 1.197
2 benchmarks are slower, with geometric mean 1.128
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.79μs	47.8ns	279ns	0.0139	0.00465	0	7.49 KB
master	`StartStopWithChild`	netcoreapp3.1	10.7μs	58.6ns	342ns	0.0327	0.0163	0	7.58 KB
master	`StartStopWithChild`	net472	17.1μs	53.8ns	208ns	1.35	0.347	0.11	7.94 KB
#5018	`StartStopWithChild`	net6.0	8.8μs	46.7ns	229ns	0.0357	0.0134	0	7.48 KB
#5018	`StartStopWithChild`	netcoreapp3.1	10.9μs	57.8ns	300ns	0.044	0.022	0.0055	7.58 KB
#5018	`StartStopWithChild`	net472	17.2μs	38.5ns	149ns	1.33	0.354	0.121	7.94 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	461μs	147ns	571ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	629μs	160ns	619ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	802μs	246ns	954ns	0.396	3.3 KB
#5018	`WriteAndFlushEnrichedTraces`	net6.0	458μs	138ns	515ns	0	2.7 KB
#5018	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	626μs	136ns	492ns	0	2.7 KB
#5018	`WriteAndFlushEnrichedTraces`	net472	791μs	296ns	1.11μs	0.396	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	38.7μs	68ns	264ns	0.019	0	1.77 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	41.8μs	91.8ns	356ns	0.0206	0	1.74 KB
master	`AllCycleSimpleBody`	net472	44.2μs	40ns	155ns	0.286	0	1.81 KB
master	`AllCycleMoreComplexBody`	net6.0	201μs	64.7ns	251ns	0.1	0	9.25 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	212μs	151ns	585ns	0.106	0	9.14 KB
master	`AllCycleMoreComplexBody`	net472	225μs	73.4ns	284ns	1.46	0	9.32 KB
master	`ObjectExtractorSimpleBody`	net6.0	144ns	0.0612ns	0.237ns	0.00393	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	220ns	0.0932ns	0.336ns	0.00365	0	272 B
master	`ObjectExtractorSimpleBody`	net472	170ns	0.0501ns	0.194ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.98μs	1.69ns	6.33ns	0.0538	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.06μs	0.96ns	3.59ns	0.0488	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.81μs	2.21ns	8.57ns	0.602	0.0057	3.8 KB
#5018	`AllCycleSimpleBody`	net6.0	38.7μs	47.1ns	182ns	0.0192	0	1.77 KB
#5018	`AllCycleSimpleBody`	netcoreapp3.1	41.2μs	26.1ns	101ns	0.0206	0	1.74 KB
#5018	`AllCycleSimpleBody`	net472	44.3μs	13.1ns	50.8ns	0.286	0	1.81 KB
#5018	`AllCycleMoreComplexBody`	net6.0	200μs	122ns	474ns	0.0995	0	9.25 KB
#5018	`AllCycleMoreComplexBody`	netcoreapp3.1	211μs	187ns	725ns	0.107	0	9.14 KB
#5018	`AllCycleMoreComplexBody`	net472	227μs	63.8ns	247ns	1.47	0	9.32 KB
#5018	`ObjectExtractorSimpleBody`	net6.0	145ns	0.0702ns	0.272ns	0.00396	0	280 B
#5018	`ObjectExtractorSimpleBody`	netcoreapp3.1	200ns	0.0822ns	0.319ns	0.00363	0	272 B
#5018	`ObjectExtractorSimpleBody`	net472	169ns	0.067ns	0.242ns	0.0446	0	281 B
#5018	`ObjectExtractorMoreComplexBody`	net6.0	3.03μs	0.738ns	2.66ns	0.053	0	3.78 KB
#5018	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.96μs	1.89ns	7.09ns	0.0494	0	3.69 KB
#5018	`ObjectExtractorMoreComplexBody`	net472	3.77μs	1.62ns	6.28ns	0.601	0.00566	3.8 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5018

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (10))‑net6.0	1.138	50,334.42	57,262.01

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	50.3μs	17.8ns	64.3ns	0.227	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	70.3μs	386ns	2.29μs	0.198	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	net472	95.5μs	55.2ns	199ns	2.53	0.0953	16.14 KB
master	`RunWafTwice(args=NestedMap (10))`	net6.0	57μs	334ns	3.11μs	0.237	0	16.6 KB
master	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	74μs	429ns	1.92μs	0.189	0	16.58 KB
master	`RunWafTwice(args=NestedMap (10))`	net472	104μs	44.7ns	173ns	2.6	0.104	16.69 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	109μs	54.7ns	212ns	0.287	0	22.41 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	129μs	600ns	2.55μs	0.316	0	22.36 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	164μs	793ns	3.46μs	3.55	0.158	22.7 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	105μs	582ns	3.73μs	0.464	0	32.76 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	131μs	740ns	4.97μs	0.408	0	33.33 KB
master	`RunWaf(args=NestedMap (100))`	net472	184μs	64.2ns	240ns	5.26	0.369	33.67 KB
master	`RunWafTwice(args=NestedMap (100))`	net6.0	99.6μs	27ns	101ns	0.449	0	33.3 KB
master	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	137μs	730ns	3.93μs	0.418	0	33.86 KB
master	`RunWafTwice(args=NestedMap (100))`	net472	193μs	211ns	817ns	5.38	0.384	34.23 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	157μs	134ns	482ns	0.546	0	39.1 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	201μs	989ns	4.2μs	0.479	0	39.63 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	260μs	425ns	1.65μs	6.35	0.508	40.23 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	104μs	292ns	1.13μs	0.438	0	32.18 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	127μs	525ns	1.89μs	0.377	0	32.3 KB
master	`RunWaf(args=NestedMap (20))`	net472	190μs	874ns	3.39μs	5.13	0.367	32.63 KB
master	`RunWafTwice(args=NestedMap (20))`	net6.0	113μs	81.6ns	316ns	0.45	0	32.72 KB
master	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	141μs	686ns	3.07μs	0.413	0	32.82 KB
master	`RunWafTwice(args=NestedMap (20))`	net472	195μs	805ns	3.12μs	5.22	0.386	33.19 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	163μs	70.3ns	272ns	0.543	0	38.53 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	200μs	1.01μs	4.5μs	0.484	0	38.6 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	250μs	128ns	497ns	6.23	0.498	39.2 KB
#5018	`RunWaf(args=NestedMap (10))`	net6.0	56.1μs	313ns	1.98μs	0.225	0	16.06 KB
#5018	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	70.2μs	367ns	1.94μs	0.207	0	16.06 KB
#5018	`RunWaf(args=NestedMap (10))`	net472	100μs	500ns	2.29μs	2.54	0.0959	16.14 KB
#5018	`RunWafTwice(args=NestedMap (10))`	net6.0	57.7μs	319ns	1.96μs	0.222	0	16.6 KB
#5018	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	74.4μs	360ns	1.44μs	0.209	0	16.58 KB
#5018	`RunWafTwice(args=NestedMap (10))`	net472	106μs	73.7ns	255ns	2.6	0.104	16.69 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	106μs	49.2ns	191ns	0.318	0	22.41 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	134μs	628ns	2.43μs	0.265	0	22.36 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	163μs	607ns	2.35μs	3.53	0.161	22.7 KB
#5018	`RunWaf(args=NestedMap (100))`	net6.0	99.6μs	36.2ns	140ns	0.422	0	32.76 KB
#5018	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	130μs	209ns	783ns	0.39	0	33.33 KB
#5018	`RunWaf(args=NestedMap (100))`	net472	192μs	869ns	3.37μs	5.32	0.373	33.67 KB
#5018	`RunWafTwice(args=NestedMap (100))`	net6.0	102μs	37ns	143ns	0.464	0	33.3 KB
#5018	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	141μs	744ns	3.79μs	0.429	0	33.86 KB
#5018	`RunWafTwice(args=NestedMap (100))`	net472	198μs	891ns	3.45μs	5.37	0.391	34.23 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	162μs	65.3ns	235ns	0.513	0	39.1 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	207μs	667ns	2.58μs	0.505	0	39.63 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	256μs	113ns	436ns	6.29	0.513	40.23 KB
#5018	`RunWaf(args=NestedMap (20))`	net6.0	105μs	550ns	2.69μs	0.458	0	32.18 KB
#5018	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	138μs	745ns	4.28μs	0.391	0	32.3 KB
#5018	`RunWaf(args=NestedMap (20))`	net472	187μs	871ns	3.48μs	5.15	0.368	32.63 KB
#5018	`RunWafTwice(args=NestedMap (20))`	net6.0	110μs	591ns	3.13μs	0.426	0	32.72 KB
#5018	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	138μs	737ns	4.04μs	0.416	0	32.82 KB
#5018	`RunWafTwice(args=NestedMap (20))`	net472	200μs	404ns	1.56μs	5.27	0.391	33.19 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	157μs	85.2ns	295ns	0.549	0	38.53 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	205μs	1.01μs	4.15μs	0.484	0	38.6 KB
#5018	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	253μs	97.2ns	376ns	6.21	0.507	39.2 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	171μs	145ns	542ns	0.171	18.25 KB
master	`SendRequest`	netcoreapp3.1	193μs	247ns	958ns	0.191	20.41 KB
master	`SendRequest`	net472	0.0013ns	0.00036ns	0.00135ns	0	0 b
#5018	`SendRequest`	net6.0	174μs	201ns	778ns	0.259	18.26 KB
#5018	`SendRequest`	netcoreapp3.1	191μs	371ns	1.44μs	0.19	20.41 KB
#5018	`SendRequest`	net472	0.000108ns	5.72E‑05ns	0.000222ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	555μs	1.2μs	4.49μs	0.553	0	0	41.86 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	661μs	1.38μs	5.36μs	0.332	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	net472	850μs	2.88μs	11.1μs	8.45	2.53	0.422	53.23 KB
#5018	`WriteAndFlushEnrichedTraces`	net6.0	563μs	1.4μs	5.4μs	0.566	0	0	41.76 KB
#5018	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	641μs	670ns	2.51μs	0.332	0	0	41.75 KB
#5018	`WriteAndFlushEnrichedTraces`	net472	808μs	3.52μs	13.6μs	8.06	2.42	0.403	53.22 KB

Benchmarks.Trace.DbCommandBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5018

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0	1.117	1,031.07	1,152.20

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.03μs	0.682ns	2.64ns	0.0105	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.49μs	1.24ns	4.79ns	0.0104	768 B
master	`ExecuteNonQuery`	net472	1.82μs	3.21ns	12.4ns	0.115	730 B
#5018	`ExecuteNonQuery`	net6.0	1.15μs	0.743ns	2.68ns	0.0107	768 B
#5018	`ExecuteNonQuery`	netcoreapp3.1	1.48μs	1.37ns	5.29ns	0.0104	768 B
#5018	`ExecuteNonQuery`	net472	1.82μs	1.32ns	5.1ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.21μs	1.15ns	4.44ns	0.0127	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.587ns	2.27ns	0.0129	936 B
master	`CallElasticsearch`	net472	2.45μs	0.894ns	3.46ns	0.152	955 B
master	`CallElasticsearchAsync`	net6.0	1.27μs	0.618ns	2.39ns	0.0128	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.671ns	2.51ns	0.0132	984 B
master	`CallElasticsearchAsync`	net472	2.57μs	1.55ns	6ns	0.16	1.01 KB
#5018	`CallElasticsearch`	net6.0	1.15μs	1.67ns	6.47ns	0.0129	936 B
#5018	`CallElasticsearch`	netcoreapp3.1	1.57μs	0.444ns	1.66ns	0.0126	936 B
#5018	`CallElasticsearch`	net472	2.5μs	1.16ns	4.49ns	0.151	955 B
#5018	`CallElasticsearchAsync`	net6.0	1.37μs	1.04ns	4.03ns	0.0123	912 B
#5018	`CallElasticsearchAsync`	netcoreapp3.1	1.71μs	5.56ns	21.5ns	0.0135	984 B
#5018	`CallElasticsearchAsync`	net472	2.53μs	0.628ns	2.43ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.33μs	0.459ns	1.78ns	0.0127	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.63μs	1.03ns	3.84ns	0.0122	912 B
master	`ExecuteAsync`	net472	1.84μs	0.612ns	2.29ns	0.139	875 B
#5018	`ExecuteAsync`	net6.0	1.2μs	0.405ns	1.52ns	0.0127	912 B
#5018	`ExecuteAsync`	netcoreapp3.1	1.63μs	3.84ns	13.8ns	0.0121	912 B
#5018	`ExecuteAsync`	net472	1.85μs	1.07ns	4.14ns	0.138	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.04μs	0.739ns	2.66ns	0.0285	2.1 KB
master	`SendAsync`	netcoreapp3.1	4.93μs	8.96ns	34.7ns	0.0368	2.63 KB
master	`SendAsync`	net472	7.66μs	4.19ns	15.7ns	0.522	3.31 KB
#5018	`SendAsync`	net6.0	4.15μs	1.53ns	5.93ns	0.0291	2.1 KB
#5018	`SendAsync`	netcoreapp3.1	5.02μs	1.69ns	6.53ns	0.035	2.63 KB
#5018	`SendAsync`	net472	7.84μs	3.28ns	12.7ns	0.525	3.31 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5018

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472	60.13 KB	61.09 KB	960 B	1.60%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net6.0	51.4μs	226ns	846ns	43.44 KB
master	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	53μs	263ns	1.09μs	42.64 KB
master	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net472	37.7μs	131ns	491ns	60.13 KB
master	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net6.0	66.4μs	180ns	696ns	43.29 KB
master	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	109μs	3.58μs	35.5μs	42.64 KB
master	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net472	64μs	224ns	807ns	57.34 KB
#5018	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net6.0	52.7μs	271ns	1.21μs	43.44 KB
#5018	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	57.8μs	592ns	5.77μs	42.64 KB
#5018	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net472	37.5μs	161ns	622ns	61.09 KB
#5018	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net6.0	66.1μs	296ns	1.15μs	43.29 KB
#5018	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	96.6μs	3.06μs	30.6μs	42.64 KB
#5018	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net472	63μs	283ns	1.1μs	57.34 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.43μs	0.811ns	3.14ns	0.0222	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	2.18μs	0.715ns	2.77ns	0.0207	1.57 KB
master	`EnrichedLog`	net472	2.63μs	2.36ns	9.15ns	0.237	1.5 KB
#5018	`EnrichedLog`	net6.0	1.42μs	0.774ns	2.9ns	0.0221	1.57 KB
#5018	`EnrichedLog`	netcoreapp3.1	2.22μs	0.899ns	3.48ns	0.0211	1.57 KB
#5018	`EnrichedLog`	net472	2.67μs	1.4ns	5.42ns	0.238	1.5 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	212ns	820ns	0.0568	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	167ns	647ns	0.0586	0	4.21 KB
master	`EnrichedLog`	net472	148μs	85.5ns	331ns	0.664	0.221	4.39 KB
#5018	`EnrichedLog`	net6.0	113μs	99.7ns	373ns	0.0567	0	4.21 KB
#5018	`EnrichedLog`	netcoreapp3.1	117μs	115ns	445ns	0	0	4.21 KB
#5018	`EnrichedLog`	net472	148μs	66.6ns	258ns	0.662	0.221	4.39 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.94μs	1.09ns	4.2ns	0.0295	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	4.16μs	2ns	7.21ns	0.0295	2.13 KB
master	`EnrichedLog`	net472	4.87μs	1.36ns	5.1ns	0.307	1.95 KB
#5018	`EnrichedLog`	net6.0	3.14μs	7.87ns	30.5ns	0.0295	2.13 KB
#5018	`EnrichedLog`	netcoreapp3.1	4.12μs	2.72ns	10.5ns	0.0289	2.13 KB
#5018	`EnrichedLog`	net472	4.93μs	2.04ns	7.92ns	0.308	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.42μs	0.802ns	3ns	0.0156	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.68μs	0.97ns	3.76ns	0.0153	1.1 KB
master	`SendReceive`	net472	2.04μs	1ns	3.74ns	0.177	1.12 KB
#5018	`SendReceive`	net6.0	1.32μs	1.13ns	4.37ns	0.0152	1.1 KB
#5018	`SendReceive`	netcoreapp3.1	1.8μs	1.97ns	7.63ns	0.0144	1.1 KB
#5018	`SendReceive`	net472	2.07μs	2.4ns	8.99ns	0.177	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.77μs	1.2ns	4.5ns	0.0208	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.97μs	1.37ns	5.12ns	0.02	1.58 KB
master	`EnrichedLog`	net472	4.28μs	2.27ns	8.8ns	0.312	1.97 KB
#5018	`EnrichedLog`	net6.0	2.74μs	0.501ns	1.94ns	0.0219	1.53 KB
#5018	`EnrichedLog`	netcoreapp3.1	3.95μs	2.32ns	8.97ns	0.0216	1.58 KB
#5018	`EnrichedLog`	net472	4.41μs	2.06ns	7.99ns	0.311	1.97 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5018

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.180	746.88	633.17
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.178	632.86	537.28

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	512ns	0.148ns	0.571ns	0.00743	536 B
master	`StartFinishSpan`	netcoreapp3.1	747ns	0.76ns	2.84ns	0.00729	536 B
master	`StartFinishSpan`	net472	785ns	0.404ns	1.57ns	0.0852	538 B
master	`StartFinishScope`	net6.0	633ns	0.459ns	1.72ns	0.00921	656 B
master	`StartFinishScope`	netcoreapp3.1	891ns	0.256ns	0.957ns	0.00893	656 B
master	`StartFinishScope`	net472	1.02μs	1.02ns	3.96ns	0.0978	618 B
#5018	`StartFinishSpan`	net6.0	461ns	0.272ns	1.02ns	0.00753	536 B
#5018	`StartFinishSpan`	netcoreapp3.1	633ns	0.255ns	0.955ns	0.00703	536 B
#5018	`StartFinishSpan`	net472	760ns	0.494ns	1.91ns	0.0852	538 B
#5018	`StartFinishScope`	net6.0	537ns	0.164ns	0.634ns	0.00933	656 B
#5018	`StartFinishScope`	netcoreapp3.1	898ns	0.355ns	1.38ns	0.00864	656 B
#5018	`StartFinishScope`	net472	951ns	1.77ns	6.85ns	0.0981	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5018

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.235	746.03	604.25

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	746ns	0.346ns	1.34ns	0.00918	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	924ns	0.474ns	1.83ns	0.00884	656 B
master	`RunOnMethodBegin`	net472	1.15μs	0.542ns	2.03ns	0.0982	618 B
#5018	`RunOnMethodBegin`	net6.0	605ns	1ns	3.75ns	0.00914	656 B
#5018	`RunOnMethodBegin`	netcoreapp3.1	963ns	0.52ns	2.01ns	0.00863	656 B
#5018	`RunOnMethodBegin`	net472	1.13μs	0.344ns	1.33ns	0.098	618 B

This reverts commit 3ce21bd.

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/AWS/AwsLambdaTests.cs

…LambdaTests.cs

andrewlock

It gives me a tiny bit of pleasure to know that was even more tedious for you to implement than it was for me to review

andrewlock · 2024-01-16T10:23:09Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/AWS/AwsLambdaTests.cs

@@ -3,7 +3,7 @@
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>

-#if NET6_0_OR_GREATER
+#if NET6_OR_GREATER


Trying to undo my good work? 😛

Suggested change

#if NET6_OR_GREATER

#if NET6_OR_GREATER

robertpi

As discussed on slack, did not review in depth.

GreenMatan

Debugger part seem fine by me.

kevingosse changed the base branch from master to kevin/create_suspended January 5, 2024 16:11

kevingosse force-pushed the kevin/create_suspended branch 2 times, most recently from beee0d1 to 621229b Compare January 8, 2024 11:56

Base automatically changed from kevin/create_suspended to master January 10, 2024 13:48

kevingosse force-pushed the kevin/async_all_the_things branch from 3e38854 to 50c36bc Compare January 10, 2024 14:32

kevingosse added 4 commits January 15, 2024 11:33

Asyncify what has to be asyncified

8806d26

Add missing await

0621eca

Add a few more

ad3011a

Move calls to SetHttpPort

627c9c2

kevingosse force-pushed the kevin/async_all_the_things branch from 446fff2 to 627c9c2 Compare January 15, 2024 10:38

kevingosse added 3 commits January 15, 2024 14:13

Moar async

4cf75ac

Always log the rejit errors

3ce21bd

Revert "Always log the rejit errors"

53d7ecd

This reverts commit 3ce21bd.

kevingosse marked this pull request as ready for review January 16, 2024 10:14

kevingosse requested a review from a team as a code owner January 16, 2024 10:14

kevingosse requested a review from a team January 16, 2024 10:14

kevingosse requested review from a team as code owners January 16, 2024 10:14

kevingosse changed the title ~~Kevin/async all the things~~ Convert integration tests to async Jan 16, 2024

kevingosse commented Jan 16, 2024

View reviewed changes

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/AWS/AwsLambdaTests.cs Outdated Show resolved Hide resolved

Update tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/AWS/Aws…

9d478da

…LambdaTests.cs

andrewlock approved these changes Jan 16, 2024

View reviewed changes

robertpi approved these changes Jan 16, 2024

View reviewed changes

GreenMatan approved these changes Jan 16, 2024

View reviewed changes

kevingosse merged commit 064b938 into master Jan 17, 2024
60 checks passed

kevingosse deleted the kevin/async_all_the_things branch January 17, 2024 10:07

github-actions bot added this to the vNext milestone Jan 17, 2024

andrewlock added the area:tests unit tests, integration tests label Jan 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert integration tests to async #5018

Convert integration tests to async #5018

kevingosse commented Jan 5, 2024 •

edited

Loading

datadog-ddstaging bot commented Jan 5, 2024 •

edited

Loading

andrewlock commented Jan 5, 2024 •

edited

Loading

andrewlock commented Jan 5, 2024 •

edited

Loading

andrewlock commented Jan 5, 2024 •

edited

Loading

Raw results

Raw results

Raw results

Slower ⚠️ in #5018

Raw results

Raw results

Raw results

Slower ⚠️ in #5018

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5018

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5018

Raw results

Faster 🎉 in #5018

Raw results

andrewlock left a comment

andrewlock Jan 16, 2024

robertpi left a comment

GreenMatan left a comment •

edited

Loading

Convert integration tests to async #5018

Convert integration tests to async #5018

Conversation

kevingosse commented Jan 5, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Jan 5, 2024 • edited Loading

Datadog Report

andrewlock commented Jan 5, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 5, 2024 • edited Loading

Throughput/Crank Report:zap:

andrewlock commented Jan 5, 2024 • edited Loading

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Slower ⚠️ in #5018

Raw results

Raw results

Raw results

Slower ⚠️ in #5018

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5018

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5018

Raw results

Faster 🎉 in #5018

Raw results

andrewlock left a comment

Choose a reason for hiding this comment

andrewlock Jan 16, 2024

Choose a reason for hiding this comment

robertpi left a comment

Choose a reason for hiding this comment

GreenMatan left a comment • edited Loading

Choose a reason for hiding this comment

kevingosse commented Jan 5, 2024 •

edited

Loading

datadog-ddstaging bot commented Jan 5, 2024 •

edited

Loading

andrewlock commented Jan 5, 2024 •

edited

Loading

andrewlock commented Jan 5, 2024 •

edited

Loading

andrewlock commented Jan 5, 2024 •

edited

Loading

GreenMatan left a comment •

edited

Loading