Merge `throughput_appsec` into `throughput` stage #4740

andrewlock · 2023-10-20T10:04:48Z

Summary of changes

Does what it says on the tin.

Reason for change

In AzureDevops all the dependencies of a stage need to run before the stage runs. Unfortunately, that falls apart if you want to conditionally run a stage (as we do with throughput_appsec). That requires a bunch of gymnastics to make things work

Implementation details

Move the AppSec job into the Throughput stage. We are still using separate jobs, so there's no impact on the tracer throughput tests - they're still independent. But as the throughput stage always runs, we don't have to do anything weird to make sure compare_throughput runs.

The only "regression" is that the throughput stage depends on:

package_linux, package_arm64, build_windows_tracer, merge_commit_id

whereas throughput_appsec previously did not depend on the windows or arm64 builds

package_linux, merge_commit_id

This probably isn't a big issue for ASM, but it's the only thing that could impact the change

Test coverage

Made a no-op change to an ASM file to try to make sure that the stage runs as expected
Removed the change, to make sure the stage doesn't run, but we still get comparisons

Other details

Stacked on

Make compare_throughput stage more robust #4739

andrewlock · 2023-10-20T10:47:34Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4740) - mean (70ms)  : 62, 78
     .   : milestone, 70,
    master - mean (70ms)  : 63, 78
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4740) - mean (1,018ms)  : 989, 1047
     .   : milestone, 1018,
    master - mean (1,022ms)  : 1000, 1043
     .   : milestone, 1022,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4740) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (107ms)  : 101, 112
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (4740) - mean (712ms)  : 688, 736
     .   : milestone, 712,
    master - mean (718ms)  : 698, 737
     .   : milestone, 718,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4740) - mean (91ms)  : 88, 93
     .   : milestone, 91,
    master - mean (90ms)  : 84, 96
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4740) - mean (682ms)  : 669, 696
     .   : milestone, 682,
    master - mean (686ms)  : 661, 711
     .   : milestone, 686,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4740) - mean (188ms)  : 185, 190
     .   : milestone, 188,
    master - mean (188ms)  : 186, 190
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4740) - mean (1,125ms)  : 1103, 1147
     .   : milestone, 1125,
    master - mean (1,127ms)  : 1106, 1149
     .   : milestone, 1127,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4740) - mean (272ms)  : 268, 276
     .   : milestone, 272,
    master - mean (272ms)  : 268, 277
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (4740) - mean (1,086ms)  : 1058, 1115
     .   : milestone, 1086,
    master - mean (1,087ms)  : 1056, 1118
     .   : milestone, 1087,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4740) - mean (261ms)  : 258, 264
     .   : milestone, 261,
    master - mean (261ms)  : 256, 267
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4740) - mean (1,054ms)  : 1035, 1074
     .   : milestone, 1054,
    master - mean (1,050ms)  : 1028, 1073
     .   : milestone, 1050,

datadog-ddstaging · 2023-10-20T11:22:39Z

Datadog Report

Branch report: andrew/ci/merge_throughput_tests
Commit report: 717c963

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 305532 Passed, 1172 Skipped, 37m 41.82s Wall Time

andrewlock · 2023-10-20T11:48:30Z

Benchmarks Report 🐌

Benchmarks for #4740 compared to master:

4 benchmarks are faster, with geometric mean 1.175
2 benchmarks are slower, with geometric mean 1.179
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.21μs	45ns	281ns	0.0244	0.00815	0	7.29 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	56.3ns	374ns	0.0304	0.0152	0	7.38 KB
master	`StartStopWithChild`	net472	15.7μs	61.3ns	237ns	1.29	0.329	0.0919	7.67 KB
#4740	`StartStopWithChild`	net6.0	8.16μs	36.6ns	142ns	0.023	0.0115	0	7.29 KB
#4740	`StartStopWithChild`	netcoreapp3.1	10.1μs	47.3ns	189ns	0.0294	0.0147	0	7.38 KB
#4740	`StartStopWithChild`	net472	15.8μs	41.6ns	161ns	1.28	0.325	0.101	7.65 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	488μs	252ns	976ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	638μs	718ns	2.59μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	795μs	550ns	2.06μs	0.398	3.3 KB
#4740	`WriteAndFlushEnrichedTraces`	net6.0	468μs	275ns	1.06μs	0	2.7 KB
#4740	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	295ns	1.14μs	0	2.7 KB
#4740	`WriteAndFlushEnrichedTraces`	net472	790μs	197ns	736ns	0.396	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4740

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0	1.235	156.04	126.35

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	41.8μs	18.6ns	72.1ns	0.0209	0	2.03 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	45.3μs	99.2ns	371ns	0.0224	0	2.01 KB
master	`AllCycleSimpleBody`	net472	46.8μs	40.2ns	156ns	0.325	0	2.08 KB
master	`AllCycleMoreComplexBody`	net6.0	230μs	90.4ns	350ns	0.115	0	8.63 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	233μs	118ns	456ns	0.119	0	8.52 KB
master	`AllCycleMoreComplexBody`	net472	239μs	111ns	431ns	1.31	0	8.7 KB
master	`ObjectExtractorSimpleBody`	net6.0	156ns	0.0384ns	0.144ns	0.00394	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	173ns	0.223ns	0.865ns	0.00375	0	272 B
master	`ObjectExtractorSimpleBody`	net472	150ns	0.0829ns	0.31ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.07μs	0.943ns	3.53ns	0.0536	0	3.88 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.08μs	2.15ns	8.32ns	0.0514	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.1μs	1.99ns	7.71ns	0.617	0.00617	3.89 KB
#4740	`AllCycleSimpleBody`	net6.0	43.6μs	34.3ns	133ns	0.0212	0	2.03 KB
#4740	`AllCycleSimpleBody`	netcoreapp3.1	45.5μs	36.7ns	137ns	0.0227	0	2.01 KB
#4740	`AllCycleSimpleBody`	net472	47.3μs	47.9ns	185ns	0.314	0	2.08 KB
#4740	`AllCycleMoreComplexBody`	net6.0	228μs	116ns	434ns	0.117	0	8.63 KB
#4740	`AllCycleMoreComplexBody`	netcoreapp3.1	235μs	193ns	722ns	0.117	0	8.52 KB
#4740	`AllCycleMoreComplexBody`	net472	237μs	79.4ns	297ns	1.3	0	8.7 KB
#4740	`ObjectExtractorSimpleBody`	net6.0	126ns	0.0601ns	0.225ns	0.00396	0	280 B
#4740	`ObjectExtractorSimpleBody`	netcoreapp3.1	177ns	0.0726ns	0.281ns	0.00372	0	272 B
#4740	`ObjectExtractorSimpleBody`	net472	149ns	0.0986ns	0.382ns	0.0446	0	281 B
#4740	`ObjectExtractorMoreComplexBody`	net6.0	3.02μs	1.46ns	5.46ns	0.0545	0	3.88 KB
#4740	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.17μs	1.16ns	4.5ns	0.0521	0	3.78 KB
#4740	`ObjectExtractorMoreComplexBody`	net472	4.12μs	2.15ns	8.33ns	0.618	0.00618	3.89 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	12.8μs	3.74ns	14ns	0.134	0	9.42 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	19.7μs	12.5ns	48.5ns	0.127	0	9.42 KB
master	`RunWaf(args=NestedMap (10))`	net472	28.4μs	10.6ns	39.6ns	1.49	0	9.48 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	75.9μs	41.1ns	154ns	0.195	0	15.77 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	86.6μs	148ns	574ns	0.175	0	15.72 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	97.1μs	44.1ns	159ns	2.53	0	16.04 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	24.2μs	9.61ns	36ns	0.268	0	19.66 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	38.2μs	17.1ns	66.3ns	0.268	0	20.42 KB
master	`RunWaf(args=NestedMap (100))`	net472	54.8μs	19.5ns	75.4ns	3.25	0.0274	20.63 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	92.8μs	31.5ns	122ns	0.37	0	26.01 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	111μs	50.3ns	181ns	0.33	0	26.72 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	130μs	54.6ns	197ns	4.29	0.065	27.19 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	25.2μs	7.11ns	27.5ns	0.265	0	19.4 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	37.8μs	12.4ns	48ns	0.264	0	19.84 KB
master	`RunWaf(args=NestedMap (20))`	net472	53.5μs	22.1ns	85.7ns	3.18	0.0534	20.04 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	90.8μs	31.6ns	122ns	0.362	0	25.74 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	108μs	171ns	662ns	0.321	0	26.14 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	129μs	56.8ns	205ns	4.21	0.0647	26.6 KB
#4740	`RunWaf(args=NestedMap (10))`	net6.0	12.8μs	3.68ns	13.8ns	0.134	0	9.42 KB
#4740	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	19.6μs	4.39ns	16.4ns	0.128	0	9.42 KB
#4740	`RunWaf(args=NestedMap (10))`	net472	28.4μs	9.96ns	38.6ns	1.5	0	9.48 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	76.2μs	65.7ns	246ns	0.229	0	15.77 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	86.3μs	161ns	601ns	0.17	0	15.72 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	99μs	166ns	643ns	2.53	0	16.04 KB
#4740	`RunWaf(args=NestedMap (100))`	net6.0	24.4μs	6.65ns	24.9ns	0.278	0	19.66 KB
#4740	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	38.1μs	15.4ns	57.5ns	0.267	0	20.42 KB
#4740	`RunWaf(args=NestedMap (100))`	net472	54.6μs	12ns	41.4ns	3.26	0.0274	20.63 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	92μs	158ns	590ns	0.325	0	26.01 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	107μs	97.7ns	378ns	0.32	0	26.72 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	132μs	191ns	741ns	4.26	0.0655	27.19 KB
#4740	`RunWaf(args=NestedMap (20))`	net6.0	23.6μs	7.51ns	28.1ns	0.271	0	19.4 KB
#4740	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	37.5μs	8.41ns	32.6ns	0.263	0	19.84 KB
#4740	`RunWaf(args=NestedMap (20))`	net472	55.6μs	14.1ns	52.7ns	3.18	0.0558	20.04 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	92.8μs	38.3ns	148ns	0.324	0	25.74 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	108μs	133ns	514ns	0.321	0	26.14 KB
#4740	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	132μs	64.2ns	240ns	4.22	0.0649	26.6 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	167μs	226ns	877ns	0.166	18.04 KB
master	`SendRequest`	netcoreapp3.1	190μs	267ns	1.04μs	0.189	20.2 KB
master	`SendRequest`	net472	0.000613ns	0.000219ns	0.000847ns	0	0 b
#4740	`SendRequest`	net6.0	168μs	171ns	638ns	0.254	18.04 KB
#4740	`SendRequest`	netcoreapp3.1	188μs	307ns	1.19μs	0.19	20.2 KB
#4740	`SendRequest`	net472	0.000104ns	0.000104ns	0.000389ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	533μs	322ns	1.25μs	0.539	0	0	41.38 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	1.74μs	6.72μs	0.329	0	0	41.78 KB
master	`WriteAndFlushEnrichedTraces`	net472	869μs	4.11μs	15.9μs	8.19	2.59	0.431	53.25 KB
#4740	`WriteAndFlushEnrichedTraces`	net6.0	543μs	1.15μs	4.46μs	0.548	0	0	41.47 KB
#4740	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	643μs	774ns	3μs	0.326	0	0	41.67 KB
#4740	`WriteAndFlushEnrichedTraces`	net472	844μs	3.43μs	12.8μs	8.13	2.57	0.428	53.24 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.03μs	0.92ns	3.56ns	0.0104	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.25μs	2.13ns	8.24ns	0.0105	768 B
master	`ExecuteNonQuery`	net472	1.61μs	0.509ns	1.84ns	0.116	730 B
#4740	`ExecuteNonQuery`	net6.0	1.01μs	0.274ns	0.948ns	0.0107	768 B
#4740	`ExecuteNonQuery`	netcoreapp3.1	1.37μs	0.904ns	3.5ns	0.0101	768 B
#4740	`ExecuteNonQuery`	net472	1.63μs	0.667ns	2.58ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.08μs	0.566ns	2.19ns	0.0131	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.38μs	2.39ns	9.24ns	0.0126	936 B
master	`CallElasticsearch`	net472	2.48μs	0.762ns	2.95ns	0.152	955 B
master	`CallElasticsearchAsync`	net6.0	1.37μs	1.36ns	5.25ns	0.0124	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.49μs	0.711ns	2.56ns	0.0132	984 B
master	`CallElasticsearchAsync`	net472	2.56μs	0.939ns	3.64ns	0.16	1.01 KB
#4740	`CallElasticsearch`	net6.0	1.12μs	0.552ns	2.14ns	0.013	936 B
#4740	`CallElasticsearch`	netcoreapp3.1	1.5μs	1.26ns	4.88ns	0.0128	936 B
#4740	`CallElasticsearch`	net472	2.37μs	0.403ns	1.45ns	0.151	955 B
#4740	`CallElasticsearchAsync`	net6.0	1.28μs	0.857ns	3.32ns	0.0127	912 B
#4740	`CallElasticsearchAsync`	netcoreapp3.1	1.52μs	0.769ns	2.98ns	0.0131	984 B
#4740	`CallElasticsearchAsync`	net472	2.55μs	0.666ns	2.4ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net6.0	1.24μs	1.34ns	5.2ns	0.0125	0	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.55μs	0.909ns	3.52ns	0.0124	0	912 B
master	`ExecuteAsync`	net472	1.7μs	0.595ns	2.22ns	0.139	0.000851	875 B
#4740	`ExecuteAsync`	net6.0	1.16μs	0.641ns	2.4ns	0.0128	0	912 B
#4740	`ExecuteAsync`	netcoreapp3.1	1.45μs	1.3ns	4.86ns	0.0122	0	912 B
#4740	`ExecuteAsync`	net472	1.74μs	0.753ns	2.82ns	0.139	0.000868	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	3.84μs	3.88ns	15ns	0.0271	1.9 KB
master	`SendAsync`	netcoreapp3.1	4.34μs	1.99ns	7.19ns	0.0324	2.43 KB
master	`SendAsync`	net472	7.06μs	1.28ns	4.62ns	0.475	2.99 KB
#4740	`SendAsync`	net6.0	3.66μs	1.05ns	3.92ns	0.0258	1.9 KB
#4740	`SendAsync`	netcoreapp3.1	4.32μs	6.47ns	24.2ns	0.0325	2.43 KB
#4740	`SendAsync`	net472	6.99μs	1.55ns	5.81ns	0.473	2.99 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.34μs	0.547ns	2.05ns	0.022	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	1.93μs	0.624ns	2.34ns	0.0212	1.57 KB
master	`EnrichedLog`	net472	2.33μs	1.31ns	5.09ns	0.236	1.49 KB
#4740	`EnrichedLog`	net6.0	1.31μs	0.577ns	2.23ns	0.0221	1.57 KB
#4740	`EnrichedLog`	netcoreapp3.1	2μs	1.31ns	5.06ns	0.021	1.57 KB
#4740	`EnrichedLog`	net472	2.18μs	1.65ns	6.17ns	0.237	1.49 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	111μs	79.6ns	298ns	0.0559	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	116μs	120ns	448ns	0	0	4.21 KB
master	`EnrichedLog`	net472	146μs	108ns	418ns	0.658	0.219	4.38 KB
#4740	`EnrichedLog`	net6.0	113μs	188ns	729ns	0.0559	0	4.21 KB
#4740	`EnrichedLog`	netcoreapp3.1	117μs	175ns	679ns	0	0	4.21 KB
#4740	`EnrichedLog`	net472	146μs	50.9ns	190ns	0.682	0.227	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.06μs	0.948ns	3.55ns	0.0305	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	2.31ns	8.96ns	0.0293	2.13 KB
master	`EnrichedLog`	net472	4.7μs	1.64ns	6.12ns	0.305	1.93 KB
#4740	`EnrichedLog`	net6.0	2.93μs	1.15ns	4.14ns	0.0306	2.13 KB
#4740	`EnrichedLog`	netcoreapp3.1	3.98μs	0.722ns	2.6ns	0.0278	2.13 KB
#4740	`EnrichedLog`	net472	4.58μs	1.32ns	5.11ns	0.307	1.93 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.29μs	0.895ns	3.47ns	0.0155	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.65μs	0.564ns	2.03ns	0.015	1.1 KB
master	`SendReceive`	net472	1.98μs	0.691ns	2.59ns	0.177	1.12 KB
#4740	`SendReceive`	net6.0	1.25μs	0.838ns	3.25ns	0.0154	1.1 KB
#4740	`SendReceive`	netcoreapp3.1	1.6μs	0.898ns	3.48ns	0.0151	1.1 KB
#4740	`SendReceive`	net472	1.96μs	1.1ns	4.25ns	0.177	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.46μs	0.522ns	2.02ns	0.0209	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.5μs	1.66ns	6.41ns	0.021	1.58 KB
master	`EnrichedLog`	net472	4.11μs	1.51ns	5.87ns	0.31	1.96 KB
#4740	`EnrichedLog`	net6.0	2.66μs	1.35ns	5.23ns	0.0213	1.53 KB
#4740	`EnrichedLog`	netcoreapp3.1	3.69μs	4.45ns	17.2ns	0.0202	1.58 KB
#4740	`EnrichedLog`	net472	4.02μs	0.982ns	3.68ns	0.31	1.96 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4740

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.182	582.78	688.75
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.177	381.15	448.45

Faster 🎉 in #4740

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.172	590.57	503.73
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.153	917.79	795.90
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.143	758.15	663.45

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	381ns	0.163ns	0.61ns	0.00757	536 B
master	`StartFinishSpan`	netcoreapp3.1	535ns	0.206ns	0.771ns	0.00725	536 B
master	`StartFinishSpan`	net472	583ns	0.161ns	0.622ns	0.0852	538 B
master	`StartFinishScope`	net6.0	591ns	0.263ns	0.948ns	0.00918	656 B
master	`StartFinishScope`	netcoreapp3.1	758ns	0.339ns	1.31ns	0.00875	656 B
master	`StartFinishScope`	net472	918ns	0.28ns	1.01ns	0.0981	618 B
#4740	`StartFinishSpan`	net6.0	448ns	0.095ns	0.355ns	0.00743	536 B
#4740	`StartFinishSpan`	netcoreapp3.1	505ns	0.121ns	0.453ns	0.00711	536 B
#4740	`StartFinishSpan`	net472	689ns	0.256ns	0.991ns	0.0853	538 B
#4740	`StartFinishScope`	net6.0	504ns	0.106ns	0.395ns	0.00931	656 B
#4740	`StartFinishScope`	netcoreapp3.1	664ns	0.263ns	0.984ns	0.00875	656 B
#4740	`StartFinishScope`	net472	796ns	0.835ns	3.24ns	0.0981	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	591ns	0.227ns	0.878ns	0.00926	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	799ns	2.17ns	7.5ns	0.00893	656 B
master	`RunOnMethodBegin`	net472	1.04μs	0.368ns	1.42ns	0.0977	618 B
#4740	`RunOnMethodBegin`	net6.0	629ns	0.149ns	0.578ns	0.00917	656 B
#4740	`RunOnMethodBegin`	netcoreapp3.1	833ns	0.248ns	0.926ns	0.00877	656 B
#4740	`RunOnMethodBegin`	net472	1.07μs	1.04ns	3.88ns	0.098	618 B

andrewlock · 2023-10-20T12:48:56Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4740) (11.417M)   : 0, 11417072
    master (11.109M)   : 0, 11108605
    benchmarks/2.38.0 (11.867M)   : 0, 11866956
    benchmarks/2.9.0 (10.896M)   : 0, 10896496

    section Automatic
    This PR (4740) (7.964M)   : 0, 7963854
    master (7.750M)   : 0, 7750268
    benchmarks/2.38.0 (8.176M)   : 0, 8175636
    benchmarks/2.9.0 (8.125M)   : 0, 8125054

    section Trace stats
    This PR (4740) (8.275M)   : 0, 8274587
    master (8.109M)   : 0, 8108510
    benchmarks/2.38.0 (8.450M)   : 0, 8450370

    section Manual
    This PR (4740) (10.115M)   : 0, 10115184
    master (10.073M)   : 0, 10073279
    benchmarks/2.38.0 (10.334M)   : 0, 10334368

    section Manual + Automatic
    This PR (4740) (7.514M)   : 0, 7513500
    master (7.455M)   : 0, 7455042
    benchmarks/2.38.0 (7.750M)   : 0, 7750484

    section Version Conflict
    This PR (4740) (6.828M)   : 0, 6827966
    master (6.865M)   : 0, 6865195
    benchmarks/2.38.0 (7.137M)   : 0, 7136691

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4740) (9.395M)   : 0, 9394634
    master (9.590M)   : 0, 9589958
    benchmarks/2.38.0 (9.548M)   : 0, 9548121
    benchmarks/2.9.0 (9.477M)   : 0, 9477392

    section Automatic
    This PR (4740) (6.669M)   : 0, 6669152
    master (6.691M)   : 0, 6691177
    benchmarks/2.38.0 (6.747M)   : 0, 6747227

    section Trace stats
    This PR (4740) (6.874M)   : 0, 6873507
    master (7.023M)   : 0, 7022949
    benchmarks/2.38.0 (6.815M)   : 0, 6814846

    section Manual
    This PR (4740) (8.335M)   : 0, 8335065
    master (8.410M)   : 0, 8410093
    benchmarks/2.38.0 (8.263M)   : 0, 8263131

    section Manual + Automatic
    This PR (4740) (6.417M)   : 0, 6416863
    master (6.345M)   : 0, 6344795
    benchmarks/2.38.0 (6.275M)   : 0, 6275411

    section Version Conflict
    This PR (4740) (5.739M)   : 0, 5739167
    master (5.807M)   : 0, 5807175
    benchmarks/2.38.0 (5.670M)   : 0, 5669744

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4740) (9.497M)   : 0, 9497337
    master (9.631M)   : 0, 9630845
    benchmarks/2.38.0 (9.007M)   : 0, 9007066
    benchmarks/2.9.0 (9.459M)   : 0, 9458597

    section Automatic
    This PR (4740) (6.818M)   : 0, 6817539
    master (6.673M)   : 0, 6673444
    benchmarks/2.38.0 (6.423M)   : 0, 6422756
    benchmarks/2.9.0 (6.995M)   : 0, 6994572

    section Trace stats
    This PR (4740) (7.014M)   : 0, 7014080
    master (6.954M)   : 0, 6954012
    benchmarks/2.38.0 (6.721M)   : 0, 6720838

    section Manual
    This PR (4740) (8.367M)   : 0, 8366555
    master (8.234M)   : 0, 8233974
    benchmarks/2.38.0 (7.977M)   : 0, 7977494

    section Manual + Automatic
    This PR (4740) (6.478M)   : 0, 6478222
    master (6.479M)   : 0, 6479184
    benchmarks/2.38.0 (6.349M)   : 0, 6349066

    section Version Conflict
    This PR (4740) (6.058M)   : 0, 6058454
    master (5.898M)   : 0, 5898107
    benchmarks/2.38.0 (5.682M)   : 0, 5681654

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.474M)   : 0, 7473524
    benchmarks/2.38.0 (7.578M)   : 0, 7577544
    benchmarks/2.9.0 (7.897M)   : 0, 7897465

    section No attack
    master (2.191M)   : 0, 2191030
    benchmarks/2.38.0 (2.184M)   : 0, 2184279
    benchmarks/2.9.0 (3.208M)   : 0, 3207532

    section Attack
    master (1.738M)   : 0, 1737665
    benchmarks/2.38.0 (1.711M)   : 0, 1710542
    benchmarks/2.9.0 (2.553M)   : 0, 2553477

    section Blocking
    master (3.459M)   : 0, 3459059
    benchmarks/2.38.0 (3.486M)   : 0, 3486124

    section IAST default
    master (6.840M)   : 0, 6839830

    section IAST full
    master (6.232M)   : 0, 6231762

    section Base vuln
    master (0.967M)   : 0, 967380

    section IAST vuln
    master (0.924M)   : 0, 924177

kevingosse · 2023-10-23T11:20:58Z

tracer/src/Datadog.Trace/AppSec/Security.cs

                _configurationStatus.Actions = actions;
+                _configurationStatus.Actions = actions;


Just to be extra sure:

Suggested change

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

_configurationStatus.Actions = actions;

NachoEchevarria

LGTM. Very useful for ASM. Thank you!!!

andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc labels Oct 20, 2023

andrewlock requested review from a team as code owners October 20, 2023 10:04

andrewlock force-pushed the andrew/ci/fix_throughput_comparison branch from 16352f8 to e8f8deb Compare October 23, 2023 09:22

andrewlock force-pushed the andrew/ci/merge_throughput_tests branch from af8daac to b0898a2 Compare October 23, 2023 09:22

kevingosse approved these changes Oct 23, 2023

View reviewed changes

andrewlock force-pushed the andrew/ci/merge_throughput_tests branch from b0898a2 to 53d5a78 Compare October 23, 2023 11:22

NachoEchevarria approved these changes Oct 23, 2023

View reviewed changes

Base automatically changed from andrew/ci/fix_throughput_comparison to master October 23, 2023 14:30

andrewlock added 5 commits October 23, 2023 15:30

Merge throughput_appsec into throughput stage

1939e30

Fix dependencies usage (I hope)

7d0d4c1

Hackety Hackety grmbl

2b0fa44

Try fix condition

97057f8

Add missing generate_variables

2845b3a

andrewlock force-pushed the andrew/ci/merge_throughput_tests branch from 53d5a78 to 2845b3a Compare October 23, 2023 14:30

Don't fail the github check if the asm tests are skipped

717c963

andrewlock force-pushed the andrew/ci/merge_throughput_tests branch from 79a6e57 to 717c963 Compare October 24, 2023 08:46

andrewlock merged commit 0d0b9e6 into master Oct 24, 2023

andrewlock deleted the andrew/ci/merge_throughput_tests branch October 24, 2023 12:49

github-actions bot added this to the vNext milestone Oct 24, 2023

andrewlock mentioned this pull request Oct 27, 2023

Run always the Throughput-AppSec stage #4730

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge `throughput_appsec` into `throughput` stage #4740

Merge `throughput_appsec` into `throughput` stage #4740

andrewlock commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

datadog-ddstaging bot commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

Raw results

Raw results

Faster 🎉 in #4740

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4740

Faster 🎉 in #4740

Raw results

Raw results

andrewlock commented Oct 20, 2023 •

edited

Loading

kevingosse Oct 23, 2023

andrewlock Oct 23, 2023

NachoEchevarria left a comment

		_configurationStatus.Actions = actions;
		_configurationStatus.Actions = actions;

Merge throughput_appsec into throughput stage #4740

Merge throughput_appsec into throughput stage #4740

Conversation

andrewlock commented Oct 20, 2023 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Oct 20, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Oct 20, 2023 • edited Loading

Datadog Report

andrewlock commented Oct 20, 2023 • edited Loading

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Faster 🎉 in #4740

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4740

Faster 🎉 in #4740

Raw results

Raw results

andrewlock commented Oct 20, 2023 • edited Loading

Throughput/Crank Report:zap:

kevingosse Oct 23, 2023

Choose a reason for hiding this comment

andrewlock Oct 23, 2023

Choose a reason for hiding this comment

NachoEchevarria left a comment

Choose a reason for hiding this comment

Merge `throughput_appsec` into `throughput` stage #4740

Merge `throughput_appsec` into `throughput` stage #4740

andrewlock commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

datadog-ddstaging bot commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading

andrewlock commented Oct 20, 2023 •

edited

Loading