[CI Visibility] - Update Code Coverage percentage reporting #5032

tonyredondo · 2024-01-09T11:18:28Z

Summary of changes

This PR changes the scenarios were CI Visibility reports the code coverage percentage value according to the new spec.

Reason for change

The spec changed following the table:

datadog-ddstaging · 2024-01-09T11:39:38Z

Datadog Report

Branch report: tony/code-coverage-percentage-reporting-change
Commit report: 8d2c28f
Test service: dd-trace-dotnet

✅ 0 Failed, 303539 Passed, 1000 Skipped, 46m 37.87s Wall Time

andrewlock · 2024-01-09T11:55:21Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5032) - mean (71ms)  : 63, 80
     .   : milestone, 71,
    master - mean (71ms)  : 62, 80
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (5032) - mean (1,021ms)  : 1000, 1042
     .   : milestone, 1021,
    master - mean (1,023ms)  : 995, 1052
     .   : milestone, 1023,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5032) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (106ms)  : 102, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (5032) - mean (719ms)  : 693, 745
     .   : milestone, 719,
    master - mean (728ms)  : 704, 753
     .   : milestone, 728,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5032) - mean (90ms)  : 87, 93
     .   : milestone, 90,
    master - mean (90ms)  : 88, 93
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (5032) - mean (681ms)  : 652, 709
     .   : milestone, 681,
    master - mean (688ms)  : 667, 708
     .   : milestone, 688,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5032) - mean (189ms)  : 186, 191
     .   : milestone, 189,
    master - mean (189ms)  : 186, 193
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (5032) - mean (1,130ms)  : 1105, 1154
     .   : milestone, 1130,
    master - mean (1,132ms)  : 1110, 1153
     .   : milestone, 1132,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5032) - mean (274ms)  : 268, 281
     .   : milestone, 274,
    master - mean (273ms)  : 269, 277
     .   : milestone, 273,

    section CallTarget+Inlining+NGEN
    This PR (5032) - mean (1,089ms)  : 1062, 1116
     .   : milestone, 1089,
    master - mean (1,081ms)  : 1057, 1104
     .   : milestone, 1081,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5032) - mean (263ms)  : 259, 268
     .   : milestone, 263,
    master - mean (263ms)  : 259, 267
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (5032) - mean (1,052ms)  : 1027, 1077
     .   : milestone, 1052,
    master - mean (1,054ms)  : 1028, 1080
     .   : milestone, 1054,

andrewlock

LGTM with one question

andrewlock · 2024-01-09T12:40:49Z

tracer/src/Datadog.Trace.Tools.Runner/RunCiCommand.cs

-                            // Adds the global code coverage percentage to the session
-                            session.SetTag(CodeCoverageTags.PercentageOfTotalLines, globalCoverage.GetTotalPercentage());
+                            // We only report the code coverage percentage if the customer manually sets the 'DD_CIVISIBILITY_CODE_COVERAGE_ENABLED' environment variable according to the new spec.
+                            if (EnvironmentHelpers.GetEnvironmentVariable(Configuration.ConfigurationKeys.CIVisibility.CodeCoverage).ToBoolean() == true)


Do we explicitly want to check the env var like this, or should we be checking CIVisibilitySettings.CodeCoverageEnabled 🤔 I know the spec says env var, but is that definitely what we want given we can have other sources too?

Currently CIVisibilitySettings.CodeCoverageEnabled gets modified with the value that comes from the settings api, so in some cases the value can be different from the actual environment variable. I might introduce an inmutable settings in the future.

andrewlock · 2024-01-09T13:55:23Z

Benchmarks Report 🐌

Benchmarks for #5032 compared to master:

1 benchmarks are slower, with geometric mean 1.166
2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.85μs	48.6ns	295ns	0.0214	0.00857	0	7.48 KB
master	`StartStopWithChild`	netcoreapp3.1	10.9μs	57.3ns	309ns	0.0273	0.0109	0	7.57 KB
master	`StartStopWithChild`	net472	16.9μs	52.7ns	204ns	1.36	0.375	0.128	7.96 KB
#5032	`StartStopWithChild`	net6.0	8.78μs	49.3ns	323ns	0.0276	0.0138	0	7.49 KB
#5032	`StartStopWithChild`	netcoreapp3.1	11μs	39.7ns	149ns	0.0166	0.0111	0	7.58 KB
#5032	`StartStopWithChild`	net472	17.1μs	51.5ns	200ns	1.35	0.372	0.127	7.95 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	456μs	226ns	847ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	636μs	120ns	464ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	784μs	368ns	1.38μs	0.391	3.3 KB
#5032	`WriteAndFlushEnrichedTraces`	net6.0	459μs	365ns	1.41μs	0	2.7 KB
#5032	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	278ns	1.04μs	0	2.7 KB
#5032	`WriteAndFlushEnrichedTraces`	net472	788μs	243ns	939ns	0.393	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	38.9μs	25.7ns	99.4ns	0.0194	0	1.77 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	41.6μs	19.9ns	71.8ns	0.0207	0	1.74 KB
master	`AllCycleSimpleBody`	net472	44.9μs	16.9ns	65.4ns	0.287	0	1.81 KB
master	`AllCycleMoreComplexBody`	net6.0	200μs	147ns	569ns	0.0994	0	9.25 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	211μs	223ns	864ns	0.105	0	9.14 KB
master	`AllCycleMoreComplexBody`	net472	225μs	94ns	352ns	1.46	0	9.32 KB
master	`ObjectExtractorSimpleBody`	net6.0	135ns	0.0635ns	0.237ns	0.00391	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	249ns	0.171ns	0.662ns	0.00362	0	272 B
master	`ObjectExtractorSimpleBody`	net472	164ns	0.121ns	0.454ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.01μs	1.09ns	4.08ns	0.0536	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.06μs	2.26ns	8.77ns	0.0508	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.73μs	2.18ns	8.44ns	0.603	0.0056	3.8 KB
#5032	`AllCycleSimpleBody`	net6.0	38.8μs	21ns	81.5ns	0.0193	0	1.77 KB
#5032	`AllCycleSimpleBody`	netcoreapp3.1	42μs	78.3ns	303ns	0.0208	0	1.74 KB
#5032	`AllCycleSimpleBody`	net472	44.4μs	9.13ns	32.9ns	0.288	0	1.81 KB
#5032	`AllCycleMoreComplexBody`	net6.0	201μs	69.5ns	269ns	0.0995	0	9.25 KB
#5032	`AllCycleMoreComplexBody`	netcoreapp3.1	213μs	381ns	1.48μs	0.107	0	9.14 KB
#5032	`AllCycleMoreComplexBody`	net472	226μs	158ns	610ns	1.46	0	9.32 KB
#5032	`ObjectExtractorSimpleBody`	net6.0	137ns	0.0602ns	0.225ns	0.00395	0	280 B
#5032	`ObjectExtractorSimpleBody`	netcoreapp3.1	235ns	0.214ns	0.8ns	0.00365	0	272 B
#5032	`ObjectExtractorSimpleBody`	net472	170ns	0.0583ns	0.226ns	0.0446	0	281 B
#5032	`ObjectExtractorMoreComplexBody`	net6.0	2.91μs	1.67ns	6.26ns	0.0537	0	3.78 KB
#5032	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.9μs	1.02ns	3.81ns	0.0489	0	3.69 KB
#5032	`ObjectExtractorMoreComplexBody`	net472	4.12μs	2.26ns	8.46ns	0.602	0.00621	3.8 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	50.9μs	116ns	435ns	0.228	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	70.3μs	390ns	2.56μs	0.22	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	net472	96.5μs	43.8ns	164ns	2.54	0.0959	16.14 KB
master	`RunWafTwice(args=NestedMap (10))`	net6.0	53.7μs	118ns	486ns	0.211	0	16.6 KB
master	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	71.1μs	185ns	783ns	0.225	0	16.58 KB
master	`RunWafTwice(args=NestedMap (10))`	net472	108μs	243ns	940ns	2.65	0.104	16.69 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	108μs	35ns	135ns	0.27	0	22.41 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	131μs	615ns	2.54μs	0.272	0	22.36 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	164μs	117ns	454ns	3.53	0.164	22.7 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	109μs	618ns	4.33μs	0.444	0	32.76 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	135μs	675ns	3.16μs	0.399	0	33.33 KB
master	`RunWaf(args=NestedMap (100))`	net472	194μs	633ns	2.45μs	5.29	0.371	33.67 KB
master	`RunWafTwice(args=NestedMap (100))`	net6.0	111μs	401ns	1.55μs	0.458	0	33.3 KB
master	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	139μs	723ns	3.68μs	0.413	0	33.86 KB
master	`RunWafTwice(args=NestedMap (100))`	net472	194μs	80ns	310ns	5.43	0.388	34.23 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	156μs	41.4ns	160ns	0.548	0	39.1 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	198μs	156ns	603ns	0.494	0	39.63 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	254μs	125ns	469ns	6.34	0.507	40.23 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	103μs	556ns	3.15μs	0.441	0	32.18 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	132μs	718ns	4.18μs	0.447	0	32.3 KB
master	`RunWaf(args=NestedMap (20))`	net472	189μs	963ns	4.2μs	5.18	0.37	32.63 KB
master	`RunWafTwice(args=NestedMap (20))`	net6.0	106μs	620ns	5.58μs	0.457	0	32.72 KB
master	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	141μs	720ns	3.3μs	0.402	0	32.82 KB
master	`RunWafTwice(args=NestedMap (20))`	net472	197μs	810ns	3.14μs	5.24	0.388	33.19 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	156μs	129ns	447ns	0.543	0	38.53 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	198μs	803ns	3.11μs	0.489	0	38.6 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	260μs	1.27μs	5.08μs	6.13	0.522	39.2 KB
#5032	`RunWaf(args=NestedMap (10))`	net6.0	51.4μs	18.7ns	70.1ns	0.221	0	16.06 KB
#5032	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	69.5μs	375ns	2.06μs	0.199	0	16.06 KB
#5032	`RunWaf(args=NestedMap (10))`	net472	100μs	55.1ns	213ns	2.55	0.0962	16.14 KB
#5032	`RunWafTwice(args=NestedMap (10))`	net6.0	57.9μs	52.1ns	195ns	0.231	0	16.6 KB
#5032	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	72.9μs	399ns	2.43μs	0.209	0	16.58 KB
#5032	`RunWafTwice(args=NestedMap (10))`	net472	104μs	58.8ns	220ns	2.63	0.103	16.69 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	115μs	599ns	3μs	0.283	0	22.41 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	131μs	693ns	3.47μs	0.313	0	22.36 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	165μs	114ns	442ns	3.59	0.16	22.7 KB
#5032	`RunWaf(args=NestedMap (100))`	net6.0	106μs	485ns	1.94μs	0.462	0	32.76 KB
#5032	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	135μs	128ns	496ns	0.404	0	33.33 KB
#5032	`RunWaf(args=NestedMap (100))`	net472	187μs	135ns	522ns	5.33	0.374	33.67 KB
#5032	`RunWafTwice(args=NestedMap (100))`	net6.0	107μs	626ns	5.97μs	0.448	0	33.3 KB
#5032	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	137μs	628ns	2.43μs	0.395	0	33.86 KB
#5032	`RunWafTwice(args=NestedMap (100))`	net472	194μs	139ns	538ns	5.4	0.386	34.22 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	166μs	58.2ns	218ns	0.497	0	39.1 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	199μs	1.05μs	5.15μs	0.508	0	39.63 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	256μs	555ns	2μs	6.29	0.513	40.23 KB
#5032	`RunWaf(args=NestedMap (20))`	net6.0	107μs	127ns	491ns	0.438	0	32.18 KB
#5032	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	132μs	751ns	5.47μs	0.381	0	32.3 KB
#5032	`RunWaf(args=NestedMap (20))`	net472	184μs	97.8ns	379ns	5.14	0.367	32.63 KB
#5032	`RunWafTwice(args=NestedMap (20))`	net6.0	107μs	120ns	466ns	0.478	0	32.72 KB
#5032	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	139μs	712ns	3.18μs	0.394	0	32.82 KB
#5032	`RunWafTwice(args=NestedMap (20))`	net472	200μs	96ns	372ns	5.27	0.383	33.19 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	156μs	37.4ns	140ns	0.505	0	38.53 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	198μs	678ns	2.63μs	0.491	0	38.6 KB
#5032	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	255μs	276ns	1.07μs	6.21	0.507	39.2 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	170μs	132ns	511ns	0.17	18.25 KB
master	`SendRequest`	netcoreapp3.1	193μs	247ns	925ns	0.193	20.41 KB
master	`SendRequest`	net472	0.00135ns	0.000386ns	0.0015ns	0	0 b
#5032	`SendRequest`	net6.0	173μs	195ns	757ns	0.172	18.25 KB
#5032	`SendRequest`	netcoreapp3.1	192μs	245ns	948ns	0.193	20.41 KB
#5032	`SendRequest`	net472	0ns	0ns	0ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5032

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.51 KB	41.9 KB	390 B	0.94%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	538μs	2.33μs	8.4μs	0.532	0	0	41.51 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	668μs	1.48μs	5.73μs	0.334	0	0	41.69 KB
master	`WriteAndFlushEnrichedTraces`	net472	820μs	3.95μs	16.8μs	8.39	2.52	0.419	53.23 KB
#5032	`WriteAndFlushEnrichedTraces`	net6.0	555μs	1.68μs	6.5μs	0.558	0	0	41.9 KB
#5032	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	665μs	1.09μs	4.09μs	0.327	0	0	41.87 KB
#5032	`WriteAndFlushEnrichedTraces`	net472	882μs	4.16μs	17.6μs	8.3	2.62	0.437	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.08μs	0.671ns	2.6ns	0.0108	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.48μs	0.812ns	3.14ns	0.0105	768 B
master	`ExecuteNonQuery`	net472	1.8μs	0.784ns	3.03ns	0.116	730 B
#5032	`ExecuteNonQuery`	net6.0	1.12μs	0.885ns	3.43ns	0.011	768 B
#5032	`ExecuteNonQuery`	netcoreapp3.1	1.56μs	1.1ns	4.27ns	0.0101	768 B
#5032	`ExecuteNonQuery`	net472	1.82μs	0.638ns	2.39ns	0.115	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.19μs	0.53ns	1.98ns	0.0131	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	1.6ns	6ns	0.0129	936 B
master	`CallElasticsearch`	net472	2.47μs	0.425ns	1.59ns	0.151	955 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	1.99ns	7.45ns	0.013	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	1.24ns	4.81ns	0.013	984 B
master	`CallElasticsearchAsync`	net472	2.74μs	1.03ns	4ns	0.16	1.01 KB
#5032	`CallElasticsearch`	net6.0	1.18μs	0.289ns	1.08ns	0.0131	936 B
#5032	`CallElasticsearch`	netcoreapp3.1	1.55μs	0.496ns	1.86ns	0.0124	936 B
#5032	`CallElasticsearch`	net472	2.61μs	0.904ns	3.38ns	0.151	955 B
#5032	`CallElasticsearchAsync`	net6.0	1.27μs	0.41ns	1.59ns	0.0129	912 B
#5032	`CallElasticsearchAsync`	netcoreapp3.1	1.57μs	0.777ns	3.01ns	0.0133	984 B
#5032	`CallElasticsearchAsync`	net472	2.74μs	0.594ns	2.3ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.34μs	1.17ns	4.53ns	0.013	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	0.641ns	2.48ns	0.0119	912 B
master	`ExecuteAsync`	net472	1.72μs	1.61ns	6.25ns	0.139	875 B
#5032	`ExecuteAsync`	net6.0	1.31μs	0.827ns	3.09ns	0.0131	912 B
#5032	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.651ns	2.43ns	0.012	912 B
#5032	`ExecuteAsync`	net472	1.84μs	0.905ns	3.5ns	0.139	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.17μs	2.57ns	9.95ns	0.0288	2.1 KB
master	`SendAsync`	netcoreapp3.1	4.94μs	2.04ns	7.9ns	0.0347	2.63 KB
master	`SendAsync`	net472	7.62μs	4.02ns	15.6ns	0.522	3.31 KB
#5032	`SendAsync`	net6.0	3.99μs	1.3ns	5.03ns	0.0303	2.1 KB
#5032	`SendAsync`	netcoreapp3.1	5.16μs	10.9ns	42.3ns	0.0336	2.63 KB
#5032	`SendAsync`	net472	7.81μs	3.24ns	12.6ns	0.522	3.31 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5032

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472	58.52 KB	60.13 KB	1.61 KB	2.75%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net6.0	55μs	283ns	1.3μs	43.44 KB
master	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	53μs	201ns	752ns	42.64 KB
master	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net472	37.6μs	97.7ns	366ns	58.52 KB
master	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net6.0	65μs	250ns	936ns	43.29 KB
master	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	101μs	3.88μs	38.4μs	42.64 KB
master	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net472	63.5μs	253ns	912ns	57.34 KB
#5032	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net6.0	60.4μs	725ns	7.25μs	43.44 KB
#5032	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	52.7μs	197ns	738ns	42.64 KB
#5032	`StringConcatBenchmark(parameters=Syste(...)ring] [48])`	net472	37.6μs	73ns	273ns	60.13 KB
#5032	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net6.0	89.6μs	2.45μs	24.4μs	43.29 KB
#5032	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	netcoreapp3.1	97.2μs	3.65μs	36.1μs	42.64 KB
#5032	`StringConcatAspectBenchmark(parameters=Syste(...)ring] [48])`	net472	63.4μs	347ns	1.96μs	57.34 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.44μs	0.494ns	1.91ns	0.0223	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	2.33μs	2.12ns	7.94ns	0.0209	1.57 KB
master	`EnrichedLog`	net472	2.52μs	3.65ns	14.1ns	0.238	1.5 KB
#5032	`EnrichedLog`	net6.0	1.46μs	0.664ns	2.48ns	0.0222	1.57 KB
#5032	`EnrichedLog`	netcoreapp3.1	2.19μs	1.49ns	5.37ns	0.0209	1.57 KB
#5032	`EnrichedLog`	net472	2.61μs	2.57ns	9.97ns	0.237	1.5 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	124ns	463ns	0	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	196ns	757ns	0.0602	0	4.21 KB
master	`EnrichedLog`	net472	150μs	134ns	517ns	0.671	0.224	4.39 KB
#5032	`EnrichedLog`	net6.0	115μs	227ns	879ns	0	0	4.21 KB
#5032	`EnrichedLog`	netcoreapp3.1	121μs	315ns	1.22μs	0.0606	0	4.21 KB
#5032	`EnrichedLog`	net472	148μs	139ns	540ns	0.665	0.222	4.39 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.14μs	1.26ns	4.88ns	0.0294	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	4.13μs	1.92ns	7.43ns	0.0289	2.13 KB
master	`EnrichedLog`	net472	4.91μs	2.43ns	9.41ns	0.308	1.95 KB
#5032	`EnrichedLog`	net6.0	3.04μs	1ns	3.88ns	0.0289	2.13 KB
#5032	`EnrichedLog`	netcoreapp3.1	4.31μs	1.92ns	7.44ns	0.0281	2.13 KB
#5032	`EnrichedLog`	net472	4.89μs	1.96ns	7.61ns	0.309	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.4μs	0.755ns	2.92ns	0.0154	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	0.822ns	3.18ns	0.0149	1.1 KB
master	`SendReceive`	net472	2.13μs	3.25ns	12.6ns	0.177	1.12 KB
#5032	`SendReceive`	net6.0	1.41μs	0.532ns	2.06ns	0.0154	1.1 KB
#5032	`SendReceive`	netcoreapp3.1	1.83μs	0.892ns	3.34ns	0.0147	1.1 KB
#5032	`SendReceive`	net472	2.15μs	6.75ns	26.1ns	0.177	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.67μs	1.12ns	4.35ns	0.0214	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.81μs	1.02ns	3.97ns	0.0207	1.58 KB
master	`EnrichedLog`	net472	4.29μs	0.997ns	3.59ns	0.311	1.97 KB
#5032	`EnrichedLog`	net6.0	2.77μs	0.79ns	2.96ns	0.0208	1.53 KB
#5032	`EnrichedLog`	netcoreapp3.1	4.01μs	1.52ns	5.88ns	0.02	1.58 KB
#5032	`EnrichedLog`	net472	4.43μs	2.63ns	10.2ns	0.31	1.97 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5032

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.166	523.39	610.30

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	456ns	0.347ns	1.3ns	0.00759	536 B
master	`StartFinishSpan`	netcoreapp3.1	712ns	0.272ns	1.05ns	0.0072	536 B
master	`StartFinishSpan`	net472	766ns	0.384ns	1.49ns	0.0852	538 B
master	`StartFinishScope`	net6.0	524ns	0.268ns	1ns	0.00918	656 B
master	`StartFinishScope`	netcoreapp3.1	852ns	0.461ns	1.73ns	0.00892	656 B
master	`StartFinishScope`	net472	964ns	1.79ns	6.93ns	0.0979	618 B
#5032	`StartFinishSpan`	net6.0	453ns	0.194ns	0.725ns	0.00757	536 B
#5032	`StartFinishSpan`	netcoreapp3.1	690ns	0.278ns	1.04ns	0.0072	536 B
#5032	`StartFinishSpan`	net472	779ns	0.498ns	1.93ns	0.0854	538 B
#5032	`StartFinishScope`	net6.0	609ns	0.895ns	3.47ns	0.00918	656 B
#5032	`StartFinishScope`	netcoreapp3.1	942ns	0.278ns	1.07ns	0.00881	656 B
#5032	`StartFinishScope`	net472	1.04μs	1.11ns	4.31ns	0.0978	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	574ns	0.142ns	0.548ns	0.0094	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	988ns	1.92ns	7.44ns	0.0088	656 B
master	`RunOnMethodBegin`	net472	1.1μs	0.99ns	3.83ns	0.098	618 B
#5032	`RunOnMethodBegin`	net6.0	635ns	0.151ns	0.563ns	0.00917	656 B
#5032	`RunOnMethodBegin`	netcoreapp3.1	927ns	0.616ns	2.39ns	0.00853	656 B
#5032	`RunOnMethodBegin`	net472	1.18μs	0.404ns	1.56ns	0.0981	618 B

andrewlock · 2024-01-09T16:58:45Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5032) (11.292M)   : 0, 11292422
    master (11.753M)   : 0, 11753262
    benchmarks/2.9.0 (11.691M)   : 0, 11690725

    section Automatic
    This PR (5032) (7.552M)   : crit ,0, 7551755
    master (7.966M)   : 0, 7965560
    benchmarks/2.9.0 (8.196M)   : 0, 8195550

    section Trace stats
    This PR (5032) (7.963M)   : 0, 7963062
    master (8.343M)   : 0, 8343276

    section Manual
    This PR (5032) (9.475M)   : crit ,0, 9475131
    master (10.388M)   : 0, 10388041

    section Manual + Automatic
    This PR (5032) (7.232M)   : 0, 7231736
    master (7.588M)   : 0, 7587504

    section Version Conflict
    This PR (5032) (6.415M)   : crit ,0, 6415050
    master (6.966M)   : 0, 6965765

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5032) (9.433M)   : 0, 9433080
    master (9.242M)   : 0, 9241504
    benchmarks/2.9.0 (9.533M)   : 0, 9533360

    section Automatic
    This PR (5032) (6.470M)   : 0, 6470044
    master (6.483M)   : 0, 6483329

    section Trace stats
    This PR (5032) (6.976M)   : 0, 6975775
    master (6.779M)   : 0, 6778543

    section Manual
    This PR (5032) (8.329M)   : 0, 8329442
    master (8.348M)   : 0, 8347735

    section Manual + Automatic
    This PR (5032) (6.253M)   : 0, 6252993
    master (6.187M)   : 0, 6186728

    section Version Conflict
    This PR (5032) (5.699M)   : 0, 5699456
    master (5.682M)   : 0, 5681803

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5032) (10.434M)   : 0, 10434289
    master (10.602M)   : 0, 10601787
    benchmarks/2.9.0 (10.006M)   : 0, 10005933

    section Automatic
    This PR (5032) (7.402M)   : 0, 7402308
    master (7.375M)   : 0, 7375201
    benchmarks/2.9.0 (7.242M)   : 0, 7242466

    section Trace stats
    This PR (5032) (7.355M)   : 0, 7354772
    master (7.665M)   : 0, 7665372

    section Manual
    This PR (5032) (8.895M)   : 0, 8894740
    master (8.718M)   : 0, 8717600

    section Manual + Automatic
    This PR (5032) (7.075M)   : 0, 7075141
    master (7.040M)   : 0, 7040484

    section Version Conflict
    This PR (5032) (6.336M)   : 0, 6336363
    master (6.300M)   : 0, 6299859

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.358M)   : 0, 7358395
    benchmarks/2.9.0 (7.870M)   : 0, 7869823

    section No attack
    master (1.765M)   : 0, 1764782
    benchmarks/2.9.0 (3.239M)   : 0, 3239486

    section Attack
    master (1.411M)   : 0, 1411188
    benchmarks/2.9.0 (2.556M)   : 0, 2556231

    section Blocking
    master (3.099M)   : 0, 3099104

    section IAST default
    master (6.495M)   : 0, 6495419

    section IAST full
    master (5.855M)   : 0, 5855342

    section Base vuln
    master (0.970M)   : 0, 969506

    section IAST vuln
    master (0.870M)   : 0, 870485

Update Code Coverage percentage reporting

8d2c28f

tonyredondo self-assigned this Jan 9, 2024

github-actions bot added the area:ci-visibility label Jan 9, 2024

tonyredondo marked this pull request as ready for review January 9, 2024 12:21

tonyredondo requested review from a team as code owners January 9, 2024 12:21

juan-fernandez approved these changes Jan 9, 2024

View reviewed changes

andrewlock approved these changes Jan 9, 2024

View reviewed changes

tonyredondo merged commit 93d9aca into master Jan 10, 2024
55 checks passed

tonyredondo deleted the tony/code-coverage-percentage-reporting-change branch January 10, 2024 13:04

github-actions bot added this to the vNext milestone Jan 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI Visibility] - Update Code Coverage percentage reporting #5032

[CI Visibility] - Update Code Coverage percentage reporting #5032

tonyredondo commented Jan 9, 2024 •

edited

Loading

datadog-ddstaging bot commented Jan 9, 2024 •

edited

Loading

andrewlock commented Jan 9, 2024

andrewlock left a comment

andrewlock Jan 9, 2024

tonyredondo Jan 9, 2024 •

edited

Loading

andrewlock commented Jan 9, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5032

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5032

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5032

Raw results

Raw results

andrewlock commented Jan 9, 2024

[CI Visibility] - Update Code Coverage percentage reporting #5032

[CI Visibility] - Update Code Coverage percentage reporting #5032

Conversation

tonyredondo commented Jan 9, 2024 • edited Loading

Summary of changes

Reason for change

datadog-ddstaging bot commented Jan 9, 2024 • edited Loading

Datadog Report

andrewlock commented Jan 9, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock left a comment

Choose a reason for hiding this comment

andrewlock Jan 9, 2024

Choose a reason for hiding this comment

tonyredondo Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

andrewlock commented Jan 9, 2024

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5032

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5032

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5032

Raw results

Raw results

andrewlock commented Jan 9, 2024

Throughput/Crank Report:zap:

tonyredondo commented Jan 9, 2024 •

edited

Loading

datadog-ddstaging bot commented Jan 9, 2024 •

edited

Loading

tonyredondo Jan 9, 2024 •

edited

Loading