Fix telemetry metrics and add validator #4543

andrewlock · 2023-08-23T15:08:21Z

Summary of changes

Rename direct_log_queue -> direct_log_queue.length
Rename direct_log_api.errors.responses -> direct_log_api.errors
Add tests so it doesn't happen again

Reason for change

We were sending the wrong values in metrics, but they're not always emitted, so we missed them.

Implementation details

Two one-line changes + source generators to fix the bug.

Test coverage

Added unit tests that compare the JSON metrics defined in the intake against our list of metrics. Confirmed that this test would have caught the bugs

Note that the JSON is just copy-pasted for simplicity, so it will have a tendency to drift, but I don't expect this to be a big issue in practice, as metrics should just be added to the intake, not removed.

datadog-ddstaging · 2023-08-23T15:29:37Z

Datadog Report

Branch report: andrew/telemetry/fix-metric-name
Commit report: 05be8ad

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 298978 Passed, 1115 Skipped, 22m 18.19s Wall Time

andrewlock · 2023-08-23T15:30:35Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4543) - mean (70ms)  : 62, 78
     .   : milestone, 70,
    master - mean (69ms)  : 64, 74
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (4543) - mean (1,045ms)  : 1013, 1078
     .   : milestone, 1045,
    master - mean (1,033ms)  : 1004, 1062
     .   : milestone, 1033,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4543) - mean (106ms)  : 104, 108
     .   : milestone, 106,
    master - mean (105ms)  : 100, 111
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (4543) - mean (733ms)  : 695, 771
     .   : milestone, 733,
    master - mean (733ms)  : 700, 766
     .   : milestone, 733,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4543) - mean (90ms)  : 88, 92
     .   : milestone, 90,
    master - mean (90ms)  : 83, 97
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4543) - mean (693ms)  : 670, 717
     .   : milestone, 693,
    master - mean (689ms)  : 666, 711
     .   : milestone, 689,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4543) - mean (190ms)  : 185, 195
     .   : milestone, 190,
    master - mean (188ms)  : 183, 193
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4543) - mean (1,111ms)  : 1078, 1143
     .   : milestone, 1111,
    master - mean (1,127ms)  : 1087, 1168
     .   : milestone, 1127,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4543) - mean (269ms)  : 262, 277
     .   : milestone, 269,
    master - mean (271ms)  : 263, 278
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4543) - mean (1,075ms)  : 1038, 1112
     .   : milestone, 1075,
    master - mean (1,077ms)  : 1049, 1105
     .   : milestone, 1077,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4543) - mean (260ms)  : 253, 266
     .   : milestone, 260,
    master - mean (257ms)  : 249, 266
     .   : milestone, 257,

    section CallTarget+Inlining+NGEN
    This PR (4543) - mean (1,034ms)  : 997, 1072
     .   : milestone, 1034,
    master - mean (1,033ms)  : 994, 1072
     .   : milestone, 1033,

andrewlock · 2023-08-23T15:44:41Z

Benchmarks Report 🐌

Benchmarks for #4543 compared to master:

2 benchmarks are faster, with geometric mean 1.243
1 benchmarks are slower, with geometric mean 1.141
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	481μs	610ns	2.11μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	640μs	123ns	478ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	794μs	462ns	1.79μs	0.396	3.3 KB
#4543	`WriteAndFlushEnrichedTraces`	net6.0	469μs	516ns	2μs	0	2.7 KB
#4543	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	630μs	330ns	1.23μs	0	2.7 KB
#4543	`WriteAndFlushEnrichedTraces`	net472	801μs	277ns	1.07μs	0.398	3.3 KB

Benchmarks.Trace.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4543

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0	1.382	171.32	124.00

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	39.5μs	27.6ns	103ns	0	0	1.69 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	42.1μs	50.2ns	181ns	0.0207	0	1.67 KB
master	`AllCycleSimpleBody`	net472	43.8μs	13.4ns	50.1ns	0.258	0	1.73 KB
master	`AllCycleMoreComplexBody`	net6.0	231μs	204ns	790ns	0.116	0	9.26 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	240μs	383ns	1.48μs	0.119	0	9.16 KB
master	`AllCycleMoreComplexBody`	net472	249μs	106ns	409ns	1.37	0	9.33 KB
master	`ObjectExtractorSimpleBody`	net6.0	171ns	0.0742ns	0.277ns	0.00395	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	176ns	0.0938ns	0.351ns	0.00373	0	272 B
master	`ObjectExtractorSimpleBody`	net472	146ns	0.338ns	1.22ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.2μs	1.62ns	5.84ns	0.0543	0	3.88 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.1μs	1.9ns	7.1ns	0.0512	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.13μs	4.11ns	15.9ns	0.617	0.00624	3.89 KB
#4543	`AllCycleSimpleBody`	net6.0	39.2μs	11.1ns	41.6ns	0.0195	0	1.69 KB
#4543	`AllCycleSimpleBody`	netcoreapp3.1	41.8μs	58.8ns	204ns	0.0208	0	1.67 KB
#4543	`AllCycleSimpleBody`	net472	43.8μs	70.6ns	273ns	0.259	0	1.73 KB
#4543	`AllCycleMoreComplexBody`	net6.0	232μs	55.1ns	214ns	0.115	0	9.26 KB
#4543	`AllCycleMoreComplexBody`	netcoreapp3.1	241μs	341ns	1.28μs	0.12	0	9.16 KB
#4543	`AllCycleMoreComplexBody`	net472	250μs	72.2ns	270ns	1.38	0	9.33 KB
#4543	`ObjectExtractorSimpleBody`	net6.0	124ns	0.0428ns	0.16ns	0.00395	0	280 B
#4543	`ObjectExtractorSimpleBody`	netcoreapp3.1	174ns	0.0836ns	0.324ns	0.0037	0	272 B
#4543	`ObjectExtractorSimpleBody`	net472	145ns	0.101ns	0.392ns	0.0446	0	281 B
#4543	`ObjectExtractorMoreComplexBody`	net6.0	3.25μs	0.882ns	3.3ns	0.0537	0	3.88 KB
#4543	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.09μs	1.19ns	4.3ns	0.0511	0	3.78 KB
#4543	`ObjectExtractorMoreComplexBody`	net472	4.17μs	2.22ns	8.62ns	0.618	0.00633	3.89 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	168μs	146ns	565ns	0.251	18.08 KB
master	`SendRequest`	netcoreapp3.1	192μs	306ns	1.15μs	0.191	20.14 KB
master	`SendRequest`	net472	0.000947ns	0.000235ns	0.00088ns	0	0 b
#4543	`SendRequest`	net6.0	169μs	107ns	416ns	0.169	18.08 KB
#4543	`SendRequest`	netcoreapp3.1	189μs	297ns	1.15μs	0.188	20.14 KB
#4543	`SendRequest`	net472	0.000824ns	0.000268ns	0.00104ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #4543

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.63 KB	41.89 KB	265 B	0.64%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	533μs	218ns	815ns	0.519	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	633μs	1.6μs	6.2μs	0.317	0	0	41.63 KB
master	`WriteAndFlushEnrichedTraces`	net472	801μs	2.38μs	9.21μs	8.31	2.37	0.396	53.23 KB
#4543	`WriteAndFlushEnrichedTraces`	net6.0	523μs	1.17μs	4.53μs	0.525	0	0	41.78 KB
#4543	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	632μs	2.56μs	9.93μs	0.314	0	0	41.89 KB
#4543	`WriteAndFlushEnrichedTraces`	net472	788μs	3.12μs	12.1μs	8.36	2.39	0.398	53.22 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.15μs	0.36ns	1.4ns	0.0106	0	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.43μs	0.615ns	2.38ns	0.01	0	768 B
master	`ExecuteNonQuery`	net472	1.64μs	0.576ns	2.23ns	0.116	0.000823	730 B
#4543	`ExecuteNonQuery`	net6.0	1.1μs	0.479ns	1.79ns	0.0108	0	768 B
#4543	`ExecuteNonQuery`	netcoreapp3.1	1.34μs	0.528ns	2.05ns	0.0103	0	768 B
#4543	`ExecuteNonQuery`	net472	1.67μs	0.685ns	2.47ns	0.115	0	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.19μs	0.547ns	1.97ns	0.0138	0	992 B
master	`CallElasticsearch`	netcoreapp3.1	1.44μs	0.461ns	1.72ns	0.013	0	992 B
master	`CallElasticsearch`	net472	2.26μs	0.519ns	1.94ns	0.159	0.00113	1 KB
master	`CallElasticsearchAsync`	net6.0	1.28μs	0.917ns	3.55ns	0.0134	0	968 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.57μs	0.895ns	3.47ns	0.0142	0	1.04 KB
master	`CallElasticsearchAsync`	net472	2.58μs	0.914ns	3.42ns	0.168	0.00129	1.06 KB
#4543	`CallElasticsearch`	net6.0	1.15μs	0.388ns	1.45ns	0.0138	0	992 B
#4543	`CallElasticsearch`	netcoreapp3.1	1.44μs	0.86ns	3.22ns	0.013	0	992 B
#4543	`CallElasticsearch`	net472	2.35μs	0.459ns	1.78ns	0.159	0.00117	1 KB
#4543	`CallElasticsearchAsync`	net6.0	1.29μs	0.571ns	2.21ns	0.0135	0	968 B
#4543	`CallElasticsearchAsync`	netcoreapp3.1	1.51μs	0.979ns	3.79ns	0.0137	0	1.04 KB
#4543	`CallElasticsearchAsync`	net472	2.5μs	0.643ns	2.49ns	0.168	0.00125	1.06 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net6.0	1.24μs	0.606ns	2.35ns	0.0129	0	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.49μs	3.26ns	12.6ns	0.0118	0	912 B
master	`ExecuteAsync`	net472	1.72μs	1.8ns	6.98ns	0.138	0.000849	875 B
#4543	`ExecuteAsync`	net6.0	1.2μs	0.378ns	1.36ns	0.0126	0	912 B
#4543	`ExecuteAsync`	netcoreapp3.1	1.43μs	0.842ns	3.26ns	0.0125	0	912 B
#4543	`ExecuteAsync`	net472	1.73μs	1.57ns	6.08ns	0.138	0.000864	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.01μs	1.58ns	5.71ns	0.026	1.94 KB
master	`SendAsync`	netcoreapp3.1	4.54μs	1.34ns	5.02ns	0.0333	2.48 KB
master	`SendAsync`	net472	7.05μs	2.6ns	10.1ns	0.483	3.05 KB
#4543	`SendAsync`	net6.0	3.74μs	1.32ns	4.96ns	0.0262	1.94 KB
#4543	`SendAsync`	netcoreapp3.1	4.55μs	1.88ns	7.05ns	0.0317	2.48 KB
#4543	`SendAsync`	net472	7.09μs	2.34ns	8.75ns	0.482	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.37μs	0.536ns	2.01ns	0.0226	1.62 KB
master	`EnrichedLog`	netcoreapp3.1	2.01μs	0.929ns	3.48ns	0.0221	1.62 KB
master	`EnrichedLog`	net472	2.35μs	1.25ns	4.84ns	0.245	1.54 KB
#4543	`EnrichedLog`	net6.0	1.3μs	0.888ns	3.32ns	0.0229	1.62 KB
#4543	`EnrichedLog`	netcoreapp3.1	1.98μs	2.88ns	10.4ns	0.0216	1.62 KB
#4543	`EnrichedLog`	net472	2.37μs	2.73ns	9.83ns	0.244	1.54 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	147ns	570ns	0.0564	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	283ns	1.1μs	0	0	4.21 KB
master	`EnrichedLog`	net472	148μs	172ns	666ns	0.665	0.222	4.38 KB
#4543	`EnrichedLog`	net6.0	113μs	216ns	837ns	0.0565	0	4.21 KB
#4543	`EnrichedLog`	netcoreapp3.1	118μs	268ns	1.04μs	0	0	4.21 KB
#4543	`EnrichedLog`	net472	148μs	155ns	600ns	0.666	0.222	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.92μs	1.09ns	4.21ns	0.0306	2.18 KB
master	`EnrichedLog`	netcoreapp3.1	4.03μs	1.44ns	5.4ns	0.0282	2.18 KB
master	`EnrichedLog`	net472	4.58μs	1.01ns	3.9ns	0.315	1.99 KB
#4543	`EnrichedLog`	net6.0	2.81μs	1.11ns	4.31ns	0.0297	2.18 KB
#4543	`EnrichedLog`	netcoreapp3.1	3.75μs	1.86ns	6.98ns	0.0299	2.18 KB
#4543	`EnrichedLog`	net472	4.58μs	1.54ns	5.75ns	0.315	1.99 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.39μs	0.427ns	1.65ns	0.0161	0	1.16 KB
master	`SendReceive`	netcoreapp3.1	1.57μs	0.573ns	2.15ns	0.0158	0	1.16 KB
master	`SendReceive`	net472	2.14μs	0.806ns	2.91ns	0.184	0	1.16 KB
#4543	`SendReceive`	net6.0	1.26μs	0.765ns	2.96ns	0.0164	0	1.16 KB
#4543	`SendReceive`	netcoreapp3.1	1.62μs	0.432ns	1.5ns	0.0155	0	1.16 KB
#4543	`SendReceive`	net472	1.99μs	2.43ns	8.76ns	0.184	0.00099	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.61μs	0.967ns	3.74ns	0.0209	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.65μs	2.42ns	9.36ns	0.02	1.58 KB
master	`EnrichedLog`	net472	4.09μs	2.37ns	9.19ns	0.311	1.96 KB
#4543	`EnrichedLog`	net6.0	2.59μs	2.37ns	8.88ns	0.0207	1.53 KB
#4543	`EnrichedLog`	netcoreapp3.1	3.63μs	1.2ns	4.5ns	0.0198	1.58 KB
#4543	`EnrichedLog`	net472	4.04μs	2.16ns	8.38ns	0.311	1.96 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4543

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.118	745.96	666.95

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	447ns	0.152ns	0.568ns	0.00741	536 B
master	`StartFinishSpan`	netcoreapp3.1	548ns	0.276ns	1.07ns	0.00712	536 B
master	`StartFinishSpan`	net472	727ns	0.274ns	1.02ns	0.0851	538 B
master	`StartFinishScope`	net6.0	523ns	0.383ns	1.43ns	0.00921	656 B
master	`StartFinishScope`	netcoreapp3.1	746ns	0.195ns	0.755ns	0.00877	656 B
master	`StartFinishScope`	net472	885ns	0.627ns	2.35ns	0.0978	618 B
#4543	`StartFinishSpan`	net6.0	472ns	0.162ns	0.604ns	0.00762	536 B
#4543	`StartFinishSpan`	netcoreapp3.1	585ns	0.182ns	0.656ns	0.00736	536 B
#4543	`StartFinishSpan`	net472	665ns	0.153ns	0.552ns	0.0854	538 B
#4543	`StartFinishScope`	net6.0	507ns	0.264ns	0.989ns	0.00906	656 B
#4543	`StartFinishScope`	netcoreapp3.1	667ns	0.329ns	1.23ns	0.00872	656 B
#4543	`StartFinishScope`	net472	893ns	1.68ns	6.52ns	0.0979	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4543

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1	1.141	744.67	849.82

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	616ns	0.299ns	1.04ns	0.00913	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	750ns	2.4ns	9.3ns	0.00898	656 B
master	`RunOnMethodBegin`	net472	1.04μs	0.479ns	1.73ns	0.0981	618 B
#4543	`RunOnMethodBegin`	net6.0	620ns	0.135ns	0.504ns	0.00927	656 B
#4543	`RunOnMethodBegin`	netcoreapp3.1	847ns	1.38ns	5.36ns	0.00882	656 B
#4543	`RunOnMethodBegin`	net472	1.01μs	0.293ns	1.09ns	0.0978	618 B

andrewlock · 2023-08-23T16:19:01Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4543) (11.740M)   : 0, 11739661
    master (11.644M)   : 0, 11644043
    benchmarks/2.36.0 (11.752M)   : 0, 11752454
    benchmarks/2.9.0 (11.752M)   : 0, 11752427

    section Automatic
    This PR (4543) (8.058M)   : 0, 8058106
    master (8.291M)   : 0, 8291142
    benchmarks/2.36.0 (8.280M)   : 0, 8280416
    benchmarks/2.9.0 (8.494M)   : 0, 8494490

    section Trace stats
    master (8.262M)   : 0, 8262399
    benchmarks/2.36.0 (8.148M)   : 0, 8147852

    section Manual
    This PR (4543) (10.216M)   : 0, 10215900
    master (10.527M)   : 0, 10527241
    benchmarks/2.36.0 (10.336M)   : 0, 10335525

    section Manual + Automatic
    This PR (4543) (7.700M)   : 0, 7699881
    master (7.944M)   : 0, 7944369
    benchmarks/2.36.0 (7.907M)   : 0, 7907307

    section Version Conflict
    master (7.251M)   : 0, 7250761
    benchmarks/2.36.0 (7.084M)   : 0, 7084420

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4543) (9.365M)   : 0, 9364504
    master (9.600M)   : 0, 9599884
    benchmarks/2.36.0 (9.582M)   : 0, 9582169
    benchmarks/2.9.0 (9.455M)   : 0, 9455151

    section Automatic
    This PR (4543) (6.810M)   : 0, 6810129
    master (6.795M)   : 0, 6794541
    benchmarks/2.36.0 (6.775M)   : 0, 6775278

    section Trace stats
    master (6.671M)   : 0, 6671385
    benchmarks/2.36.0 (6.725M)   : 0, 6725030

    section Manual
    This PR (4543) (8.411M)   : 0, 8411468
    master (8.468M)   : 0, 8467833
    benchmarks/2.36.0 (8.664M)   : 0, 8663817

    section Manual + Automatic
    This PR (4543) (6.408M)   : 0, 6408256
    master (6.192M)   : 0, 6191835
    benchmarks/2.36.0 (6.670M)   : 0, 6669993

    section Version Conflict
    master (5.780M)   : 0, 5780026
    benchmarks/2.36.0 (5.931M)   : 0, 5931025

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4543) (10.073M)   : 0, 10072515
    master (10.358M)   : 0, 10357887
    benchmarks/2.36.0 (10.209M)   : 0, 10209016
    benchmarks/2.9.0 (10.103M)   : 0, 10102653

    section Automatic
    This PR (4543) (7.312M)   : 0, 7312137
    master (7.273M)   : 0, 7273411
    benchmarks/2.36.0 (7.281M)   : 0, 7280896
    benchmarks/2.9.0 (7.341M)   : 0, 7340763

    section Trace stats
    master (7.316M)   : 0, 7316175
    benchmarks/2.36.0 (7.267M)   : 0, 7267042

    section Manual
    This PR (4543) (9.287M)   : 0, 9286758
    master (9.235M)   : 0, 9234639
    benchmarks/2.36.0 (9.078M)   : 0, 9077794

    section Manual + Automatic
    This PR (4543) (7.182M)   : 0, 7182059
    master (7.227M)   : 0, 7227130
    benchmarks/2.36.0 (7.075M)   : 0, 7075480

    section Version Conflict
    master (6.644M)   : 0, 6643598
    benchmarks/2.36.0 (6.479M)   : 0, 6478676

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4543) (7.325M)   : 0, 7325347
    master (7.438M)   : 0, 7437662
    benchmarks/2.36.0 (7.453M)   : 0, 7453304
    benchmarks/2.9.0 (7.826M)   : 0, 7825811

    section No attack
    This PR (4543) (2.108M)   : 0, 2107697
    master (2.151M)   : 0, 2150930
    benchmarks/2.36.0 (2.136M)   : 0, 2136014
    benchmarks/2.9.0 (3.260M)   : 0, 3260143

    section Attack
    This PR (4543) (1.656M)   : 0, 1656470
    master (1.670M)   : 0, 1669947
    benchmarks/2.36.0 (1.687M)   : 0, 1687301
    benchmarks/2.9.0 (2.532M)   : 0, 2532213

    section Blocking
    This PR (4543) (3.054M)   : 0, 3053991
    master (3.105M)   : 0, 3105259
    benchmarks/2.36.0 (3.104M)   : 0, 3103809

- Do we have the correct metric setting values? - Are we using the correct prefixes? - Are we supplying all the expected tags?

bouwkast

Looks good to me!

lucaspimentel · 2023-08-24T16:02:27Z

tracer/test/Datadog.Trace.Tests/Telemetry/Metrics/MetricTests.cs

+        // https://github.com/DataDog/dd-go/trace/apps/tracer-telemetry-intake/telemetry-metrics/static/common_metrics.json
+        // https://github.com/DataDog/dd-go/trace/apps/tracer-telemetry-intake/telemetry-metrics/static/dotnet_metrics.json


Should we try to download these files and compare them to the local copy?

Yeah, I considered that, but I wasn't sure if that would complicate things because the repo is private...

There's also the workflow problem - you really want to add them to the local copy + tracer before making a PR to the telemetry intake. So you don't want to always use the live version.

I was also wondering about a GitHub action that does this and creates a PR to update them, but that has some of the same issues, so I'm not really sure what's best :/ Open to suggestions!

* Add unit test for verifying that metrics conform to expectations - Do we have the correct metric setting values? - Are we using the correct prefixes? - Are we supplying all the expected tags? * Fix the name of the direct log queue-length metric * Run source generators * Fix name of direct_log_api.errors.responses metric * Run source generators

andrewlock added type:bug area:telemetry labels Aug 23, 2023

andrewlock requested a review from a team as a code owner August 23, 2023 15:08

github-actions bot added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Aug 23, 2023

andrewlock added 5 commits August 24, 2023 11:40

Add unit test for verifying that metrics conform to expectations

d662884

- Do we have the correct metric setting values? - Are we using the correct prefixes? - Are we supplying all the expected tags?

Fix the name of the direct log queue-length metric

29ba647

Run source generators

aef7531

Fix name of direct_log_api.errors.responses metric

f942f4a

Run source generators

05be8ad

andrewlock force-pushed the andrew/telemetry/fix-metric-name branch from 05ec2b3 to 05be8ad Compare August 24, 2023 10:43

andrewlock changed the title ~~Fix direct_log_queue metric name~~ Fix telemetry metrics and add validator Aug 24, 2023

bouwkast approved these changes Aug 24, 2023

View reviewed changes

lucaspimentel approved these changes Aug 24, 2023

View reviewed changes

lucaspimentel reviewed Aug 24, 2023

View reviewed changes

andrewlock merged commit 83b2d5b into master Aug 25, 2023

andrewlock deleted the andrew/telemetry/fix-metric-name branch August 25, 2023 06:19

github-actions bot added this to the vNext milestone Aug 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix telemetry metrics and add validator #4543

Fix telemetry metrics and add validator #4543

andrewlock commented Aug 23, 2023 •

edited

Loading

datadog-ddstaging bot commented Aug 23, 2023 •

edited

Loading

andrewlock commented Aug 23, 2023 •

edited

Loading

andrewlock commented Aug 23, 2023

Raw results

Faster 🎉 in #4543

Raw results

Raw results

More allocations ⚠️ in #4543

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4543

Raw results

Slower ⚠️ in #4543

Raw results

andrewlock commented Aug 23, 2023 •

edited

Loading

bouwkast left a comment

lucaspimentel Aug 24, 2023

andrewlock Aug 24, 2023 •

edited

Loading

		// https://github.com/DataDog/dd-go/trace/apps/tracer-telemetry-intake/telemetry-metrics/static/common_metrics.json
		// https://github.com/DataDog/dd-go/trace/apps/tracer-telemetry-intake/telemetry-metrics/static/dotnet_metrics.json

Fix telemetry metrics and add validator #4543

Fix telemetry metrics and add validator #4543

Conversation

andrewlock commented Aug 23, 2023 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Aug 23, 2023 • edited Loading

Datadog Report

andrewlock commented Aug 23, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Aug 23, 2023

Benchmarks Report 🐌

Benchmark details

Raw results

Faster 🎉 in #4543

Raw results

Raw results

More allocations ⚠️ in #4543

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4543

Raw results

Slower ⚠️ in #4543

Raw results

andrewlock commented Aug 23, 2023 • edited Loading

Throughput/Crank Report:zap:

bouwkast left a comment

Choose a reason for hiding this comment

lucaspimentel Aug 24, 2023

Choose a reason for hiding this comment

andrewlock Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

andrewlock commented Aug 23, 2023 •

edited

Loading

datadog-ddstaging bot commented Aug 23, 2023 •

edited

Loading

andrewlock commented Aug 23, 2023 •

edited

Loading

andrewlock commented Aug 23, 2023 •

edited

Loading

andrewlock Aug 24, 2023 •

edited

Loading