[Dynamic Instrumentation] Hotfix SymDB and ER #6468

dudikeleti · 2024-12-19T13:53:14Z

Disable compresison in SymDB and fix out of range exception while collecting arguments of some methods
Exception Replay should work even Live Debugger has not initialized

GreenMatan

LGTM

andrewlock · 2024-12-19T14:56:43Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6468) - mean (68ms)  : 66, 71
     .   : milestone, 68,
    master - mean (69ms)  : 65, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6468) - mean (976ms)  : 951, 1002
     .   : milestone, 976,
    master - mean (977ms)  : 955, 999
     .   : milestone, 977,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6468) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6468) - mean (678ms)  : 664, 692
     .   : milestone, 678,
    master - mean (678ms)  : 667, 689
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6468) - mean (91ms)  : 88, 94
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6468) - mean (635ms)  : 622, 648
     .   : milestone, 635,
    master - mean (635ms)  : 615, 655
     .   : milestone, 635,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6468) - mean (193ms)  : 189, 197
     .   : milestone, 193,
    master - mean (194ms)  : 187, 201
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6468) - mean (1,103ms)  : 1072, 1135
     .   : milestone, 1103,
    master - mean (1,103ms)  : 1070, 1136
     .   : milestone, 1103,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6468) - mean (279ms)  : 275, 283
     .   : milestone, 279,
    master - mean (278ms)  : 274, 282
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6468) - mean (873ms)  : 844, 902
     .   : milestone, 873,
    master - mean (870ms)  : 846, 894
     .   : milestone, 870,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6468) - mean (266ms)  : 262, 270
     .   : milestone, 266,
    master - mean (267ms)  : 263, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6468) - mean (851ms)  : 818, 884
     .   : milestone, 851,
    master - mean (848ms)  : 814, 882
     .   : milestone, 848,

datadog-ddstaging · 2024-12-19T15:40:28Z

Datadog Report

Branch report: dudik/hotfix-37
Commit report: 8ad6dd9
Test service: dd-trace-dotnet

✅ 0 Failed, 471917 Passed, 3708 Skipped, 52h 10m 9.87s Total Time

andrewlock

I'm not really sure where the bug was - in general it would be nice if there was a test showing the failure which is then fixed 🙂

tracer/src/Datadog.Trace/Debugger/Upload/SymbolUploadApi.cs

dudikeleti · 2024-12-19T17:07:22Z

I'm not really sure where the bug was - in general it would be nice if there was a test showing the failure which is then fixed 🙂

The issue was with the compressed symbols. It cause to missing symbols in all .net services.
There is a test in the debugger settings that makes sure compression is disabled.

2. Do not use LD in ER 3. Fix OutOfRange in SymDB

andrewlock · 2024-12-20T00:41:09Z

Benchmarks Report for tracer 🐌

Benchmarks for #6468 compared to master:

2 benchmarks are slower, with geometric mean 1.149
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.96μs	44.9ns	321ns	0.0119	0.00397	0	5.62 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	56.3ns	356ns	0.0204	0.0102	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	42.8ns	166ns	1.05	0.314	0.0966	6.2 KB
#6468	`StartStopWithChild`	net6.0	8.08μs	45.2ns	300ns	0.0207	0.00828	0	5.61 KB
#6468	`StartStopWithChild`	netcoreapp3.1	10.1μs	56.9ns	386ns	0.0149	0.00992	0	5.8 KB
#6468	`StartStopWithChild`	net472	16.3μs	55.7ns	216ns	1.07	0.326	0.114	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	481μs	450ns	1.62μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	640μs	296ns	1.15μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	847μs	887ns	3.32μs	0.425	3.3 KB
#6468	`WriteAndFlushEnrichedTraces`	net6.0	497μs	539ns	2.09μs	0	2.7 KB
#6468	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	650μs	718ns	2.78μs	0	2.7 KB
#6468	`WriteAndFlushEnrichedTraces`	net472	843μs	673ns	2.52μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	156μs	893ns	6.8μs	0.143	14.47 KB
master	`SendRequest`	netcoreapp3.1	180μs	1.24μs	12.3μs	0.171	17.27 KB
master	`SendRequest`	net472	0.00296ns	0.00117ns	0.00453ns	0	0 b
#6468	`SendRequest`	net6.0	154μs	934ns	9.15μs	0.162	14.47 KB
#6468	`SendRequest`	netcoreapp3.1	168μs	981ns	8.88μs	0.182	17.27 KB
#6468	`SendRequest`	net472	0ns	0ns	0ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	565μs	3.1μs	19.1μs	0.296	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	707μs	4.14μs	39.3μs	0.331	0	0	41.79 KB
master	`WriteAndFlushEnrichedTraces`	net472	835μs	4.16μs	17.7μs	8.75	2.5	0.417	53.27 KB
#6468	`WriteAndFlushEnrichedTraces`	net6.0	559μs	3.11μs	20.1μs	0.571	0	0	41.65 KB
#6468	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	686μs	3.63μs	18.5μs	0.326	0	0	41.9 KB
#6468	`WriteAndFlushEnrichedTraces`	net472	803μs	1.98μs	7.14μs	8.41	2.4	0.401	53.36 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.27μs	1.79ns	6.94ns	0.0142	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.89ns	7.08ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.19μs	3.96ns	15.3ns	0.156	0.00109	987 B
#6468	`ExecuteNonQuery`	net6.0	1.28μs	1.84ns	7.14ns	0.0141	0	1.02 KB
#6468	`ExecuteNonQuery`	netcoreapp3.1	1.75μs	2.56ns	9.9ns	0.0139	0	1.02 KB
#6468	`ExecuteNonQuery`	net472	2.17μs	2.68ns	10.4ns	0.157	0.00108	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.22μs	0.52ns	1.95ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.53μs	0.756ns	2.83ns	0.013	976 B
master	`CallElasticsearch`	net472	2.53μs	0.959ns	3.59ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.35μs	0.886ns	3.43ns	0.0134	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	3.03ns	11.3ns	0.0138	1.02 KB
master	`CallElasticsearchAsync`	net472	2.64μs	0.953ns	3.44ns	0.167	1.05 KB
#6468	`CallElasticsearch`	net6.0	1.24μs	0.572ns	2.14ns	0.0137	976 B
#6468	`CallElasticsearch`	netcoreapp3.1	1.49μs	0.769ns	2.98ns	0.0128	976 B
#6468	`CallElasticsearch`	net472	2.51μs	0.887ns	3.32ns	0.157	995 B
#6468	`CallElasticsearchAsync`	net6.0	1.35μs	0.496ns	1.79ns	0.0136	952 B
#6468	`CallElasticsearchAsync`	netcoreapp3.1	1.68μs	1.09ns	4.21ns	0.0137	1.02 KB
#6468	`CallElasticsearchAsync`	net472	2.55μs	1.23ns	4.75ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.4μs	0.816ns	3.16ns	0.0133	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	0.585ns	2.27ns	0.0126	952 B
master	`ExecuteAsync`	net472	1.83μs	0.365ns	1.37ns	0.145	915 B
#6468	`ExecuteAsync`	net6.0	1.43μs	0.613ns	2.37ns	0.0136	952 B
#6468	`ExecuteAsync`	netcoreapp3.1	1.67μs	0.457ns	1.71ns	0.0126	952 B
#6468	`ExecuteAsync`	net472	1.83μs	0.376ns	1.41ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.51μs	1.11ns	4ns	0.0315	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.26μs	3.27ns	12.7ns	0.0368	2.85 KB
master	`SendAsync`	net472	7.52μs	1.93ns	7.48ns	0.496	3.12 KB
#6468	`SendAsync`	net6.0	4.43μs	1.37ns	5.14ns	0.0308	2.31 KB
#6468	`SendAsync`	netcoreapp3.1	5.25μs	2.5ns	9.68ns	0.0365	2.85 KB
#6468	`SendAsync`	net472	7.28μs	1.1ns	4.11ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.49μs	0.564ns	2.11ns	0.0232	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.27μs	0.733ns	2.74ns	0.0219	1.64 KB
master	`EnrichedLog`	net472	2.68μs	1.1ns	4.1ns	0.249	1.57 KB
#6468	`EnrichedLog`	net6.0	1.48μs	0.826ns	3.09ns	0.023	1.64 KB
#6468	`EnrichedLog`	netcoreapp3.1	2.26μs	1.14ns	4.26ns	0.0226	1.64 KB
#6468	`EnrichedLog`	net472	2.69μs	1.32ns	5.11ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	155ns	559ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	122μs	224ns	866ns	0.0608	0	4.28 KB
master	`EnrichedLog`	net472	150μs	129ns	498ns	0.675	0.225	4.46 KB
#6468	`EnrichedLog`	net6.0	116μs	232ns	899ns	0.0581	0	4.28 KB
#6468	`EnrichedLog`	netcoreapp3.1	121μs	255ns	986ns	0	0	4.28 KB
#6468	`EnrichedLog`	net472	152μs	193ns	746ns	0.688	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.07μs	1.14ns	4.41ns	0.0309	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.13μs	1.62ns	6.26ns	0.0292	2.2 KB
master	`EnrichedLog`	net472	4.89μs	1.28ns	4.96ns	0.319	2.02 KB
#6468	`EnrichedLog`	net6.0	2.98μs	1.44ns	5.4ns	0.0303	2.2 KB
#6468	`EnrichedLog`	netcoreapp3.1	4.13μs	1.72ns	6.65ns	0.029	2.2 KB
#6468	`EnrichedLog`	net472	4.94μs	1.43ns	5.53ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	1.39ns	5.2ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	1.02ns	3.81ns	0.0157	1.14 KB
master	`SendReceive`	net472	2.14μs	1.21ns	4.67ns	0.183	1.16 KB
#6468	`SendReceive`	net6.0	1.45μs	0.344ns	1.29ns	0.0159	1.14 KB
#6468	`SendReceive`	netcoreapp3.1	1.75μs	1.06ns	3.96ns	0.0149	1.14 KB
#6468	`SendReceive`	net472	2.12μs	2.91ns	11.3ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.88μs	3ns	11.6ns	0.0216	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.85μs	3.69ns	14.3ns	0.0213	1.65 KB
master	`EnrichedLog`	net472	4.19μs	2.82ns	10.9ns	0.323	2.04 KB
#6468	`EnrichedLog`	net6.0	2.85μs	1.19ns	4.47ns	0.0225	1.6 KB
#6468	`EnrichedLog`	netcoreapp3.1	3.88μs	1.92ns	7.44ns	0.0215	1.65 KB
#6468	`EnrichedLog`	net472	4.32μs	2.05ns	7.95ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6468

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.172	397.50	465.95
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.126	684.80	771.05

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	397ns	0.239ns	0.895ns	0.008	576 B
master	`StartFinishSpan`	netcoreapp3.1	617ns	0.43ns	1.67ns	0.00776	576 B
master	`StartFinishSpan`	net472	704ns	0.203ns	0.785ns	0.0918	578 B
master	`StartFinishScope`	net6.0	558ns	0.432ns	1.67ns	0.0098	696 B
master	`StartFinishScope`	netcoreapp3.1	685ns	0.368ns	1.42ns	0.00942	696 B
master	`StartFinishScope`	net472	897ns	0.523ns	2.03ns	0.104	658 B
#6468	`StartFinishSpan`	net6.0	466ns	0.413ns	1.6ns	0.00811	576 B
#6468	`StartFinishSpan`	netcoreapp3.1	557ns	0.246ns	0.92ns	0.00783	576 B
#6468	`StartFinishSpan`	net472	687ns	0.349ns	1.35ns	0.0915	578 B
#6468	`StartFinishScope`	net6.0	570ns	0.237ns	0.919ns	0.00976	696 B
#6468	`StartFinishScope`	netcoreapp3.1	772ns	0.709ns	2.65ns	0.00931	696 B
#6468	`StartFinishScope`	net472	827ns	0.626ns	2.26ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	615ns	0.399ns	1.54ns	0.00958	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1μs	0.866ns	3.35ns	0.00899	696 B
master	`RunOnMethodBegin`	net472	1.14μs	1.08ns	4.19ns	0.104	658 B
#6468	`RunOnMethodBegin`	net6.0	610ns	0.168ns	0.649ns	0.00973	696 B
#6468	`RunOnMethodBegin`	netcoreapp3.1	964ns	0.631ns	2.36ns	0.00917	696 B
#6468	`RunOnMethodBegin`	net472	1.13μs	0.662ns	2.56ns	0.104	658 B

- Disable compression in SymDB - Fix out of range exception while collecting arguments of some methods - Fix type mismatch in dnlib metadata reader that causes invalid cast exception - Exception Replay should work even Live Debugger has not initialized

dudikeleti requested a review from a team as a code owner December 19, 2024 13:53

dudikeleti force-pushed the dudik/hotfix-37 branch from 391c9a9 to 77976dc Compare December 19, 2024 13:56

GreenMatan approved these changes Dec 19, 2024

View reviewed changes

github-actions bot added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Dec 19, 2024

dudikeleti force-pushed the dudik/hotfix-37 branch from 77976dc to baca6a2 Compare December 19, 2024 15:37

andrewlock approved these changes Dec 19, 2024

View reviewed changes

tracer/src/Datadog.Trace/Debugger/Upload/SymbolUploadApi.cs Outdated Show resolved Hide resolved

dudikeleti added 3 commits December 20, 2024 00:08

1. Disable symdb compression

7642321

2. Do not use LD in ER 3. Fix OutOfRange in SymDB

Update debugger settings tests

0abb540

fix pr comment

8ad6dd9

dudikeleti force-pushed the dudik/hotfix-37 branch from 139cfcb to 8ad6dd9 Compare December 19, 2024 23:08

dudikeleti changed the title ~~[DI] Hotfix SymDB and ER~~ [Dynamic Instrumentation] Hotfix SymDB and ER Dec 19, 2024

dudikeleti merged commit b1963ed into master Dec 20, 2024
63 of 68 checks passed

dudikeleti deleted the dudik/hotfix-37 branch December 20, 2024 04:45

github-actions bot added this to the vNext-v3 milestone Dec 20, 2024

andrewlock added type:bug area:debugger identified-by:customer and removed area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) labels Jan 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dynamic Instrumentation] Hotfix SymDB and ER #6468

[Dynamic Instrumentation] Hotfix SymDB and ER #6468

dudikeleti commented Dec 19, 2024 •

edited

Loading

GreenMatan left a comment

andrewlock commented Dec 19, 2024 •

edited

Loading

datadog-ddstaging bot commented Dec 19, 2024 •

edited

Loading

andrewlock left a comment

dudikeleti commented Dec 19, 2024

andrewlock commented Dec 20, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6468

Raw results

Raw results

[Dynamic Instrumentation] Hotfix SymDB and ER #6468

[Dynamic Instrumentation] Hotfix SymDB and ER #6468

Conversation

dudikeleti commented Dec 19, 2024 • edited Loading

GreenMatan left a comment

Choose a reason for hiding this comment

andrewlock commented Dec 19, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Dec 19, 2024 • edited Loading

Datadog Report

andrewlock left a comment

Choose a reason for hiding this comment

dudikeleti commented Dec 19, 2024

andrewlock commented Dec 20, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6468

Raw results

Raw results

dudikeleti commented Dec 19, 2024 •

edited

Loading

andrewlock commented Dec 19, 2024 •

edited

Loading

datadog-ddstaging bot commented Dec 19, 2024 •

edited

Loading