Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Dynamic Instrumentation] Upload symbols to SymDB (with System.Reflection.Metadata) #4782

Merged
merged 91 commits into from
Jan 18, 2024
Merged

[Dynamic Instrumentation] Upload symbols to SymDB (with System.Reflection.Metadata) #4782

merged 91 commits into from
Jan 18, 2024

Conversation

dudikeleti
Copy link
Contributor

@dudikeleti dudikeleti commented Oct 30, 2023 •
edited
Loading

Summary of changes

This PR implements uploading of code symbols Datadog, which enables customers to enjoy an auto-complete experience when they use the Dynamic Instrumentation to generate dynamic logs, spans, and metrics. The symbol information enables autocomplete both when selecting a class/method to instrument, and when typing in expressions.

Reason for change

Currently, when a user wants to create a probe or add an expression to a probe in Dynamic Instrumentation, they do not get any auto-complete while typing in names of method or local variables, and so they are likely to make spelling mistakes, which would cause them to have a frustrating experience with the product. With this change, we are uploading all the symbols info for a given service / environment / version to "SymDB", our new Symbol Database, so we can display to the user which symbols exist for the current scope.

Implementation details

SymDB upload currently only works if (1) is it explicitly enabled via an environment variable (2) the tracer is working against an updated version of datadog-agent which supports the new symdb endpoint.

When the application loads, we extract symbols for all loaded assemblies (excluding Microsoft assemblies and some third party assemblies), and as more assemblies are loaded, we add them to a processing queue.
To avoid performance bottlenecks, we perform the (fairly resource intensive) work of symbol extraction in a background thread using async await. To prevent exceeding memory limits, we do not parallelize the work and extract symbols from one assembly at a time, chunking the uploads and carefully managing memory consumption.

For obtaining assembly metadata we rely on System.Reflection.Metadata. For PDB information, we rely on System.Reflection.Metadata in the of case portable or embedded PDBs, and dnlib library in the case of Windows PDBs.

Other details

This PR includes also the vendoring of the following libraries:

This was necessary in order to perform the symbol extraction with an acceptable CPU and memory overhead. Now that we have vendored in these libraries, it may open ample opportunities for further performance optimization across the client library.

Regarding the change in the size of the library after the vendoring, it is an addition of between 730 and 950 KB. But since we can now avoid the dnlib vendoring which is about 1000 KB (planned for a feature PR), in the end, there will be no significant change in size of the library.

To make it all work also in NET Framework, we base the vendoring process on decompiling the assemblies of the above packages for NET Framework version. For some of the code we used InlineIL.Fody in order to generate a suitable IL code.

Test coverage

AsyncMethod
ComprehensiveLinqWithClosure
ComprehensiveLinqWithoutClosure
EmptyType
LambdaWithClosure
LambdaWithFieldClosure
LambdaWithoutClosure
LambdaWithStaticFieldClosure
LocalFunctionWithClosure
LocalFunctionWithoutClosure
MultipleHoistedLocals
MultipleHoistedLocalsInStateMachine
MultipleScopes
NestedFluentLinq
PartialClass

@dudikeleti dudikeleti changed the title Dudik/upload symbols vendoring [Dynamic Instrumentation] Symbols DB (with System.Reflection.Metadata) Nov 3, 2023
@dudikeleti dudikeleti changed the title [Dynamic Instrumentation] Symbols DB (with System.Reflection.Metadata) [Dynamic Instrumentation] Upload symbols to SymDB (with System.Reflection.Metadata) Nov 3, 2023
@dudikeleti dudikeleti marked this pull request as ready for review November 3, 2023 19:17
@dudikeleti dudikeleti requested review from a team as code owners November 3, 2023 19:17
@dudikeleti dudikeleti force-pushed the dudik/upload-symbols-vendoring branch from 601f2f6 to be34a1b Compare November 3, 2023 19:25
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 3, 2023 •
edited
Loading

Datadog Report

Branch report: dudik/upload-symbols-vendoring
Commit report: 4cf1caa

❌ dd-trace-dotnet: 2405 Failed (1 Known Flaky), 0 New Flaky, 159431 Passed, 1291 Skipped, 35m 43.54s Wall Time

❌ Failed Tests (2405)

This report shows up to 5 failed tests.

  • ResolveLineThatIncludesLambdaExpressions - Datadog.Trace.Debugger.IntegrationTests.LineProbeResolverTest - Details

    Expand for error 
     Token 0x000000d1 is not valid in the scope of module Samples.Probes.TestRuns.dll. (Parameter 'metadataToken')

  • ResolveLineThatIncludesLambdaExpressions - Datadog.Trace.Debugger.IntegrationTests.LineProbeResolverTest - Details

    Expand for error 
     Token 0x000000d1 is not a valid MethodBase token in the scope of module Samples.Probes.TestRuns.dll. (Parameter 'metadataToken')

  • ResolveLineThatIncludesLambdaExpressions - Datadog.Trace.Debugger.IntegrationTests.LineProbeResolverTest - Details

    Expand for error 
     Token 0x000000d1 is not valid in the scope of module Samples.Probes.TestRuns.dll. (Parameter 'metadataToken')

  • ResolveLineThatIncludesLambdaExpressions - Datadog.Trace.Debugger.IntegrationTests.LineProbeResolverTest - Details

    Expand for error 
     Token 0x000000d1 is not valid in the scope of module Samples.Probes.TestRuns.dll. (Parameter 'metadataToken')

  • ResolveLineThatIncludesLambdaExpressions - Datadog.Trace.Debugger.IntegrationTests.LineProbeResolverTest - Details

    Expand for error 
     Token 0x000000ce is not valid in the scope of module Samples.Probes.TestRuns.dll.
 Parameter name: metadataToken

@andrewlock
Copy link
Member

andrewlock commented Nov 3, 2023 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4782) - mean (73ms)  : 61, 85
     .   : milestone, 73,
    master - mean (70ms)  : 63, 78
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4782) - mean (960ms)  : 939, 981
     .   : milestone, 960,
    master - mean (1,030ms)  : 1013, 1048
     .   : milestone, 1030,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4782) - mean (106ms)  : 104, 109
     .   : milestone, 106,
    master - mean (106ms)  : 103, 109
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4782) - mean (705ms)  : 689, 721
     .   : milestone, 705,
    master - mean (736ms)  : 718, 755
     .   : milestone, 736,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4782) - mean (91ms)  : 88, 93
     .   : milestone, 91,
    master - mean (90ms)  : 88, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4782) - mean (655ms)  : 636, 674
     .   : milestone, 655,
    master - mean (694ms)  : 673, 714
     .   : milestone, 694,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4782) - mean (188ms)  : 185, 191
     .   : milestone, 188,
    master - mean (188ms)  : 186, 190
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (4782) - mean (1,072ms)  : 1046, 1098
     .   : milestone, 1072,
    master - mean (1,132ms)  : 1110, 1154
     .   : milestone, 1132,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4782) - mean (272ms)  : 268, 276
     .   : milestone, 272,
    master - mean (271ms)  : 268, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4782) - mean (1,051ms)  : 1028, 1075
     .   : milestone, 1051,
    master - mean (1,089ms)  : 1062, 1115
     .   : milestone, 1089,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4782) - mean (261ms)  : 258, 265
     .   : milestone, 261,
    master - mean (261ms)  : 258, 265
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4782) - mean (1,018ms)  : 996, 1039
     .   : milestone, 1018,
    master - mean (1,050ms)  : 1029, 1071
     .   : milestone, 1050,
Loading

@andrewlock
Copy link
Member

andrewlock commented Nov 3, 2023 •
edited
Loading

Benchmarks Report 🐌

Benchmarks for #4782 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.300
  • 3 benchmarks are slower, with geometric mean 1.129
  • 2 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.66μs 47.4ns 284ns 0.0302 0.0129 0 7.48 KB
master StartStopWithChild netcoreapp3.1 10.9μs 58.6ns 337ns 0.0264 0.0106 0 7.57 KB
master StartStopWithChild net472 17.4μs 51.8ns 187ns 1.32 0.326 0.112 7.96 KB
#4782 StartStopWithChild net6.0 8.83μs 48.9ns 293ns 0.0331 0.0166 0 7.48 KB
#4782 StartStopWithChild netcoreapp3.1 10.7μs 52.7ns 236ns 0.0212 0.0106 0 7.57 KB
#4782 StartStopWithChild net472 17.3μs 46.7ns 181ns 1.33 0.368 0.111 7.95 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 457μs 119ns 461ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 630μs 341ns 1.32μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 790μs 247ns 854ns 0.393 0 0 3.3 KB
#4782 WriteAndFlushEnrichedTraces net6.0 467μs 349ns 1.35μs 0 0 0 2.7 KB
#4782 WriteAndFlushEnrichedTraces netcoreapp3.1 637μs 172ns 666ns 0 0 0 2.7 KB
#4782 WriteAndFlushEnrichedTraces net472 784μs 516ns 1.93μs 0.391 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4782

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.114 191.39 213.18

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 38.2μs 23ns 85.9ns 0.0191 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 41.4μs 41.7ns 161ns 0.0204 0 0 1.74 KB
master AllCycleSimpleBody net472 44.1μs 38.3ns 148ns 0.267 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 201μs 58.7ns 219ns 0.101 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 211μs 234ns 907ns 0.105 0 0 9.14 KB
master AllCycleMoreComplexBody net472 227μs 82ns 296ns 1.47 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 139ns 0.0449ns 0.168ns 0.00397 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 191ns 0.286ns 1.11ns 0.00373 0 0 272 B
master ObjectExtractorSimpleBody net472 165ns 0.0546ns 0.204ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.99μs 1.65ns 6.19ns 0.0527 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.1μs 4.34ns 16.8ns 0.051 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.76μs 1.58ns 5.9ns 0.601 0.00567 0 3.8 KB
#4782 AllCycleSimpleBody net6.0 37.9μs 5.11ns 18.4ns 0.0189 0 0 1.77 KB
#4782 AllCycleSimpleBody netcoreapp3.1 42μs 22.1ns 76.5ns 0.021 0 0 1.74 KB
#4782 AllCycleSimpleBody net472 44.3μs 15ns 58.1ns 0.288 0 0 1.81 KB
#4782 AllCycleMoreComplexBody net6.0 200μs 84.5ns 316ns 0.101 0 0 9.25 KB
#4782 AllCycleMoreComplexBody netcoreapp3.1 212μs 218ns 845ns 0.106 0 0 9.14 KB
#4782 AllCycleMoreComplexBody net472 225μs 69.6ns 260ns 1.46 0 0 9.32 KB
#4782 ObjectExtractorSimpleBody net6.0 141ns 0.0368ns 0.133ns 0.00392 0 0 280 B
#4782 ObjectExtractorSimpleBody netcoreapp3.1 213ns 0.0792ns 0.286ns 0.00365 0 0 272 B
#4782 ObjectExtractorSimpleBody net472 166ns 0.101ns 0.39ns 0.0446 0 0 281 B
#4782 ObjectExtractorMoreComplexBody net6.0 2.96μs 1.07ns 4.02ns 0.0533 0 0 3.78 KB
#4782 ObjectExtractorMoreComplexBody netcoreapp3.1 4μs 2.41ns 9.33ns 0.0499 0 0 3.69 KB
#4782 ObjectExtractorMoreComplexBody net472 3.78μs 1.58ns 5.69ns 0.602 0.00566 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 51.5μs 84.8ns 370ns 0.227 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 67.9μs 320ns 1.2μs 0.197 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 95.3μs 43.6ns 163ns 2.52 0.0953 0 16.14 KB
master RunWafTwice(args=NestedMap (10)) net6.0 59.5μs 38.2ns 148ns 0.226 0 0 16.6 KB
master RunWafTwice(args=NestedMap (10)) netcoreapp3.1 74.6μs 394ns 2.08μs 0.221 0 0 16.58 KB
master RunWafTwice(args=NestedMap (10)) net472 102μs 53.6ns 208ns 2.65 0.102 0 16.69 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 108μs 43.2ns 156ns 0.269 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 129μs 648ns 3.11μs 0.259 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 163μs 783ns 3.23μs 3.56 0.158 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 109μs 132ns 509ns 0.436 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 130μs 623ns 2.64μs 0.448 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 193μs 77.2ns 289ns 5.28 0.384 0 33.67 KB
master RunWafTwice(args=NestedMap (100)) net6.0 112μs 69ns 267ns 0.47 0 0 33.3 KB
master RunWafTwice(args=NestedMap (100)) netcoreapp3.1 142μs 220ns 851ns 0.399 0 0 33.86 KB
master RunWafTwice(args=NestedMap (100)) net472 194μs 822ns 3.18μs 5.44 0.382 0 34.23 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 158μs 38.4ns 144ns 0.509 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 199μs 940ns 3.76μs 0.506 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 254μs 172ns 664ns 6.33 0.506 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 108μs 38.1ns 148ns 0.431 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 132μs 657ns 3.01μs 0.399 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 186μs 141ns 545ns 5.17 0.369 0 32.63 KB
master RunWafTwice(args=NestedMap (20)) net6.0 106μs 618ns 5.39μs 0.449 0 0 32.72 KB
master RunWafTwice(args=NestedMap (20)) netcoreapp3.1 139μs 616ns 2.39μs 0.423 0 0 32.82 KB
master RunWafTwice(args=NestedMap (20)) net472 192μs 136ns 526ns 5.26 0.383 0 33.19 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 156μs 83.3ns 289ns 0.543 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 199μs 283ns 1.02μs 0.484 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 256μs 966ns 3.74μs 6.17 0.504 0 39.2 KB
#4782 RunWaf(args=NestedMap (10)) net6.0 54.4μs 17.1ns 64ns 0.218 0 0 16.06 KB
#4782 RunWaf(args=NestedMap (10)) netcoreapp3.1 67.7μs 379ns 2.54μs 0.194 0 0 16.06 KB
#4782 RunWaf(args=NestedMap (10)) net472 97.7μs 504ns 2.31μs 2.52 0.0953 0 16.14 KB
#4782 RunWafTwice(args=NestedMap (10)) net6.0 53.4μs 17.7ns 66.2ns 0.214 0 0 16.6 KB
#4782 RunWafTwice(args=NestedMap (10)) netcoreapp3.1 74.1μs 386ns 1.93μs 0.22 0 0 16.58 KB
#4782 RunWafTwice(args=NestedMap (10)) net472 104μs 38.2ns 148ns 2.65 0.104 0 16.69 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 112μs 623ns 3.79μs 0.266 0 0 22.41 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 128μs 372ns 1.29μs 0.256 0 0 22.36 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [22]) net472 163μs 714ns 2.77μs 3.53 0.16 0 22.7 KB
#4782 RunWaf(args=NestedMap (100)) net6.0 109μs 87.6ns 339ns 0.437 0 0 32.76 KB
#4782 RunWaf(args=NestedMap (100)) netcoreapp3.1 134μs 632ns 2.53μs 0.405 0 0 33.33 KB
#4782 RunWaf(args=NestedMap (100)) net472 195μs 436ns 1.69μs 5.32 0.38 0 33.67 KB
#4782 RunWafTwice(args=NestedMap (100)) net6.0 101μs 37.7ns 146ns 0.444 0 0 33.3 KB
#4782 RunWafTwice(args=NestedMap (100)) netcoreapp3.1 140μs 659ns 2.55μs 0.458 0 0 33.86 KB
#4782 RunWafTwice(args=NestedMap (100)) net472 193μs 98.5ns 382ns 5.42 0.387 0 34.23 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 167μs 704ns 2.73μs 0.491 0 0 39.1 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 199μs 499ns 1.8μs 0.494 0 0 39.63 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [23]) net472 264μs 890ns 3.45μs 6.34 0.507 0 40.23 KB
#4782 RunWaf(args=NestedMap (20)) net6.0 98.7μs 42ns 163ns 0.445 0 0 32.18 KB
#4782 RunWaf(args=NestedMap (20)) netcoreapp3.1 125μs 71.9ns 249ns 0.438 0 0 32.3 KB
#4782 RunWaf(args=NestedMap (20)) net472 187μs 899ns 3.48μs 5.11 0.365 0 32.63 KB
#4782 RunWafTwice(args=NestedMap (20)) net6.0 106μs 617ns 5.45μs 0.445 0 0 32.72 KB
#4782 RunWafTwice(args=NestedMap (20)) netcoreapp3.1 135μs 536ns 1.93μs 0.4 0 0 32.82 KB
#4782 RunWafTwice(args=NestedMap (20)) net472 198μs 448ns 1.73μs 5.26 0.383 0 33.19 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 166μs 102ns 397ns 0.482 0 0 38.53 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 196μs 1.01μs 4.86μs 0.485 0 0 38.6 KB
#4782 RunWafWithAttack(args=Neste(...)tack) [22]) net472 258μs 678ns 2.63μs 6.16 0.503 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 174μs 142ns 531ns 0.173 0 0 18.25 KB
master SendRequest netcoreapp3.1 194μs 203ns 787ns 0.194 0 0 20.41 KB
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#4782 SendRequest net6.0 175μs 230ns 892ns 0.175 0 0 18.26 KB
#4782 SendRequest netcoreapp3.1 193μs 336ns 1.3μs 0.193 0 0 20.41 KB
#4782 SendRequest net472 0.000289ns 0.000122ns 0.000441ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 550μs 2.17μs 8.42μs 0.558 0 0 41.6 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 646μs 878ns 3.4μs 0.326 0 0 41.87 KB
master WriteAndFlushEnrichedTraces net472 827μs 1.61μs 5.81μs 8.33 2.5 0.417 53.24 KB
#4782 WriteAndFlushEnrichedTraces net6.0 544μs 1.45μs 5.61μs 0.568 0 0 41.69 KB
#4782 WriteAndFlushEnrichedTraces netcoreapp3.1 648μs 1.13μs 4.39μs 0.322 0 0 41.78 KB
#4782 WriteAndFlushEnrichedTraces net472 829μs 3.3μs 12.8μs 8.12 2.44 0.406 53.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.1μs 2.38ns 9.23ns 0.0108 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.52μs 2.33ns 9.04ns 0.0106 0 0 768 B
master ExecuteNonQuery net472 1.67μs 0.715ns 2.58ns 0.115 0 0 730 B
#4782 ExecuteNonQuery net6.0 1.05μs 0.506ns 1.96ns 0.0104 0 0 768 B
#4782 ExecuteNonQuery netcoreapp3.1 1.46μs 0.438ns 1.64ns 0.0103 0 0 768 B
#4782 ExecuteNonQuery net472 1.76μs 4.54ns 17.6ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.19μs 0.581ns 2.25ns 0.0132 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.52μs 3.04ns 11.8ns 0.0122 0 0 936 B
master CallElasticsearch net472 2.5μs 1.86ns 7.22ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.38μs 0.318ns 1.19ns 0.0125 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.71μs 0.523ns 1.89ns 0.0136 0 0 984 B
master CallElasticsearchAsync net472 2.6μs 2.15ns 8.33ns 0.16 0 0 1.01 KB
#4782 CallElasticsearch net6.0 1.25μs 0.467ns 1.75ns 0.0132 0 0 936 B
#4782 CallElasticsearch netcoreapp3.1 1.53μs 0.527ns 1.9ns 0.0126 0 0 936 B
#4782 CallElasticsearch net472 2.55μs 0.559ns 2.09ns 0.152 0 0 955 B
#4782 CallElasticsearchAsync net6.0 1.28μs 0.579ns 2.17ns 0.0129 0 0 912 B
#4782 CallElasticsearchAsync netcoreapp3.1 1.66μs 0.542ns 2.03ns 0.0133 0 0 984 B
#4782 CallElasticsearchAsync net472 2.69μs 1.24ns 4.81ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.24μs 0.474ns 1.71ns 0.0127 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.67μs 1.34ns 4.84ns 0.0125 0 0 912 B
master ExecuteAsync net472 1.75μs 1.05ns 3.91ns 0.139 0 0 875 B
#4782 ExecuteAsync net6.0 1.21μs 0.987ns 3.69ns 0.0127 0 0 912 B
#4782 ExecuteAsync netcoreapp3.1 1.64μs 1.58ns 6.12ns 0.0122 0 0 912 B
#4782 ExecuteAsync net472 1.88μs 1.35ns 5.22ns 0.139 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.15μs 2.62ns 9.8ns 0.029 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.95μs 1.72ns 6.65ns 0.0346 0 0 2.63 KB
master SendAsync net472 7.83μs 4.66ns 18ns 0.525 0 0 3.31 KB
#4782 SendAsync net6.0 4.26μs 2.36ns 8.83ns 0.0276 0 0 2.1 KB
#4782 SendAsync netcoreapp3.1 4.92μs 1.91ns 6.88ns 0.0345 0 0 2.63 KB
#4782 SendAsync net472 7.71μs 4.48ns 17.3ns 0.523 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster 🎉 Fewer allocations 🎉

Faster 🎉 in #4782

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 1.300 63,700.00 49,000.00

Fewer allocations 🎉 in #4782

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 62.39 KB 59.7 KB -2.7 KB -4.32%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 65.54 KB 57.34 KB -8.19 KB -12.50%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 51.6μs 137ns 496ns 0 0 0 43.44 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53.3μs 266ns 1.13μs 0 0 0 42.64 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 38μs 140ns 524ns 0 0 0 62.39 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 96.9μs 2.83μs 28.3μs 0 0 0 43.29 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 89.8μs 3.79μs 37.5μs 0 0 0 42.64 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 61.4μs 674ns 6.46μs 0 0 0 65.54 KB
#4782 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 50.7μs 248ns 991ns 0 0 0 43.44 KB
#4782 StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53.3μs 230ns 861ns 0 0 0 42.64 KB
#4782 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 37.6μs 107ns 399ns 0 0 0 59.7 KB
#4782 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 94.4μs 3.14μs 31.4μs 0 0 0 43.34 KB
#4782 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 98.1μs 3.66μs 36.3μs 0 0 0 42.64 KB
#4782 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 49.1μs 139ns 502ns 0 0 0 57.34 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.41μs 1.03ns 3.99ns 0.0219 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.3μs 0.994ns 3.72ns 0.0205 0 0 1.57 KB
master EnrichedLog net472 2.68μs 2.2ns 7.95ns 0.238 0 0 1.5 KB
#4782 EnrichedLog net6.0 1.42μs 0.748ns 2.8ns 0.022 0 0 1.57 KB
#4782 EnrichedLog netcoreapp3.1 2.14μs 1.75ns 6.78ns 0.0216 0 0 1.57 KB
#4782 EnrichedLog net472 2.62μs 3.01ns 11.6ns 0.237 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 122ns 471ns 0.0558 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 181ns 678ns 0 0 0 4.21 KB
master EnrichedLog net472 150μs 105ns 408ns 0.669 0.223 0 4.39 KB
#4782 EnrichedLog net6.0 112μs 138ns 516ns 0.0555 0 0 4.21 KB
#4782 EnrichedLog netcoreapp3.1 118μs 135ns 522ns 0 0 0 4.21 KB
#4782 EnrichedLog net472 149μs 174ns 674ns 0.662 0.221 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.18μs 3.49ns 13.5ns 0.0301 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.14μs 1.52ns 5.88ns 0.0288 0 0 2.13 KB
master EnrichedLog net472 4.86μs 2.48ns 9.62ns 0.308 0 0 1.95 KB
#4782 EnrichedLog net6.0 3.11μs 2.2ns 8.54ns 0.0296 0 0 2.13 KB
#4782 EnrichedLog netcoreapp3.1 4.17μs 1.14ns 4.42ns 0.0291 0 0 2.13 KB
#4782 EnrichedLog net472 4.83μs 2.17ns 8.39ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.42μs 0.925ns 3.34ns 0.0156 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.75μs 0.461ns 1.73ns 0.0149 0 0 1.1 KB
master SendReceive net472 2.12μs 1.76ns 6.59ns 0.177 0 0 1.12 KB
#4782 SendReceive net6.0 1.32μs 1.2ns 4.63ns 0.0153 0 0 1.1 KB
#4782 SendReceive netcoreapp3.1 1.79μs 0.565ns 2.19ns 0.0152 0 0 1.1 KB
#4782 SendReceive net472 2.16μs 2.54ns 9.85ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.62μs 0.718ns 2.59ns 0.021 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 4.02μs 2.06ns 8ns 0.0203 0 0 1.58 KB
master EnrichedLog net472 4.42μs 1.73ns 6.49ns 0.312 0 0 1.97 KB
#4782 EnrichedLog net6.0 2.8μs 0.73ns 2.83ns 0.0211 0 0 1.53 KB
#4782 EnrichedLog netcoreapp3.1 3.82μs 0.429ns 1.66ns 0.021 0 0 1.58 KB
#4782 EnrichedLog net472 4.34μs 1.4ns 5.23ns 0.311 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4782

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.138 692.18 787.70
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.136 450.88 512.27

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 451ns 0.264ns 1.02ns 0.00757 0 0 536 B
master StartFinishSpan netcoreapp3.1 701ns 0.437ns 1.63ns 0.00726 0 0 536 B
master StartFinishSpan net472 692ns 0.274ns 1.02ns 0.0854 0 0 538 B
master StartFinishScope net6.0 593ns 0.541ns 2.1ns 0.00913 0 0 656 B
master StartFinishScope netcoreapp3.1 872ns 0.386ns 1.49ns 0.00882 0 0 656 B
master StartFinishScope net472 971ns 3.63ns 14.1ns 0.0979 0 0 618 B
#4782 StartFinishSpan net6.0 512ns 0.169ns 0.656ns 0.00748 0 0 536 B
#4782 StartFinishSpan netcoreapp3.1 712ns 0.466ns 1.8ns 0.00742 0 0 536 B
#4782 StartFinishSpan net472 788ns 0.355ns 1.37ns 0.0852 0 0 538 B
#4782 StartFinishScope net6.0 540ns 0.23ns 0.86ns 0.0092 0 0 656 B
#4782 StartFinishScope netcoreapp3.1 871ns 0.331ns 1.28ns 0.00864 0 0 656 B
#4782 StartFinishScope net472 994ns 0.629ns 2.44ns 0.0979 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 601ns 0.494ns 1.91ns 0.00913 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 910ns 0.278ns 1.08ns 0.0086 0 0 656 B
master RunOnMethodBegin net472 1.09μs 0.377ns 1.41ns 0.0981 0 0 618 B
#4782 RunOnMethodBegin net6.0 657ns 0.163ns 0.632ns 0.00924 0 0 656 B
#4782 RunOnMethodBegin netcoreapp3.1 936ns 0.437ns 1.69ns 0.0085 0 0 656 B
#4782 RunOnMethodBegin net472 1.15μs 0.962ns 3.72ns 0.098 0 0 618 B

@andrewlock
Copy link
Member

andrewlock commented Nov 4, 2023 •
edited
Loading

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4782) (11.453M)   : 0, 11453322
    master (11.178M)   : 0, 11178299
    benchmarks/2.9.0 (11.278M)   : 0, 11277678

    section Automatic
    This PR (4782) (7.956M)   : 0, 7956193
    master (7.684M)   : 0, 7683793
    benchmarks/2.9.0 (8.280M)   : 0, 8279938

    section Trace stats
    This PR (4782) (8.219M)   : 0, 8219179
    master (8.084M)   : 0, 8083685

    section Manual
    This PR (4782) (10.046M)   : 0, 10046060
    master (9.608M)   : 0, 9607594

    section Manual + Automatic
    This PR (4782) (7.506M)   : 0, 7506390
    master (7.046M)   : 0, 7046208

    section Version Conflict
    This PR (4782) (6.846M)   : 0, 6845579
    master (6.354M)   : 0, 6353736
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4782) (9.445M)   : 0, 9445008
    master (9.696M)   : 0, 9695911
    benchmarks/2.9.0 (9.522M)   : 0, 9521915

    section Automatic
    This PR (4782) (6.502M)   : 0, 6501968
    master (6.567M)   : 0, 6567464

    section Trace stats
    This PR (4782) (6.926M)   : 0, 6925611
    master (6.664M)   : 0, 6664101

    section Manual
    This PR (4782) (8.135M)   : 0, 8135495
    master (8.282M)   : 0, 8281743

    section Manual + Automatic
    This PR (4782) (6.162M)   : 0, 6161832
    master (6.122M)   : 0, 6121676

    section Version Conflict
    This PR (4782) (5.577M)   : 0, 5576518
    master (5.807M)   : 0, 5806841
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4782) (9.929M)   : 0, 9928515
    master (10.305M)   : 0, 10305271
    benchmarks/2.9.0 (10.126M)   : 0, 10125923

    section Automatic
    This PR (4782) (7.017M)   : 0, 7016762
    master (7.166M)   : 0, 7166143
    benchmarks/2.9.0 (7.604M)   : 0, 7603744

    section Trace stats
    This PR (4782) (7.440M)   : 0, 7440273
    master (7.482M)   : 0, 7481676

    section Manual
    This PR (4782) (8.762M)   : 0, 8761678
    master (8.866M)   : 0, 8865589

    section Manual + Automatic
    This PR (4782) (6.808M)   : 0, 6807712
    master (6.850M)   : 0, 6849808

    section Version Conflict
    This PR (4782) (6.044M)   : 0, 6043843
    master (6.290M)   : 0, 6290213
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.472M)   : 0, 7472418
    benchmarks/2.9.0 (8.011M)   : 0, 8011085

    section No attack
    master (1.778M)   : 0, 1778201
    benchmarks/2.9.0 (3.233M)   : 0, 3232865

    section Attack
    master (1.414M)   : 0, 1413790
    benchmarks/2.9.0 (2.564M)   : 0, 2563832

    section Blocking
    master (3.202M)   : 0, 3202424

    section IAST default
    master (6.525M)   : 0, 6525089

    section IAST full
    master (5.675M)   : 0, 5674796

    section Base vuln
    master (0.956M)   : 0, 955896

    section IAST vuln
    master (0.889M)   : 0, 888789
Loading

GreenMatan
GreenMatan approved these changes Nov 6, 2023
View reviewed changes
Copy link
Contributor

@GreenMatan GreenMatan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've already reviewed most of the juice in #4412.
Adding a few comments, I will review it further and possibly submit another batch of comments.

Good job on vendoring in all these wonderful packages that unlocks the great power of memory spans, array pools, and more. 💯

tracer/src/Datadog.Trace/Debugger/Symbols/SymbolBatchUploadApi.cs Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolExtractor.cs Outdated Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolExtractor.cs Outdated Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolExtractor.cs Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolExtractor.cs Outdated Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolPdbExtractor.cs Outdated Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolPdbExtractor.cs Outdated Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolsUploader.cs Outdated Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolsUploader.cs Show resolved Hide resolved
tracer/src/Datadog.Trace/Debugger/Symbols/SymbolsUploader.cs Outdated Show resolved Hide resolved
@dudikeleti dudikeleti force-pushed the dudik/upload-symbols-vendoring branch 4 times, most recently from 204bbf8 to 53a6068 Compare November 8, 2023 12:40
@dudikeleti dudikeleti force-pushed the dudik/upload-symbols-vendoring branch from e8580a2 to 34b26b3 Compare November 9, 2023 11:12
@dudikeleti dudikeleti force-pushed the dudik/upload-symbols-vendoring branch from dfc53c4 to 490b79e Compare January 17, 2024 16:43
@dudikeleti dudikeleti merged commit a0379fe into master Jan 18, 2024
58 of 60 checks passed
@dudikeleti dudikeleti deleted the dudik/upload-symbols-vendoring branch January 18, 2024 10:15
@github-actions github-actions bot added this to the vNext milestone Jan 18, 2024
This was referenced Jan 19, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock left review comments

@GreenMatan GreenMatan GreenMatan approved these changes

@OmerRaviv OmerRaviv OmerRaviv left review comments

@zacharycmontoya zacharycmontoya zacharycmontoya approved these changes

Assignees

@OmerRaviv OmerRaviv

@GreenMatan GreenMatan

Labels
area:debugger
Projects
None yet
Milestone
2.46.0
Development

Successfully merging this pull request may close these issues.
5 participants
@dudikeleti @andrewlock @OmerRaviv @zacharycmontoya @GreenMatan