Skip to content

Commit

Permalink
[Dynamic Instrumentation] Upload symbols to SymDB (with System.Reflec…
Browse files Browse the repository at this point in the history 
…tion.Metadata) (#4782)

* Initial commit for upload symbols

* Add sym db api endpoint and remove blockingcollection

* Tests WIP

* Add tests and fixed issue after Oskar's review

* Add service version and environment

* Fix NRE and reset byte index after clearing array

* PR comments and more fixes

* Add discovery service sand environment variable to enable\disable symbols upload

* Tests WIP

* Refactor SymbolExtractor class and support of compiler generated methods and types

* Change keys to snake_case

* Change the way we are waiting for a discovery service notification about symbol database endpoint

* Extract ShouldRetry logic and simplify get delay time

* Fix build and update tests

* Partial class test

* Update local function test

* revert SourceLinkInformationExtractor changes

* PR comments

* Change symbol batch size from MB to Bytes

* Fix build

* Fix brolen json and NRE

* Fix json, tests and enum string representation

* Fix missing method attributes and add class source file

* Fix assembly name

* Make sure to set Unknown in case there is not source info and set -1 for missing line info

* enum names should be SnakeCaseNamingStrategy

* update approvals tests

* change unknown line numbers to 0 or int.Max

* Set unknown for missing assembly path

* Unknown method end line should be 0

* Fix missing pdb info in async methods and update approvals

* Vendor System.Reflection.Metadata and dependencies

* WIP - Replace dnlib with System.Reflection.Metadata

* WIP - Replace dnlib with System.Reflection.Metadata #2

* Everything compiles

* Now some of the tests work

* More work on closures and update tests approvals

* All tests are passed

* Some strings to spans conversation

* Wrap dnlib and reflection metadata pdb usage in DatadogMetadataReader

Add more tests

* Update vendored code

* Revert "Update vendored code"

This reverts commit 39efa76.

* Update vendored code

* Naming and logs

* Fix Span and ReadOnlySpan vendored code

* Add PrivateAssets="All"

* Skip interfaces, methods without IL and more not relevant types

* Handle properties and hoisted arguments

* Fix pdb related info issues

* Fox array out of range in argument symbols

* Fix build error

* Fixed arguments duplication

* Fixed arguments duplication

* PR comments

* Change log level and text

* Fix typo in configuration key

* PR comments: configuration keys and error handling

* Add test for async method with different kind of locals

* Configuration keys

* * Add missing closures

* Handle hoisted arg as arg and not as local
* Change async method locals line number to 0

* * Add LanguageSpecific to field symbols

* Add local Scopes
* Use memory and span instead of allocating list inside loop
* Support local constants
* More tests

* * Add PdbExist to language specific of class

* Fix source start and end column
* Improve performance
* Add some missing vendored code

* Fix CI build

* Fix symbol extractor tests to pass CI

* Add System.IO.MemoryMappedFiles and System.Resources.ResourceManager to AssemblyReferencesHaveNotChanged test

* Update Trimming.xml

* Sanitize more symbol fields

* Fix tests

* Fix vendor version

* Change vendored libs version

* revert me

* Revert "revert me"

This reverts commit b98445b.

* Try run symbol extractor tests only on windows

* Update approvals

* Enable nullable

* Add SymbolDatabaseIncludes EV

* Add `readonly`, `const` and `async` annotations

* Symbol DB enablement WIP

* Update assembly filter

* Update missing-nullability-files

* Add VendoredCodeTests

* Fix CI tests

* Skip AsyncVoid test in linux

* Update depenabot versions

* Fix compilation error

* Update DefinitionPaths

* Fix missing field attributes

* Fix typo in upload symbols enablement

* update nullability

---------

Co-authored-by: Matan Green <matangrn@gmail.com>
  • Loading branch information
@dudikeleti @GreenMatan
dudikeleti and GreenMatan authored Jan 18, 2024
1 parent 47594e9 commit a0379fe
Show file tree
Hide file tree
Showing 609 changed files with 78,917 additions and 319 deletions.
62 changes: 57 additions & 5 deletions tracer/build/_build/UpdateVendors/VendoredDependency.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ static VendoredDependency()
pathToSrc: new[] { "dnlib-3.4.0", "src" },
transform: filePath => RewriteCsFileWithStandardTransform(filePath, originalNamespace: "dnlib"));


Add(
libraryName: "Datadog.Sketches",
version: "1.0.0",
Expand All @@ -82,6 +81,58 @@ static VendoredDependency()
transform: filePath => RewriteCsFileWithStandardTransform(filePath, originalNamespace: "IndieSystem.Text.RegularExpressions",
AddIfNetcoreapp31OrGreater, AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma),
relativePathsToExclude: new[] { "additional/HashCode.cs", "SR.resx" });

Add(
libraryName: "System.Collections.Immutable",
version: "7.0.0",
downloadUrl: "https://github.com/DataDog/dotnet-vendored-code/archive/refs/tags/1.0.0.zip",
pathToSrc: new[] { "dotnet-vendored-code-1.0.0", "System.Reflection.Metadata", "System.Collections.Immutable" },
transform: filePath =>
{
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Collections.", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
});

Add(
libraryName: "System.Memory",
version: "4.5.5",
downloadUrl: "https://github.com/DataDog/dotnet-vendored-code/archive/refs/tags/1.0.0.zip",
pathToSrc: new[] { "dotnet-vendored-code-1.0.0", "System.Reflection.Metadata", "System.Memory" },
transform: filePath =>
{
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
});

Add(
libraryName: "System.Private.CoreLib",
version: "1.0.0",
downloadUrl: "https://github.com/DataDog/dotnet-vendored-code/archive/refs/tags/1.0.0.zip",
pathToSrc: new[] { "dotnet-vendored-code-1.0.0", "System.Reflection.Metadata", "System.Private.CoreLib" },
transform: filePath =>
{

RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Runtime", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Diagnostics", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "FxResources", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
});

Add(
libraryName: "System.Reflection.Metadata",
version: "7.0.2",
downloadUrl: "https://github.com/DataDog/dotnet-vendored-code/archive/refs/tags/1.0.0.zip",
pathToSrc: new[] { "dotnet-vendored-code-1.0.0", "System.Reflection.Metadata", "System.Reflection.Metadata" },
transform: filePath =>
{
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Reflection.", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Collections.", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Runtime.", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma);
});

Add(
libraryName: "System.Runtime.CompilerServices.Unsafe",
version: "1.0.0",
downloadUrl: "https://github.com/DataDog/dotnet-vendored-code/archive/refs/tags/1.0.0.zip",
pathToSrc: new[] { "dotnet-vendored-code-1.0.0", "System.Reflection.Metadata", "System.Runtime.CompilerServices.Unsafe" },
transform: filePath => RewriteCsFileWithStandardTransform(filePath, originalNamespace: "System.Runtime", AddNullableDirectiveTransform, AddIgnoreNullabilityWarningDisablePragma));
}

public static List<VendoredDependency> All { get; set; } = new List<VendoredDependency>();
Expand Down Expand Up @@ -240,9 +291,11 @@ private bool TryGetValue(string key, [NotNullWhen(true)]out JsonProperty? item)
// want to take any risk of calling it, ever, so replace it with a noop.
builder.Replace("Debugger.Break();", "{}");

string datadogVendoredNamespace = originalNamespace.StartsWith("System") ? "Datadog.Trace.VendoredMicrosoftCode." : "Datadog.Trace.Vendors.";

// Prevent namespace conflicts
builder.Replace($"using {originalNamespace}", $"using Datadog.Trace.Vendors.{originalNamespace}");
builder.Replace($"namespace {originalNamespace}", $"namespace Datadog.Trace.Vendors.{originalNamespace}");
builder.Replace($"using {originalNamespace}", $"using {datadogVendoredNamespace}{originalNamespace}");
builder.Replace($"namespace {originalNamespace}", $"namespace {datadogVendoredNamespace}{originalNamespace}");
builder.Replace($"[CLSCompliant(false)]", $"// [CLSCompliant(false)]");

// Fix namespace conflicts in `using alias` directives. For example, transform:
Expand All @@ -253,8 +306,7 @@ private bool TryGetValue(string key, [NotNullWhen(true)]out JsonProperty? item)
Regex.Replace(
builder.ToString(),
@$"using\s+(\S+)\s+=\s+{Regex.Escape(originalNamespace)}.(.*);",
match => $"using {match.Groups[1].Value} = Datadog.Trace.Vendors.{originalNamespace}.{match.Groups[2].Value};");

match => $"using {match.Groups[1].Value} = {datadogVendoredNamespace}{originalNamespace}.{match.Groups[2].Value};");


// Don't expose anything we don't intend to
Expand Down
9 changes: 9 additions & 0 deletions tracer/dependabot/Datadog.Dependabot.Vendors.csproj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@
<!-- https://www.nuget.org/packages/IndieSystem.Text.RegularExpressions/0.6 -->
<PackageReference Include="IndieSystem.Text.RegularExpressions" Version="0.6" />

<!-- https://www.nuget.org/packages/System.Collections.Immutable/1.0.0 -->
<PackageReference Include="System.Collections.Immutable" Version="7.0.0" />

<!-- https://www.nuget.org/packages/System.Memory/1.0.0 -->
<PackageReference Include="System.Memory" Version="4.5.5" />

<!-- https://www.nuget.org/packages/System.Reflection.Metadata/1.0.0 -->
<PackageReference Include="System.Reflection.Metadata" Version="7.0.2" />

</ItemGroup>

</Project>
3 changes: 0 additions & 3 deletions tracer/missing-nullability-files.csv
View file
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,6 @@ src/Datadog.Trace/ContinuousProfiler/ProfilerStatus.cs
src/Datadog.Trace/DatabaseMonitoring/DatabaseMonitoringPropagator.cs
src/Datadog.Trace/DataStreamsMonitoring/CheckpointKind.cs
src/Datadog.Trace/Debugger/ILineProbeResolver.cs
src/Datadog.Trace/Debugger/LineProbeResolver.cs
src/Datadog.Trace/Debugger/LiveDebugger.cs
src/Datadog.Trace/Debugger/ProbeLocationType.cs
src/Datadog.Trace/DiagnosticListeners/AspNetCoreDiagnosticObserver.cs
Expand Down Expand Up @@ -173,8 +172,6 @@ src/Datadog.Trace/HttpOverStreams/IHttpContent.cs
src/Datadog.Trace/Iast/Iast.cs
src/Datadog.Trace/Iast/ITaintedMap.cs
src/Datadog.Trace/Iast/SourceType.cs
src/Datadog.Trace/PDBs/DatadogPdbReader.cs
src/Datadog.Trace/PDBs/SymbolMethodExtensions.cs
src/Datadog.Trace/PlatformHelpers/AspNetCoreHttpRequestHandler.cs
src/Datadog.Trace/PlatformHelpers/AzureContext.cs
src/Datadog.Trace/PlatformHelpers/ContainerMetadata.cs
Expand Down
Loading

0 comments on commit a0379fe

Please sign in to comment.