Reduce the performance impact of large amount instrumentation

[jifeingo]

Description
The New Relic Profiler responds to CLR events for module loading and JIT compilation. It reads Instrumentation.xml files which identify the methods that are to be instrumented. The signatures of these methods are stored in a List (Set). When a JIT request is made for any function (instrumented or not), the list is consulted to see if the JITting function should have instrumentation instructions injected.

This process is inefficient and causes performance impacts when this list is large. As a list, every search results in a linear scan of the list.

The problem is confounded by the fact that instrumentation is agent/system wide. So, it affects all processes on the server.

Recommendation

• Instead of using a list can a look-up structure be used (such as Map) that is keyed based on the signature.
• Could a tree structure organized by Assembly -> Class -> Method Signature be used. This would result in a much quicker exit for any assembly or class that is not instrumented.

Steps to Reproduce

• Build a moderately complex application
• Add an instrumentation xml with 12K methods in it.

Your Environment
.NET Framework (but suspect it is also in .NET Core)

Additional context
[TIP]: # ( Add any other context about the problem here. For example, relevant community posts or support tickets. )

[kellycampbe]

The issue I have noticed is:

1. Dropping a large custom instrumentation file causes all of the app pools on the server to start consuming large amounts of CPU. This may last for 30 seconds if there are only a couple of app pools on the server, but with a large number of app pools it can lock up the server for 10 minutes or more.

2. If a large custom instrumentation file already exists on the server, adding a tiny custom instrumentation file (i.e. having only or two methods) also has the same impact causing large app pool CPU usage potentially crippling a server for a minutes.

3. Has the same impact across all app pools on the IIS server even if the instrumentation file is for methods that do not match any of the methods in the application.

[jifeingo]

Thanks for reaching out @kellycampbe.

Specifically for point 3:
This method is how we determine if a method should be instrumented. Ideally we would be able to start with our list and apply the instrumentation to methods. Unfortunately, .NET does not load all of the code (JIT it) at the start of the app. Only on the first usage does this occur. As a result, we respond to JIT events and have to do the lookup the other way around.

The more I think about it, I really think that a tree structure (assembly->class->method->signature), instead of a list, would help here. I think that we could quickly rule out 90% of methods based on the assembly alone, which would be a short operation.

[jifeingo]

https://newrelic.atlassian.net/browse/GTSE-4872
https://newrelic.zendesk.com/agent/tickets/429022