Follow up with ApplicationInsight concerning Thread usages

[suwatch]

While investigating thread exhaustion issue, we observed many thread stack from ApplicationInsight (few examples below and full info in here. We should follow up with ApplicationInsight team to see if this is normal.

OS Thread Id: 0x1584 (454)
        Child SP               IP Call Site
000000c17a97e128 00007ff929f46724 [HelperMethodFrame: 000000c17a97e128] System.Threading.Thread.SleepInternal(Int32)
000000c17a97e220 00007ff91972d32a System.Threading.Thread.Sleep(Int32)
000000c17a97e250 00007ff919f8eff6 System.Threading.Thread.Sleep(System.TimeSpan)
000000c17a97e290 00007ff8bd154689 Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryModule.CollectionThreadWorker(System.Object)
000000c17a97e360 00007ff919764750 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000c17a97e430 00007ff9197645e4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000c17a97e460 00007ff9197645b2 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000c17a97e4b0 00007ff91a1201ac System.Threading.ThreadHelper.ThreadStart(System.Object)
000000c17a97e708 00007ff91bd25a03 [GCFrame: 000000c17a97e708] 
000000c17a97ea58 00007ff91bd25a03 [DebuggerU2MCatchHandlerFrame: 000000c17a97ea58] 
000000c17a97ebe8 00007ff91bd25a03 [ContextTransitionFrame: 000000c17a97ebe8] 
000000c17a97ee18 00007ff91bd25a03 [DebuggerU2MCatchHandlerFrame: 000000c17a97ee18] 
OS Thread Id: 0x1e40 (455)
        Child SP               IP Call Site
000000c17aafde88 00007ff929f46124 [InlinedCallFrame: 000000c17aafde88] System.Net.UnsafeNclNativeMethods+OSSOCK.select(Int32, IntPtr[], IntPtr[], IntPtr[], System.Net.Sockets.TimeValue ByRef)
000000c17aafde88 00007ff9187d2a68 [InlinedCallFrame: 000000c17aafde88] System.Net.UnsafeNclNativeMethods+OSSOCK.select(Int32, IntPtr[], IntPtr[], IntPtr[], System.Net.Sockets.TimeValue ByRef)
000000c17aafde50 00007ff9187d2a68 DomainNeutralILStubClass.IL_STUB_PInvoke(Int32, IntPtr[], IntPtr[], IntPtr[], System.Net.Sockets.TimeValue ByRef)
000000c17aafdf30 00007ff9187370c7 System.Net.Sockets.Socket.Poll(Int32, System.Net.Sockets.SelectMode)
000000c17aafdfd0 00007ff918738978 System.Net.Connection.SyncRead(System.Net.HttpWebRequest, Boolean, Boolean)
000000c17aafe030 00007ff9187399bf System.Net.Connection.PollAndRead(System.Net.HttpWebRequest, Boolean)
000000c17aafe080 00007ff9187521fe System.Net.ConnectStream.WriteHeaders(Boolean)
000000c17aafe110 00007ff91875114c System.Net.HttpWebRequest.EndSubmitRequest()
000000c17aafe150 00007ff91872e96c System.Net.HttpWebRequest.GetResponse()
000000c17aafe1f0 00007ff8bd013d58 Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.Implementation.QuickPulse.QuickPulseServiceClient.SendRequest(System.String, Boolean, System.String, System.String, Microsoft.ApplicationInsights.Extensibility.Filtering.CollectionConfigurationInfo ByRef, System.Action`1<System.IO.Stream>)
000000c17aafe280 00007ff8bd0137a2 Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.Implementation.QuickPulse.QuickPulseCollectionStateManager.UpdateState(System.String, System.String)
000000c17aafe350 00007ff8bd0132dc Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryModule.StateThreadWorker(System.Object)
000000c17aafe410 00007ff919764750 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000c17aafe4e0 00007ff9197645e4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000c17aafe510 00007ff9197645b2 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000c17aafe560 00007ff91a1201ac System.Threading.ThreadHelper.ThreadStart(System.Object)
000000c17aafe7b8 00007ff91bd25a03 [GCFrame: 000000c17aafe7b8] 
000000c17aafeb08 00007ff91bd25a03 [DebuggerU2MCatchHandlerFrame: 000000c17aafeb08] 
000000c17aafec98 00007ff91bd25a03 [ContextTransitionFrame: 000000c17aafec98] 
000000c17aafeec8 00007ff91bd25a03 [DebuggerU2MCatchHandlerFrame: 000000c17aafeec8] 

[paulbatum]

I analyzed a dump in this state. It looks like we have a memory leak as I found 8 QuickPulseTelemetryProcessor instances, all with isCollecting:true.

The objects appear to be rooted within a generated class (__DisplayClass) from DefaultTelemetryClientFactory:

I suspect its due to the closure our telemetry processors capture of state within the factory:
https://github.com/Azure/azure-webjobs-sdk/blob/v2.x/src/Microsoft.Azure.WebJobs.Logging.ApplicationInsights/DefaultTelemetryClientFactory.cs#L77-L95

Further investigation required...

[brettsam]

We're not disposing of our logger that we create before parsing host.json (in case we need to log errors). Running locally, I could see this thread popped up twice with every host start, then reduced by one when we disposed the host.

#2655 resolves this.