NetCore 3.1 incompatibility with SDK dependencies?

[TimothyMothra]

This issue was originally reported on Azure docs

---

@SamShekhovtsov reported an issue:

Could you confirm that current latest version of the ApplicationInsights 'Microsoft.ApplicationInsights.AspNetCore' v2.12.0 is not compatible with the Core 3.1? When I try to add that library reference, it also adds more than 10 other libraries with older versions, that might conflict with Core 3.1

I was unable to reproduce this in a new web application with default settings.

@GoguX was able to reproduce this:

I was able to reproduce this issue by setting AutoGenerateBindingRedirects = true in the project file and targeting netcoreapp3.1. In my case,
the warnings I get are:

1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.Data.SqlClient" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.Diagnostics.PerformanceCounter" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.IO.FileSystem.AccessControl" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.

This is just creating a new console application and adding Microsoft.ApplicationInsights.AspNetCore version 2.13.0.

[TimothyMothra]

This is the full verbose log:

System.Data.SqlClient

1>    There was a conflict between "System.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" and "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
1>        "System.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was chosen because it was primary and "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was not.
1>        References which depend on "System.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll].
1>            C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll".
1>                C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll
1>        References which depend on "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [].
1>            C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll
1>              Project file item includes which caused reference "C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll".
1>                C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll

System.Diagnostics.PerformanceCounter

1>    There was a conflict between "System.IO.FileSystem.AccessControl, Version=4.0.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" and "System.IO.FileSystem.AccessControl, Version=4.0.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
1>        "System.IO.FileSystem.AccessControl, Version=4.0.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was chosen because it was primary and "System.IO.FileSystem.AccessControl, Version=4.0.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was not.
1>        References which depend on "System.IO.FileSystem.AccessControl, Version=4.0.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [C:\Users\tilee\.nuget\packages\system.io.filesystem.accesscontrol\4.5.0\ref\netstandard2.0\System.IO.FileSystem.AccessControl.dll].
1>            C:\Users\tilee\.nuget\packages\system.io.filesystem.accesscontrol\4.5.0\ref\netstandard2.0\System.IO.FileSystem.AccessControl.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\system.io.filesystem.accesscontrol\4.5.0\ref\netstandard2.0\System.IO.FileSystem.AccessControl.dll".
1>                C:\Users\tilee\.nuget\packages\system.io.filesystem.accesscontrol\4.5.0\ref\netstandard2.0\System.IO.FileSystem.AccessControl.dll
1>        References which depend on "System.IO.FileSystem.AccessControl, Version=4.0.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [].
1>            C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.windowsserver.telemetrychannel\2.13.1\lib\netstandard2.0\Microsoft.AI.ServerTelemetryChannel.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.windowsserver.telemetrychannel\2.13.1\lib\netstandard2.0\Microsoft.AI.ServerTelemetryChannel.dll".
1>                C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.windowsserver.telemetrychannel\2.13.1\lib\netstandard2.0\Microsoft.AI.ServerTelemetryChannel.dll
1>                C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.aspnetcore\2.13.1\lib\netstandard2.0\Microsoft.ApplicationInsights.AspNetCore.dll
1>            C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\mscorlib.dll
1>              Project file item includes which caused reference "C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\mscorlib.dll".
1>                C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\mscorlib.dll

System.IO.FileSystem.AccessControl

1>    There was a conflict between "System.Diagnostics.PerformanceCounter, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" and "System.Diagnostics.PerformanceCounter, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51".
1>        "System.Diagnostics.PerformanceCounter, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" was chosen because it was primary and "System.Diagnostics.PerformanceCounter, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" was not.
1>        References which depend on "System.Diagnostics.PerformanceCounter, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" [C:\Users\tilee\.nuget\packages\system.diagnostics.performancecounter\4.5.0\ref\netstandard2.0\System.Diagnostics.PerformanceCounter.dll].
1>            C:\Users\tilee\.nuget\packages\system.diagnostics.performancecounter\4.5.0\ref\netstandard2.0\System.Diagnostics.PerformanceCounter.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\system.diagnostics.performancecounter\4.5.0\ref\netstandard2.0\System.Diagnostics.PerformanceCounter.dll".
1>                C:\Users\tilee\.nuget\packages\system.diagnostics.performancecounter\4.5.0\ref\netstandard2.0\System.Diagnostics.PerformanceCounter.dll
1>        References which depend on "System.Diagnostics.PerformanceCounter, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51" [].
1>            C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.perfcountercollector\2.13.1\lib\netstandard2.0\Microsoft.AI.PerfCounterCollector.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.perfcountercollector\2.13.1\lib\netstandard2.0\Microsoft.AI.PerfCounterCollector.dll".
1>                C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.perfcountercollector\2.13.1\lib\netstandard2.0\Microsoft.AI.PerfCounterCollector.dll
1>                C:\Users\tilee\.nuget\packages\microsoft.applicationinsights.aspnetcore\2.13.1\lib\netstandard2.0\Microsoft.ApplicationInsights.AspNetCore.dll
1>            C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Core.dll
1>              Project file item includes which caused reference "C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Core.dll".
1>                C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Core.dll
1>            C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.dll
1>              Project file item includes which caused reference "C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.dll".
1>                C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.dll

[TimothyMothra]

This is super confusing. I'm trying to make sense of the verbose log.

I have this test app:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.13.1" />
  </ItemGroup>

  <ItemGroup>
    <WCFMetadata Include="Connected Services" />
  </ItemGroup>

</Project>

Microsoft.ApplicationInsights.AspNetCore depends on Microsoft.ApplicationInsights.DependencyCollector depends on Microsoft.Data.SqlClient (v4.3.1).
SqlClient package version 4.3.1 is assembly version 4.1.0.
This is the full build error from my test app:

1>    There was a conflict between "System.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" and "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
1>        "System.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was chosen because it was primary and "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was not.
1>        References which depend on "System.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll].
1>            C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll".
1>                C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.3.1\ref\netstandard1.3\System.Data.SqlClient.dll
1>        References which depend on "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [].
1>            C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll
1>              Project file item includes which caused reference "C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll".
1>                C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll

So I tested a local build using SqlClient package version v4.7.0.
Apparently package 4.7.0 is assembly 4.6.0.
This generates a new error:

1>    There was a conflict between "System.Data.SqlClient, Version=4.6.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" and "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
1>        "System.Data.SqlClient, Version=4.6.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was chosen because it was primary and "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was not.
1>        References which depend on "System.Data.SqlClient, Version=4.6.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.7.0\ref\netcoreapp2.1\System.Data.SqlClient.dll].
1>            C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.7.0\ref\netcoreapp2.1\System.Data.SqlClient.dll
1>              Project file item includes which caused reference "C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.7.0\ref\netcoreapp2.1\System.Data.SqlClient.dll".
1>                C:\Users\tilee\.nuget\packages\system.data.sqlclient\4.7.0\ref\netcoreapp2.1\System.Data.SqlClient.dll
1>        References which depend on "System.Data.SqlClient, Version=4.6.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [].
1>            C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll
1>              Project file item includes which caused reference "C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll".
1>                C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.Data.dll

Also worth noting, when I downgrade my test app from netcoreapp3.1 to netcoreapp3.0, this problem goes away.

Open Questions

• So where is the SqlClient assembly version 4.6.1 coming from? The verbose log suggests it comes from Microsoft.NETCore.App.Ref, but the NuGet for that package claims it has zero dependencies.
• Our DependencyCollector package uses the greater-than-or-equal-to System.Data.SqlClient (>= 4.3.1). So if something had a newer reference, I would have expected that newer version to take precedence. At this moment I don't understand why that's not happening here.

[TimothyMothra]

Reaching out to dotnet runtime for help... @joperezr do you have any thoughts?

[TimothyMothra]

I got confirmation that this appears to be related to #1707.
We should get a deeper explanation of the problem in the next few days.
I've been told the dotnet team already has a fix for this due to be released in mid-March.
After the fix is available, we (Application Insights) will need to update some of our dependencies.

[joperezr]

I had a bit more time to look at this today. The problem in here seems to be different than #1707. To start, the first problem here is that you shouldn't be setting the property AutoGenerateBindingRedirects on a netcoreapp project as binding redirects aren't even a thing in netcoreapp, they are only a concept in .NET Framework. Actually I believe it's even a bug on the SDK to start running the binding redirect targets on a netcoreapp project, and those targets are the ones that are producing the warnings that you are seeing. In the case the your project dual-targets (meaning you have a <TargetFrameworks>netcoreapp3.1;net461</TargetFrameworks> then you should only set the property for binding redirects for the net461 configuration like: <AutoGenerateBindingRedirects Condition="'$(TargetFramework)' == 'net461'">true</AutogenerateBindingRedirects>

Secondly, the reason why you won't be able to repro this on 2.x but you are able to do it when targeting netcoreapp 3.x is because we actually made a change on 3.x that changed the way we build our .NET Framework compatibility shims and introduced one regression which I explain in more detail here but basically it kind of makes you have to reference the packages that we ship with 3.x so that everything works smoothly. It would be really rare to hit issues, but if you do, they would happen only in the case that your netcoreapp3.x app is also using a .NET Framework library which relied on using our compatibility shims to run, which isn't super common.

Thirdly (and kind of an extension to the second point), the App Insights package hits this because it references very old versions of System.Data.SqlClient (4.3.1, new package version is 4.8.1), System.Diagnostics.PerformanceCounter(4.5.0, new package version is 4.7.0), and System.IO.FileSystem.AccessControl (4.5.0, new package version is 4.7.0). This means that in order to fix this issue, and also to follow the .NET recommendations of always trying to reference the latest package versions (which most of the times contain fixes and improvements), App Insight's packages should reference these new versions:

• Package Microsoft.ApplicationInsights.DependencyCollector currently depends on System.Data.SqlClient version 4.3.1, which should be updated to 4.8.1
• Package Microsoft.ApplicationInsights.PerfCounterCollector depends on System.Diagnostics.PerformanceCounter version 4.5.0, which should be updated to 4.7.0 instead.
• Package Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel depends on System.IO.FileSystem.AccessControl versijon 4.5.0, which should be updated to 4.7.0 instead.

Making the above suggestions will fix the potential runtime issues and ensure that the appInsights packages will work fine in netcoreapp3.1, and as an aside recomendation @SamShekhovtsov should remove the AutoGenerateBindingRedirects property from their netcoreapp project

[TimothyMothra]

I merged a fix for this.
2.14-beta3 should be released within the next week.
2.14 stable will be available in Early April.

[joperezr]

@ericstj just as FYI, App Insights team have fixed the issue with referencing old OOB packages that we were talking about.

[georgiosd]

@TimothyMothra still getting this error on Azure App Service with a 3.1 app using 2.14 package. What do I do?

[TimothyMothra]

@georgiosd can you please share your logs?
This error had a very detailed verbose log at compile time. We would need that to comment.
If you temporarily remove the dependency on our SDK do these errors disappear?

[georgiosd]

Not sure what you mean. There's nothing at compile time. Then after it's been deployed to Azure App Service:

            <Data>Could not find 'aspnetcorev2_inprocess.dll'. Exception message:
Error:
  An assembly specified in the application dependencies manifest (Project.deps.json) was not found:
    package: 'System.Diagnostics.PerformanceCounter', version: '4.5.0'
    path: 'runtimes/win/lib/netcoreapp2.0/System.Diagnostics.PerformanceCounter.dll'
</Data>

I have set the app service config (from portal) to 64-bit and installed the 64-bit 3.1 runtime extension. I have also tried tried specifying -r win-x64 during publish and configuring App Insights from the portal to coordinate with the SDK, all to no avail.

EDIT: the only way I could make it work was by completely disabling APM with App Insights from the App Service control. It's very frustrating to be using the latest and greatest, on the native hosting environment and have it not work...