Don't run ResolveAssemblyReference if there are no conflicts in the project.assets.json

[DamianEdwards]

In a new ASP.NET Core 2.0 application, dotnet build is slower than in an ASP.NET Core 1.x application. This is primarily due to the new meta-package Microsoft.AspNetCore.All that brings in the entire product graph (~180 assemblies) causing ResolveAssemblyReference (RAR) to take a long time (5-7 seconds on my cold very powerful, dev machine, still greater than 1 second after warm-up), which accounts for the vast majority of the build time.

Can we detect when running RAR is unnecessary, based on if no assembly name appears more than once in the project.assets.json file? Even better if this is determined during restore, so that it doesn't need to be calculated during build at all.

This combined with the upcoming no-op dotnet restore performance work incoming for preview2, would get build times down for simple .cs file changes to just 100's of milliseconds (assuming we can avoid the lock file and package related tasks and targets profiled below).

Result of warmed up dotnet build execution on ASP.NET Core 2.0-preview1 project

Microsoft (R) Build Engine version 15.3.117.23532
Copyright (C) Microsoft Corporation. All rights reserved.

  RazorPagesSample -> C:\src\GitHub\damianedwards\RazorPagesSample\RazorPagesSample\bin\Debug\netcoreapp2.0\RazorPagesSample.dll

Project Performance Summary:
     1976 ms  C:\src\GitHub\damianedwards\RazorPagesSample\RazorPagesSample\RazorPagesSample.csproj   1 calls
               1976 ms  Build                                      1 calls

Target Performance Summary:
        0 ms  GetFrameworkPaths                          1 calls
        0 ms  CoreBuild                                  1 calls
        0 ms  PrepareForRun                              1 calls
        0 ms  AfterBuild                                 1 calls
        0 ms  ResolvePackageDependenciesForBuild         1 calls
        0 ms  GetReferenceAssemblyPaths                  1 calls
        0 ms  BeforeResolveReferences                    1 calls
        0 ms  PrepareProjectReferences                   1 calls
        0 ms  PrepareResourceNames                       1 calls
        0 ms  BeforeResGen                               1 calls
        0 ms  InjectReference_c9d046ba-023f-4780-b146-c50613a1c79f   1 calls
        0 ms  ResolveReferences                          1 calls
        0 ms  ResGen                                     1 calls
        0 ms  _CopySourceItemsToOutputDirectory          1 calls
        0 ms  Compile                                    1 calls
        0 ms  Build                                      1 calls
        0 ms  InjectReference_9b0c0bce-b127-49ab-878f-732dbd5a9266   1 calls
        0 ms  SetWin32ManifestProperties                 1 calls
        0 ms  CreateSatelliteAssemblies                  1 calls
        0 ms  AfterResGen                                1 calls
        0 ms  BeforeCompile                              1 calls
        0 ms  GenerateUserSecretsAttribute               1 calls
        0 ms  GenerateAssemblyInfo                       1 calls
        0 ms  IncludeTransitiveProjectReferences         1 calls
        0 ms  PrepareResources                           1 calls
        0 ms  CreateCustomManifestResourceNames          1 calls
        0 ms  BeforeBuild                                1 calls
        0 ms  AfterResolveReferences                     1 calls
        0 ms  ResolveSDKReferences                       1 calls
        0 ms  ResolveLockFileAnalyzers                   1 calls
        0 ms  ExpandSDKReferences                        1 calls
        0 ms  AfterCompile                               1 calls
        0 ms  BuildOnlySettings                          1 calls
        0 ms  _DefaultMicrosoftNETPlatformLibrary        1 calls
        0 ms  _SetTargetFrameworkMonikerAttribute        1 calls
        0 ms  InjectReference_e8e0da65-2413-440d-8f20-4b7e700afc39   1 calls
        0 ms  GetInstalledSDKLocations                   1 calls
        0 ms  ComputeDependencyFileCompilerOptions       1 calls
        0 ms  GetTargetPath                              1 calls
        0 ms  _SplitProjectReferencesByFileExistence     1 calls
        0 ms  _CheckForUnsupportedSelfContained          1 calls
        0 ms  _CheckForCompileOutputs                    1 calls
        0 ms  ResolveProjectReferences                   1 calls
        0 ms  _GenerateSatelliteAssemblyInputs           1 calls
        1 ms  _GenerateCompileInputs                     1 calls
        1 ms  CoreResGen                                 1 calls
        1 ms  _ComputeLockFileFrameworks                 1 calls
        1 ms  GenerateBuildDependencyFile                1 calls
        1 ms  IncrementalClean                           1 calls
        1 ms  _CopyOutOfDateSourceItemsToOutputDirectory   1 calls
        2 ms  _ComputeActiveTFMFileDependencies          1 calls
        2 ms  _ComputeTransitiveProjectReferences        1 calls
        2 ms  _ComputeNETCoreBuildOutputFiles            1 calls
        2 ms  GetCopyToOutputDirectoryItems              1 calls
        2 ms  CopyFilesToOutputDirectory                 1 calls
        2 ms  _ComputeTFMOnlyFileDependencies            1 calls
        2 ms  SplitResourcesByCulture                    1 calls
        2 ms  _ComputeLockFileAnalyzers                  1 calls
        2 ms  GetAssemblyVersion                         1 calls
        3 ms  _GetProjectReferenceTargetFrameworkProperties   1 calls
        3 ms  CheckForDuplicateItems                     1 calls
        3 ms  _SetEmbeddedWin32ManifestProperties        1 calls
        3 ms  _ComputeActiveTFMPackageDependencies       1 calls
        3 ms  CoreGenerateUserSecretsAttribute           1 calls
        3 ms  CoreGenerateAssemblyInfo                   1 calls
        4 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
        4 ms  _CleanGetCurrentAndPriorFileWrites         1 calls
        5 ms  AssignTargetPaths                          1 calls
        5 ms  GenerateTargetFrameworkMonikerAttribute    1 calls
        6 ms  _CheckForUnsupportedTargetFramework        1 calls
       10 ms  PrepareForBuild                            1 calls
       12 ms  CheckForImplicitPackageReferenceOverrides   1 calls
       15 ms  _GenerateCompileDependencyCache            1 calls
       36 ms  RunProduceContentAssets                    1 calls
       38 ms  CoreCompile                                1 calls
       87 ms  _ComputeLockFileReferences                 1 calls
       99 ms  GenerateBuildRuntimeConfigurationFiles     1 calls
      116 ms  ResolveLockFileReferences                  1 calls
      116 ms  _ComputeLockFileCopyLocal                  1 calls
      141 ms  _HandlePackageFileConflicts                1 calls
      164 ms  RunResolvePackageDependencies              1 calls
     1061 ms  ResolveAssemblyReferences                  1 calls

Task Performance Summary:
        0 ms  Delete                                     1 calls
        0 ms  RemoveDuplicates                           2 calls
        0 ms  ReadLinesFromFile                          1 calls
        1 ms  FindAppConfigFile                          1 calls
        1 ms  ConvertToAbsolutePath                      1 calls
        1 ms  AssignCulture                              1 calls
        1 ms  MSBuild                                    1 calls
        1 ms  FindUnderPath                              5 calls
        1 ms  Copy                                       2 calls
        1 ms  Message                                    3 calls
        1 ms  WriteLinesToFile                           1 calls
        2 ms  GetFrameworkPath                           1 calls
        2 ms  GetAssemblyVersion                         1 calls
        3 ms  Hash                                       1 calls
        3 ms  CheckForDuplicateItems                     3 calls
        4 ms  AssignTargetPath                           5 calls
        6 ms  CheckForImplicitPackageReferenceOverrides   1 calls
        9 ms  MakeDir                                    1 calls
       14 ms  ProduceContentAssets                       1 calls
       97 ms  ResolvePackageFileConflicts                1 calls
       98 ms  GenerateRuntimeConfigurationFiles          1 calls
      161 ms  ResolvePackageDependencies                 1 calls
     1059 ms  ResolveAssemblyReference                   1 calls

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.28

@davidfowl @livarcocc @JunTaoLuo

[eerhardt]

This is related to dotnet/msbuild#2015.

[dasMulli]

Related: https://github.com/dotnet/cli/issues/5918
(TL;DR the build time increases linear to the number of projects involved in the graph. 3x "7-second" projects => 21 seconds)

[rainersigwald]

I don't think I understand the request here. RAR is the way @(ReferencePath) is determined. How could it be elided?

[davidfowl]

The theory is that RAR only needs to be run to resolve dll conflicts as nuget already figures out package conflicts. After thinking about this a bit, it may not work as cleanly as I expected because we now embed the same dlls in Microsoft.NETCore.App and NETStandard.Library so there might always be conflicts in 2.0 applications.

The goal to was to figure out ways to elide calling RAR altogether.

[DamianEdwards]

The other suggestion made was to pass all the ASP.NET Core assemblies as "Framework Assemblies" to RAR, as that might result in it processing them less (or not at all). Do we have any more information on that option?

[DamianEdwards]

ping

[DamianEdwards]

@rainersigwald @kieranmo

[eerhardt]

we now embed the same dlls in Microsoft.NETCore.App and NETStandard.Library so there might always be conflicts in 2.0 applications.

These conflicts are handled outside of RAR in the SDK. We have https://github.com/dotnet/sdk/tree/master/src/Tasks/Microsoft.NET.Build.Tasks/ConflictResolution that handles these conflicts.

/cc @ericstj @dsplaisted

[ericstj]

RAR is less about conflicts and more about resolving paths. In fact, RAR doesn't even handle conflicting primary references, it just passes them along. We'd probably want to profile RAR to find out what's taking so much time. I bet there is some opportunity to back things with a cache. I suspect that every time a project builds it reloads the assemblies from the nuget package folder and examines metadata and references. Since packages are immutable it could cache this.

Bypassing RAR for things that are already resolved to a path is doable and would definitely help, but we'd want to make sure all metadata is set correctly on ReferencePaths items. It does things like examine WinMDs and preserves assembly metadata. We'd also not want to do that for desktop since RAR is responsible for finding conflicts and feeding bindingRedirects into app.config.

I think this issue belongs in the MSBuild repo, not SDK.

[rainersigwald]

Yeah, I don't think this is distinct from dotnet/msbuild#2015.

We can't just skip RAR for the reasons @ericstj points out: that's the only way the items that get fed to the compiler get populated.

[DamianEdwards]

@kieranmo mentioned to me that it might be possible to pass more data to RAR in order for it to treat the ASP.NET Core assemblies like FX assemblies, which means don't delve into them for dependencies. Does anyone know if it's possible for us to hack that up now to see if it helps? e.g. can we create an items list during build or something?

[DamianEdwards]

I'm going to close this in lieu of dotnet/msbuild#2015