PkgProj infrastructure refactoring

[ericstj]

Let this issue serve as a meta-issue to track potential refactoring to be done to PkgProj infrastructure. I'll open issues to address these when we're ready to do them.

• Replace NuGetPack with shipping task
• Eliminate PackageDescription file: this is unnecessary seperation of package metadata. Ensure this removal covers NetCore package description commit hash could use some explanation? runtime#24074.
• Factor dependency harvesting into its own optional targets file. This is a cohesive set of features that can be described as follows:
  • Examining assemblies to determine dependencies.
  • Mapping dependencies to packages+versions.
  • Applying policy to package versions (baselining).
  • Reducing dependencies where inbox.
• Consolidate dependency-related tasks into a fewer tasks. Today we chain a bunch of tasks together in a single target. We don't gain anything from this since it raises items between the tasks that aren't extensible/intercept-able and results in extra work. Here are the candidate tasks:
  • ApplyBaseLine
  • ApplyMetaPackages
  • ApplyPreReleaseSuffix
  • CreateTrimDependencyGroups
  • FilterUnknownPackages
  • GetAssemblyReferences
  • PromoteDependencies
  • SplitReferences
• Make dependency-related tasks/targets work as an alternative to raising actual PackageDependency items from an SDK project. Folks have expressed an interest in using this package infrastructure in non PKGProjs and we can enable that.
• Consider eliminating items as source for package index data. We don't need to support both the flat items and the index. We should eliminate the duplication.
• Eliminate SplitDependenciesBySupport, GetMinimumNETStandard, GetInboxFrameworks tasks
  • These appear to be no longer used. We should confirm and remove.
• Remove / reduce runtime.json support.
  • We've largely eliminated the use of runtime.json but it remains as a proper concept in the tasks/targets. We should remove all of this and just permit the basic scenario of generating a runtime.json for our framework packages.
• Remove module->dependency mapping, GetPackageFromModule
  • This is a relic from when we needed to package native libs in a separate package and harvest the dllimport and map it to a module, then map that module to a package. We can remove this now that the majority of these are handled by the framework package. SqlClient can directly declare its package reference to the native DLL (if still needed).
• Investigate removal of GenerateNuSpec.
  • Last I checked NuGet didn't have an equivalent task, nor did they support defining all of the items we need and feeding those directly to the Pack task. That said, this isn't very specific to us and should be functionality provided by NuGet. We may need to drive features into the product to make this change.
• Factor package (reports and) testing into its own targets and consider scenarios where it might be used in something other than a pkgproj.
  • Improve supported framework declaration: can we make this more automatic? more version resilient to new frameworks/versions?
• Support SDK-style cross-targeting project references, determine if we can reduce complexity in package content definition (eg: packageLibs.targets).
  • Now that most of our packages are simpler, do we need all this extensibility?

[tmat]

Thanks!

[ViktorHofer]

Do we have plans to actually start working on this? Every time we hit a breaking change in corefx because of not using shipping tasks I'm blocked and have to wait for @ericstj to step in. In example right now: dotnet/corefx#37340 (comment)

EDIT: I don't know much about this codebase at all but I understand that it's vital to how we are shipping our product. I'm happy to have the person who wrote all this on the team.

[markwilkie]

@ViktorHofer - unfortunately this is not yet a priority. Perhaps there are pieces that can be "broken off" and implemented so that we can make progress over time? (I haven't internalized the scope of this work yet)

cc/ @alexperovich @tmat @JohnTortugo

[ericstj]

@ViktorHofer neither of the issues you hit would have been addressed by this.

You hit two issues:

1. Runtime.json generation was broken by NuGet making a breaking change to public API. The product doesn't have capability of generating runtime.json, so none of the work outlined here would have removed the code that broke.
2. Our package testing infrastructure happened to trigger a convention used by the .NET SDK. This had nothing to do with pkgproj infrastructure and was just a bad coincidence with how we copy bits around for testing.

[ericstj]

To be clear the result of this work would be to enable csproj to do what we do in pkgproj today.

[ericstj]

Much of this has been accomplished in 6.0. This issue is not really giving us much value in tracking tasks anymore.